Will Future Technologies Be Able to Impact the Digital Storage Market?

Since the 2010s, we’ve been living in the Zettabyte Era. In fact, by next year, we’re projected to have  created a mindboggling 181 Zettabytes of data across mediums! For those unfamiliar with the term, a Zettabyte is equivalent to 1 Billion Terabytes. Or about 10²¹ Bytes.

As the digital revolution continues to evolve with advancements in AI and the increasing use of IoT, we humans continue to produce vast amounts of data every single minute. While data storage technologies have evolved remarkably since the first magnetic tapes and floppy disks, the rate of these advancements needs to pick up rapidly to be able to keep up with all this new data.

As we continue to generate new data at unprecedented rates, the need for innovative storage technologies is becoming more critical than ever. If these emerging storage technologies can address the limitations of current systems, they could change the way we store and access information forever by enabling faster, more reliable, and larger-capacity storage solutions.

But Why Do We Need New Data Storage Methods?

We’ve spoken about how global data storage needs are growing at an exponential rate. But why can’t we just scale up traditional storage technology? Here are a few key reasons why we may need to innovate:

1. 🌐 Capacity: Existing storage technologies like HDDs and SSDs are continuously increasing their max capacity, and although slow, we might be able to reach Petabyte-capacity SSDs in the near future. As we generate more data, we need solutions that offer significantly higher capacities.
2. ⚡ Speed: The speed at which data can be written and retrieved is very important, especially when you consider modern applications that deal with big data, real-time analytics, and AI. Current technologies sometimes fall short in providing these required read/write speeds.
3. 🛡️ Durability: Perhaps the most important one; your data’s longevity and reliability are critical. The storage media we have currently are prone to degradation over time, which can potentially lead to data loss. Think about it: how many hard drives have you replaced in the last decade?
4. 🌱 Energy Efficiency: As data centers grow, so do their energy requirements. Some estimates put the energy usage of data centers in the US as high as tens of Terwatt-hours per year. More efficient storage technologies are the need of the hour so that they can reduce the environmental impact and operational costs of these data centers.

Emerging Storage Technologies – The Future of Digital Storage?

So, let’s look at the future of storage technology. Some of these might sound like fiction, but they truly are in development. Who knows, maybe in the near future, they might even become our primary storage method!

Holographic Storage

Image source: https://www.microsoft.com/en-us/research/project/hsd/how-does-holographic-storage-work/

Holographic storage uses laser beams to store data in three dimensions, which can potentially let us store large amounts of data in a smaller physical space.

Unlike traditional storage that records data only on the surface, holographic storage writes data throughout the medium’s volume (a small cube-shaped crystal) using a laser to create tiny holograms in it. Each data block (or page) can hold a hundred kilobytes, and within this same space, multiple pages can be recorded at different angles.

For reading the data, light is diffracted off the hologram and captured using a camera. As smartphone camera technology improves, fitting these tiny cameras into storage units is becoming easier and easier.

Let’s look at some key points about this technology:

• 🎯 Current Stage: This technology was first hypothesized as early as the 1960s and has been touted as being “commercially viable in the near future” since 2001. But it is still in the development phase, with some working prototypes demonstrating its feasibility. The most recent and promising development seems to be Microsoft’s Project HSD, which started in 2020 as a collaboration between Microsoft Research Cambridge and Microsoft Azure. Working prototypes have also been displayed at several conventions, but these have only been prototypes without a commercial product ever being released to date.
• 💹 Price and Scalability: There are no current known costs as no commercial product exists on the market. Research and development on Holographic storage obviously has high initial costs, and one of the major companies in this field, InPhase Technologies, filed for bankruptcy in 2011 (which, interestingly enough, has been indirectly acquired by Apple). As for scalability, even though it theoretically offers large storage capacities, the bigger picture lies in the super fast access speeds it provides by removing the limitations of mechanical moving parts and replacing them with light.
• 🌩️ Challenges: Transitioning from research to commercial products will certainly require significant development and resources. Manufacturing these novel holographic devices would require entirely new, high-end factory environments. The current best version of this tech, Microsoft’s Project HSD, is only focused on a single zone inside the crystal. Multiple pages can be recorded in this one zone, but the project is facing challenges in replicating performance across multiple zones. Another worry is ensuring that accidental UV light exposure doesn’t erase the stored data.

DNA Storage

While it has quite a few skeptics, DNA storage is an exciting area of research for future storage technologies. If you think about it, the human body is an insanely complex storage system that stores replication data for trillions of cells. DNA storage tries to encode data into the sequences of individual DNA molecules by leveraging this natural ability of DNA to store information in a highly dense and stable format.

Using current methods, your data is first converted into binary code and then mapped to the four nucleotide bases of DNA, commonly abbreviated as A,C,G,T. In fact, scientists have been able to tinker around and sequence and synthesize (read and write) with DNA for decades now. This handy diagram explains the process briefly:

Image source: https://spectrum.ieee.org/dna-data-storage

The best part of DNA storage tech is definitely its density; estimates say that the entirety of the Internet, about 180 Zettabytes, could be stored in a volume of DNA the same size of a sugar cube!

• 🎯 Current Stage: This technology is currently in the prototype and research phase. In 2011, researchers at the EBI institute had successfully encoded five files, including Shakespeare’s sonnets and a snippet of Martin Luther King’s “I Have a Dream” speech! This was a major milestone, no doubt, but this achievement sat at a mere 789kb. This has since been followed by a significant advancement from Microsoft, who, in collaboration with the University of Washington, managed to encode 200 megabytes of data into DNA.
• 💹 Price and Scalability: As a new and emerging storage tech, putting your data on DNA is pricey. Scalability is definitely on the horizon for DNA as a storage medium, given its extremely high memory density and half-life of 500 years. But all of that can only be addressed once the glaring issues (read/write speeds) get resolved. Currently, you can buy Biomemory’s 1KB DNA storage drives…for a whopping $1000!
• 🌩️ Challenges: Microsoft’s page on DNA storage points out the main challenge: it is not practical yet due to the current state of DNA synthesis and sequencing. That being said, these technologies are rapidly improving as we make advancements in the biotech industry. And given the impending limits of silicon technology (end of Moore’s Law), hybrid silicon and biochemical systems should be higher up on our priority list.

Ceramic Data Storage (Cerabyte)

Image source: https://www.cerabyte.com/

Recorded history starts with our earliest ancestors writing on stone tablets, and Ceramic Data Storage could bring a new twist to our ancient history.

Unlike the other technologies on this list, there’s really only one main player in this field – Cerabyte. This German company, founded in 2022, has its fair share of internet skeptics thanks to its bold claims (2 million bits written in a single laser pulse, with hundreds of terabytes of storage per ceramic disc). This tech only allows permanent writing and then reading of the inscribed information, so it makes sense for long-term storage and archival purposes. This infographic from their website explains the tech in a straightforward manner:

Image source: https://www.cerabyte.com/

• 🎯 Current Stage: Cerabyte is still in the prototype phase and is using commercial off-the-shelf components to demonstrate the feasibility of their technology. They claimed to have achieved a significant breakthrough earlier this year, by tripling the data density with a UV laser compared to their initial setup.
• 💹 Price and Scalability: Currently, the costs remain high, but Cerabyte aims to reduce these significantly for commercial versions of their product. In this article, a spokesperson for the company talk about how the transition to particle beam technology in the future is expected to lower costs to far below $1/TB/month. Scalability remains a challenge, mostly due to development needed in particle beam technology and creating their own proprietary factory units within the next 5-10 years.
• 🌩️ Challenges: The current setup that Cerabyte uses builds on an older technology called maskless lithography. The company eventually wants to transition into more advanced particle beam technology that will be costlier and complex, and hasn’t been developed yet. Furthermore, mass-producing these ceramic materials for consumer use is still years away, with loads of research and scientific advancement left to do.

Quantum Storage

While data is traditionally stored in binary bits (either 1 or 0), quantum computing utilizes a special quantum property of matter called superpositioning. This would let us store bits as 1, 0 or a combination of both in a special bit called a qubit. Think of it like a coin- you could either get heads or tails, but quantum storage would let the coin flip onto its edge, so its technically both heads and tails at the same time!

One major problem is that it is very error-prone in its current state and also fragile, storing data for only about 100 milliseconds. This non-persistence, while similar to modern-day computer RAM, is orders of magnitude more short-lived. So, while it can’t be used for regular computing, it would be a game-changer for solving complex math problems and other specialized programs.

• 🎯 Current Stage: Quantum storage is still very much theoretical, with some experimental setups demonstrating its potential. Many top universities and scientific organizations of the world are working on quantum computing and storage, but there are no commercially viable products yet. As discussed above, even with future developments, quantum memory would store data for only about 100ms and may not be set out for ubiquitous everyday use.
• 💹 Price and Scalability: The costs associated with quantum storage are hard to put an exact number on since they’re still being actively developed. That being said, these costs will likely be very high due to the advanced tech and infrastructure required, including cryogenic systems and error correction mechanisms that haven’t been developed yet. Scalability is a major challenge as well, and developing reliable and stable quantum storage systems on a commercial scale will surely require a couple more breakthroughs.
• 🌩️ Challenges: Given its specific use cases in scientific applications, re-purposing quantum storage for the mass commercial storage industry is unlikely. There are also controversies surrounding the implications of such tech, which could very well render current encryption standards obsolete (it is hypothesized that quantum computers would be able to brute force current encryption standards in a fraction of the time it would take any regular computer to do).

Other Optical Data Storage Advances

We’re all familiar with data storage tech that uses light to read and write data, as seen in our traditional formats like CDs and DVDs. The advances in this field, like 5D optical storage and multi-layer optical discs, are aiming to try and improve both data density and longevity on the stored medium.

Image source: https://techmgzn.com/5d-storage-technology-can-fit-500tb-on-a-small-disc/

While regular CDs store information in two dimensions with tiny pits on the surface reflecting laser light to convey binary data, DVDs add a third dimension by burning these pits on multiple layers. But 3D optical storage would allow us to write thousands of layers in a single disc! The new 5D optical storage technique uses something called “birefringence”, helping add another two dimensions. This method can store much more data, with each pit holding eight bits instead of one. Simply put, this would potentially allow a single disc to hold hundreds of terabytes!

• 🎯 Current Stage: Researchers at the University of Southampton have made significant progress with 5D optical storage, demonstrating a novel technique using ultrafast laser-induced nano-gratings in fused quartz. That sounds like a bunch of jargon, but they seem to know what they’re doing, as demonstrated by a working prototype. Then, there are multi-level optical discs, which are closer than ever.
• 💹 Price and Scalability: Currently, the costs and scalability of this technology are not well-defined because it is still in the research phase, and no commercial product exists. However, the prototype demonstrations show potential for extremely high capacity and long-term data retention with 5D optical storage.
• 🌩️ Challenges: Developing and perfecting the technology to achieve a reliable commercial product is still ongoing. The costs associated with production and implementation for such futuristic tech are always on the higher side, thus creating further barriers to entry.

Comparison and Potential Impact

The table below should hopefully give you a concise summary of each technology we’ve discussed above, and their potential impact on the storage industry as we know it today.

Conclusion

In the silicon age, technology changes rapidly. The futuristic technologies we’ve seen today could all just end up being vaporware. Like this xkcd comic aptly puts it, there’s no way to know for certain what “five or ten years away” actually means!

The technologies we’ve discussed are still all being actively developed, some being led by major organizations like Microsoft and others being raced to by various players in the storage industry. One thing’s for sure- when the only credible source of information and updates about some futuristic tech is an IEEE research paper, it’s probably closer to science fiction than science fact!

For all we know, they might all end up being dead ends. Or, they might actually shift the paradigm of digital data storage as we know it. Only time will tell.