The risk of lost data, regulatory fines, or inaccessible formats decades down the line is forcing highly regulated industries to rethink how they store information. Cloud archive storage is no longer a nice-to-have — it’s become the most strategic and scalable long-term data storage solution available.
Industries such as the pharma, life sciences and healthcare are required to preserve documents for very long periods of time - up to 95 years for regulatory compliance. For example, drug manufacturers must comply with several international standards for records retention like the GxP standards – Good Manufacturing Processes, Good Clinical Processes, and Good Laboratory Practices.
Why Long-Term Data Retention Matters
Drug manufacturers regulated by the Food and Drug Administration (FDA CFR Title 21 Part 11) require the easy accessibility of records for agency inspection and guaranteed integrity. These records must also remain viewable for extended periods of time. Also known as archival cloud storage, this approach offers unmatched flexibility and resilience for evolving compliance and operational needs.
Additionally, electronic healthcare records for drug test subjects must be kept for the life of the test subject plus "N" years. The exact number of years depends on the nature of the trial, relevant regulations, and organizational policies (“N” depends on the situation). These retention requirements mean that some FDA records must be retained, accessible and viewable for many decades.
Healthcare insurance companies also face long records retention requirements. Standard industry practice is to keep policies and associated documentation available and searchable for as long as the policyholder is a client – which could be decades. Another example of long-term records retention appears in the UK where patient records must be kept for ten years after the patient's death.
Clearly there are many other examples of long-term retention requirements in other sectors including the federal government with the National Archives and Records Administration (NARA), mortgage providers, and insurance companies.
With today's data analytics capabilities, many organizations are retaining much of their data for ongoing analytics for longer periods than the generally accepted industry norms – except where it could run afoul of privacy regulations.
Read More: Data Has Value, but also Risk – Get Rid of What You No Longer Need
From Banker’s Boxes to Backup Tapes: Legacy Archiving Methods
In the past, the standard practice for most corporate archiving has been to either print the records and store them in physical records storage, store them electronically in on premises archives, or to back up the electronic files onto tape.
The downside of printing and storing records includes:
- Ongoing monthly payments for storage space
- Risk of loss in transit or physical damage (fire, flood, etc.)
- Difficulty locating specific records quickly across hundreds or thousands of boxes
An additional issue of long-term hard copy records storage includes the cost of recalling records for inspection and review – there is usually a cost charged by the storage company to retrieve and deliver the document boxes manually. And because physical storage is a remote and manual process, the disposition/destruction of expired records may not be done at all. In essence, the company ends up paying an unending storage charge for long-term storage – that never ends.
Many organizations have recognized the limitations of storing hard copy records, particularly the inability to use that data for machine learning or analytics. As a result, they are increasingly digitizing physical documents to integrate them into automated information management systems.
On the other hand, using backup tape for long-term archiving does have a couple of advantages. They include very low cost, a relatively small footprint (as compared to hard copy storage), and the ability to isolate data from networked systems. This technique, known in IT circles as “air gapping” or isolated storage, refers to keeping backup tapes disconnected from digital networks. As a result, the data is protected from threats like hacking, viruses, and ransomware.
The downside of tapes includes response time – finding and retrieving specific files for inspection can take minutes, hours, or even days. Tapes also prevent the use of analytics on stored data, making them less useful for modern business intelligence efforts. In addition, they require physical storage space, which can be costly. Over time, magnetic tape data can degrade and must be periodically regenerated to remain intact and accessible.
READ MORE: Most Employees are Terrible at Information Management
What Is Data Obsolescence in Long-Term Archive Storage?
A significant hurdle companies require to store electronic files for very long periods is how to store, manage, and eventually find archived data over 20, 50, 100 years, and still be able to view them. The sheer volume of electronically stored information (ESI) flowing into and out of regulated companies today has guaranteed that converting and storing paper records is no longer feasible.
As companies adopted electronic files, many turned to on-premises, server-based archiving systems to manage regulated content. However, these systems soon revealed significant drawbacks. They were costly to maintain, requiring frequent hardware and software upgrades, and they lacked the durability needed for long-term archiving.
Many vendors behind these systems were also acquired or went out of business, raising concerns about long-term support and access. Another major issue was format obsolescence — as archiving applications evolved, older files often became unreadable due to backward compatibility failures.
Proprietary clouds for long-term data storage and archiving
Some companies have turned to third-party proprietary cloud archiving solutions. With these platforms, you get the scalability of a SaaS cloud, generally adequate levels of (one-size-fits-all) security, and potentially lower cost of storage - due to cloud economies of scale.
Downsides of proprietary cloud data archives include:
- Unwanted file conversion
- Data ransoming or restricted access to your own content
- Throttled data migration speeds
- High total cost of operation, especially when trying to exit the platform
Many of these proprietary cloud archives are housed in yet another third-party data center. This creates additional risks: if issues arise with the SaaS vendor, customers could face delays or complications retrieving or moving their data.
One practice many proprietary cloud archiving vendors do not advertise is the automatic format conversion of your data as they move it into their cloud. For example, when migrating to some platforms, original file formats may be altered without notice — making it difficult or impossible to revert or transfer those files later. This lack of transparency can severely limit portability and vendor flexibility, increasing the risk of lock-in.
They do this, they say, for storage efficiency but also to make it more difficult for you to move your archived data somewhere else if dissatisfied. Some SaaS cloud archiving vendors will charge huge additional fees for you to move your data out of their cloud to another - sometimes as much as $30+/GB. Their excuse is that they need to re-convert your data back into its original format which takes time and compute power.
On the other hand, some don't charge "conversion fees" but instead throttle the outgoing data migration speed to an unrealistically slow rate. This can drag the process out over months or even years, making it so difficult that companies abandon the migration entirely and remain locked in.
READ MORE: Proprietary Cloud Archives are like Roach Motels
Long-term data storage and cloud archiving formats
Another consideration when contemplating long-term data storage and cloud archiving is that of data format. Keep in mind that a company archiving electronic data for 30, 50, or 100 years must worry about the ability to view the data decades into the future. While it may seem unlikely, in 50 years, a document saved as a 2016 Word file might not be compatible with Microsoft Office 2070. Long-term archivists must worry about the format that data is saved to ensure viewability in 100 years.
The adoption of the cloud and advanced object-based storage services provides a whole new set of opportunities to support emerging models for long-term data archiving. The cloud offers several key benefits for long-term archiving. Chief among them is its built-in scalability and redundancy. Cloud environments can also adapt quickly to changing storage needs and compliance requirements over extended timeframes.
One long-term archiving model is the ISO 14721 Open Archival Information System (OAIS) reference model initially developed by the Consultative Committee for Space Data Systems in 2005. OAIS has been adapted to work in a cloud environment but takes special considerations when being set up and is used mainly by the aerospace industry.
On the other hand, the PDF/A file format is an ISO 19005 version of the Adobe Portable Document Format (PDF) created in 2005 explicitly for use in the archiving and long-term preservation of electronic documents. The PDF/A format differs from the standard PDF format by prohibiting features unsuitable for long-term archiving - such as font linking (as opposed to font embedding) and encryption.
The PDF/A long-term data format has become the most popular file format for commercial organizations needing to store data for decades while ensuring accessibility and viewing.
Is the cloud a viable ultra-long-term data storage solution?
Most have concluded that converting and archiving paper records is no longer feasible for long-term data storage. On-premises archives can be expensive, complicated, and less secure, making them unsuitable for very long-term archiving, and third-party SaaS cloud archives can be risky due to the reasons already discussed.
A public cloud such as Microsoft's Azure is an excellent platform for very long-term archiving for the following reasons:
- Azure is not just a cloud storage system; it's a cloud computing platform that vendors leverage to build and run specialized archiving applications
- These massive cloud solution providers have the brand, reputation, and resources to be considered a safe choice for very long-term data archiving
- They're not proprietary systems
- They're infinitely scalable
- Depending on use, they are extremely low cost – especially as compared to on-premises solutions
- Their features and capabilities grow over time, taking advantage of the newest technology such as machine learning and AI
Public cloud platforms from major technology companies such as Microsoft, Amazon, and Google lend more credibility to long-term cloud storage. These providers support multiple cloud models, including Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). This range gives organizations flexibility in selecting the architecture that aligns with their goals.
Thanks to economies of scale, public cloud storage is also significantly more cost-effective than on-premises solutions. Typical pricing ranges from $0.005 to $0.05 per GB per month, making it ideal for managing large volumes of archived data over time.
In addition, the size and stability of these vendors minimize the risk of platform deprecation or acquisition — ensuring consistent access and long-term reliability.
Choosing the Right Long-Term Data Storage Strategy
The Archive360 Platform removes the issues of vendor lock-in and migration throttling while providing state of the art information management and archiving by managing your data in your company's Azure subscription, which means you own the data as well as the cloud tenancy where it's housed. With Archive360 you can store and manage your data in its native format, or in the PDF/A format meaning that if you want to move away from the platform, you can, because the data is in your Azure cloud tenancy in a format that you control.
For companies needing to archive disparate content for extended periods of time due to regulatory, legal, or business reasons, the combination of a standardized, industry file format and utilizing the Azure public cloud platform with Archive360 Platform as the information management application, will provide you the perfect long-term data storage solution. Contact Us to learn more.
