Planning for email archive migration pitfalls and avoiding problems with API migration providers like Quadrotech

With billions of dollars spent on migration every year with API migration providers like Quadrotech, and the average migration platform costing at least a million dollars, mistakes during a move to the cloud are never welcomed. However, even in an established marketplace fueled by businesses wishing to move their data to more cost-effective and flexible infrastructure, there are often issues.

Whether it's a long, drawn-out file transfers impacting the company’s network, compliance concerns, or unforeseen costs pushing budgets in the wrong direction, cloud migration is as stressful as any major move, especially as there’s a business and its continuity at stake. And when you’re surrounded by problems, rising prices and stretched resources, it can be easy to forget all the benefits and opportunities of the cloud that moved you to migrate in the first place. So why can’t cloud migration be as simple as you might expect: fast, non-disruptive, cost-effective, and compliant?

The fact is, it should be, and our goal with this article is to flag the most common issues you might encounter and prepare you for them, while also highlighting our own approach to smooth and transparent cloud migration. Potential problems shouldn’t get in the way of a decision to move to the cloud, and proper planning will help to ensure that your move is as painless as possible, enabling you to quickly embrace an essential platform for businesses large and small. Planning your cloud migration strategy and asking questions of your provider before you begin, could be the single most important IT task you undertake.

Not only will your planning provide you with a clear picture of timing and cost, but it will also prepare you for the most common problems faced during migration and help you understand the processes of your provider. As you begin looking into migration, you’ll no doubt find a number of providers, including Quadrotech, that offer cloud migration services. These API migration vendors, on the whole, will do exactly that for you. However, what you need to determine is whether they will move your data in a way and at a cost that suits your needs and whether their techniques match your requirements in terms of timing and impact on your day-to-day business.

Preparing your archives for the cloud with API migration solutions like Quadrotech

When it comes to working with API migration providers like Quadrotech, the more you understand about your own archives, the better your outcome will be. Because the indexes and metadata associated with archives slowly degrade over time, the chances are yours are not in pristine condition. Even the slightest corruption can have knock-on effects, including missing data and a lack of compliance down the line. Ultimately, unless you can guarantee archive perfection, the standard manner in which API solutions like Quadrotech transfer your data could mean you end up with less or more of it by the time you’ve completed your migration.

Problems like this stem from the use of the source archive’s API layer to access data. Most archive migration providers work in this way, which means they’re reliant on the index within the archive to understand what it contains. Therefore, if it’s inaccurate or damaged, so is the data you’re migrating. Duplicate files and corrupt data also add to the size of a migration, which is something to be avoided. This could not only increase the cost of your migration but extend the amount of time it takes.

And when you’re transferring multiple tera or petabytes of data, the impact on your network over a long period of time can become a genuine issue with standard migration providers, sometimes even leading to a broken archive which halts the migration altogether. You also need to consider how your current archives operate and what you intend to transfer. Are you making use of Exchange Journaling?

Or are you working with another vendor such as Mimecast, Enterprise Vault or SourceOne? It’s important to understand your chosen migration provider’s level of competency when it comes to the systems you currently use and definitely worth asking if they can show you examples of previous migrations.

Typical problems with traditional cloud migration:

• Network and security issues
• Hidden database flaws
• High volumes of inactive mailboxes
• Limitations in Office 365
• Attachment and metadata loss from stub rehydration
• Issues when extracting data via the archive’s API
• Duplicate files
• Strained networks and slow data transfer

When you’re talking to an API migration provider like Quadrotech, these are the issues you should discuss straight away to ensure everyone is clear on the process and protocols.

Ask Quadrotech:

1. How long they estimate the migration will take
2. How they access and transfer your data to the cloud
3. If they can accurately predict the amount of cloud infrastructure you will need before you need it
4. Whether they can provision the migration in the cloud rather than use your network

These simple questions should help you to avoid many of the most common migration problems as well as providing you with a clear picture of the road ahead.

Are API based archive migration solutions like Quadrotech causing more problems than they’re solving?

The challenges facing a migration are clear. So, now you know about them in a little more detail, it’s important to be aware of how a cloud migration provider like Quadrotech might approach them. As we mentioned previously, most providers use standard methods to extract and migrate your data, which is where many of these problems stem from. They’re then forced to use less than efficient methods to overcome them.

1. Slow data discovery and extraction

API migration companies will probably request access to your network for the duration of your migration. As a result, you could see significant network speed dips as well as time wasted integrating new tools and users on your system. The reason this is required is so that the provider’s software can run without security restrictions and seek out the data you want to migrate. However, this doesn’t have to be the case and is more to do with the way the provider is set up. Because providers like Quadrotech access data via the archive’s API, they are limited by the speed of the system, which means longer transfer times and less available bandwidth for your network.

2. Painfully slow transfers

Another standard claim from API migration vendors is that the storage volume is what causes a bottleneck and slows down the speed of data transfer. More often than not, however, it’s the use of inefficient processes on the part of the provider themselves. Let’s say there’s a single email in your archive that was sent to ten people. It exists as a single email but, when extracted by the provider, a duplicate is created for each mailbox associated with it. It now becomes ten emails. If the original message was one megabyte, that’s now ten megabytes that need to be moved to the cloud. Scale that up to your entire archive, and you can easily jump from 50 to 500 terabytes in no time at all. As we’ve already stated, moving more data than you need to is not ideal; it slows down the overall migration, and it can slow down your network too.

3. Splitting archive and mailbox migration

According to some, archive migration takes longer than mailbox migration and the migration of an archive affects the moving of mailboxes into Office 365. This is why API migration solutions often delay the mailbox migration or perform it in stages to allow the archive migration to catch up. Working in this way severely slows down a migration, with batches of users migrated each week. This is further slowed by bandwidth limitations and Office 365 throttling before the archive migration even begins.

4. Conversions and compliance

Some API migration providers claim that data fidelity and chain of custody are maintained when using EMS (Exchange Messaging Service) and transferring data into Office 365. This is rarely ever the case and could lead to serious problems beyond your migration if you don’t take steps to address the issue. Using the EML file format – a format created by software like Outlook and Mail – creates a processing overhead and leads to a loss of data fidelity due to the conversion that takes place. This also means the activity takes longer, and your adherence to regulations could be jeopardized. Without asking about this ahead of working with a provider, you might find yourself in a situation where your legal and compliance teams are forced to retroactively deal with data loss rather than be aware of the problem before it occurs.

Ask your migration provider:

1. If ongoing network access is required for the duration of the migration
2. Whether the migration could impact your network bandwidth
3. If they replicate data and explode volumes that could cause a transfer bottleneck
4. If users will be migrated intermittently before the archive is migrated?
5. Whether they use EML files and can guarantee chain of custody and data fidelity

An efficient and predictable email archive migration approach

So far, we’ve focused on the negatives of migration, but now we’ll turn to a more positive outlook and an alternative approach to the traditional, issue-strewn methods. As we outlined earlier, it’s clear most of the problems that can arise before, during, and after a migration are caused by the provider’s reliance on the archive’s API layer to extract data. It’s for that reason that Archive360’s email archive migration software bypasses the API entirely in order to avoid any unexpected problems that can slow down a migration and increase its cost.

By taking a forklift approach to the data transfer, extracting it directly from the archive’s data layer, Archive360 is able to move an exact copy of your archived information to the cloud without making any changes to it. This method means transfer speeds and the duration of your migration are improved, as well as providing the peace of mind that no data will be left behind because there’s no reliance on a potentially damaged index.

Not only does this provide you with a clear picture of your archived data from the outset, it removes the need for any work to be done on your data before migration. This way, you know precisely what’s being transferred and know that the amount of data won’t increase before transfer. Put simply, if your archive is 50 terabytes, we only move 50 terabytes to the cloud and index and classify the data once it’s there.

The way we carry out a migration depends on the customer’s existing environment:

Organizations with an existing Azure tenancy

Because we’re a certified Microsoft Cloud Solution Provider, we can set up an Azure account for our customers.

Are asked to establish an Azure storage account or allow us to do so for them, and provide us access to it.

• We provision the infrastructure, including an Azure SQL Service.
• We implement an Azure SQL Service within Azure.
• Data is forklifted from on-premises archives to the Azure storage account.
• Data is forklifted from on-premises archives to the Azure storage account.
• The Azure SQL Service is used to reconstitute source SQL databases within the forklifted data for migration into the cloud archive or Office 365.
• The Azure SQL Service is used to reconstitute source SQL databases within the forklifted data for migration into the cloud archive or Office 365.

Start faster

By avoiding the archive’s API, Archive360 can get straight to the source of your data to reduce disruption and shorten migration time. Without API speed restrictions or issues caused by damaged databases and indexes, we can ensure we move all of your data as quickly and efficiently as possible.

What’s more, we only require read-only access to your archive, and our FastCollect solution is capable of migrating over 80 types of data, whether it’s structured, semi-structured, or unstructured. No software is installed on your archive servers, which means you won’t break your existing archiving software support and maintenance agreement, and you can be set up and running within an hour, with minimal impact on your network bandwidth.

With a single, sequential read across all of your raw storage, FastCollect can extract 5-10 terabytes of data per day, logging all data collections in the process. This information can then be included in a chain of custody report for compliance purposes, and it also guarantees data encryption. A data discovery report will also be generated in order to confirm exactly how much data will be transferred.

While API migration solutions like Quadrotech may need to sit on your network and behind your firewall for extended periods, Archive360’s efficiency significantly shortens the initial stages of migration. A faster time to transfer means you can begin decommissioning your legacy systems sooner and begin shutting down on-premises infrastructure once discovery is complete.

Maintain momentum

For the smoothest and most accurate transfer of data, Archive360 avoids carrying out any data processing on your legacy infrastructure. Rather than face bottlenecks and bandwidth issues, we move your data directly to the cloud, so there are no surprises as to the amount of data you’re migrating. We also ensure you retain ownership of your data and its encryption keys when it is moved from the cloud to Office 365. Working this way means that there’s no decompression, expansion, or any other manipulation of your data, so you can be sure that whatever exists in your archive will be moved to the cloud.

This protects the native format of your data as well, with data compressed by your archive transferred in that format. Raw C-Clips from EMC Centera storage are transferred in the same way. No changes, no wrappers, and no proprietary file formats.

We believe it’s essential for your data to remain in its native format throughout the migration.Not only does it make your life easier in the long run, but it also gives you the freedom to unlock the power of your archive data once it’s in the cloud. Whether it’s more detailed eDiscovery searches or using cutting-edge analysis tools for legal and HR tasks, keeping your data in its native format provides you with complete flexibility. It also avoids having to pay to convert files back from a proprietary format, which would be the case for a number of providers.

Only when your data reaches the cloud staging phase is the SQL metadata processed and the raw data validated. At this point, we also generate an Archive Insight Report that outlines the total number of messages and the size of the user archive, as well as the total amount and size of items segregated per archived mailbox. This information is crucial, as it not only clearly shows the type and amount of data you are moving to Office 365, but determines how many inactive users exist in your archive. On average, up to 60 percent of total users within a system are inactive, so understanding this number will help avoid issues and reduce cost before you begin moving data to Office 365.

Limiting network bandwidth impact

Archive360 provides options to ease the burden on your network, including:

1. Controlled ongoing transfers with limits to the amount of data transferred per second or per hour.
2. A scheduled approach to only transfer data at night, on weekends, and other low traffic hours.

Migrating to Office365 in batches

Some providers will spend a long time moving mailboxes to the cloud intermittently before migrating your archive to the cloud. Archive360 avoids such a slow process by carrying out a staged migration. Thanks to the fact that we move a copy of all of your data to the cloud before active user migration begins, access to your network is no longer required. This means your IT team can take advantage of its full network bandwidth to migrate mailboxes at a far higher speed. Our migration consultants will work with you on this, establishing a migration plan that’s best for your business and executing it on your behalf. We’ll also validate each item during this process, to make sure there’s no corruption anywhere in your migrated data.

The following reports are made available to you after each batch run:

• User Migration Report including: Username | Display name | SMTP | archive msg count | Office 365 msg count | validation alert count
• Rehydration Report including: Username | Display name | SMTP | Rehydration count | Rehydration Size

Final migration and chain of custody

EML files are often used by migration providers to save email data. However, Archive360 ensures your data remains in its native format by using the MAPI protocol to push data into Office 365. This avoids the usual level of throttling from Office 365 but also, crucially, prevents any conversion of your data that could cause a loss of data fidelity. When your migration is complete, we provide you with a final migration report, within which is a report on Legal Chain of Custody. This document enables you to account for the movement of every message from your archive to Office 365 and is crucial for legal defensibility.

Take a modern approach to archive migration

Having carried out thousands of successful migrations, Archive360 knows the best way to move your data to the cloud with maximum accuracy, complete compliance, and minimal impact on your network. We also understand that the idea of a cloud migration can be a daunting once, which is why we guide our customers through the process and help them avoid the pitfalls that many face when migrating with traditional providers. And we’re available to help beyond migration too. While moving your data to the cloud is a good first step, Archive360 can help to enable your business through its data once the migration is complete. We can help you to reduce cost by optimizing the storage of your data, use powerful tools to analyze information and unlock key insights, and enhance your security and regulatory policies. We can help you make the most of efficient litigation support services and enjoy the same if not better levels of compliance as you experienced on-premises, all while benefiting from the flexibility of the cloud.

If you’re planning a move to the cloud, why not talk to one of our representatives about Archive360’s modern approach to migration?

Archive 360 is trusted by companies like 3M, Netflix, McKesson, Samsung to migrate from Enterprise Vault and other major archiving platforms.