What You Need to Know About Data Migration

July 6th, 2016
Vadim Vladimirskiy
Vadim VladimirskiyFounder & CEO, Nerdio

If you’ve ever misplaced your phone, keys, glasses or wallet, you know what it feels like to absentmindedly move something important, then lose track of it entirely. You know all too well the sensation of panic swelling in your gut as you rifle through your home for evidence of the lost item’s whereabouts. You’re acquainted with the self-deprecating insults you belch—stupid, idiot, moron—as you attempt to retrace your steps. You know personally the grandiose life changes you promise supreme beings you’ll make in exchange for your belongings’ safe return. And you’re intimately familiar with the gleeful feeling of relief that sweeps over you when you find what you lost in the last place you thought to look.

Now, imagine that what you misplaced wasn’t your phone, your keys, your wallet or your glasses, but rather the mission-critical data your company relies on to execute its operations, monitor its health, serve its customers and transact with its vendors. Forget panic. Suddenly, you’re experiencing the IT equivalent of DEFCON 1.

Unfortunately, it happens more often than you think. Thanks to the big data trend, companies are collecting and storing more data than ever before. To exploit that data more effectively and efficiently, many of them are engaged in complex data migration projects designed to replace and consolidate outdated legacy systems, integrate business intelligence across silos and otherwise streamline internal operations. Sometimes, things go wrong, resulting in missing, inaccurate or incomplete data.

In fact, a 2015 study by Experian Data Quality (EDQ) found that 85 percent of businesses have experienced one or more data migration challenges—including poor data quality, a problem experienced by nearly a third (31 percent) of businesses engaged in data migration projects. Other common challenges, according to EDQ, include lack of collaboration (38 percent), lack of standardization (37 percent), poor system design (33 percent) and poor interpretation of business rules (24 percent), resulting in data migration projects that leave companies vulnerable to risks such as cost and schedule overruns, system outages and even revenue loss.

Lost wallet

A solid data migration plan will ensure that risks are averted, data is protected and, ultimately, value is realized. Here are eight things every plan should include:

1. A clear scope of work

Without a clear scope of work that’s defined up front, data migration projects can veer off course. For that reason, you should identify as early as possible what data and how much data you want to move from your old system to your new one. Because while ideally you’d like everything in your old system magically to appear in the new one, the fact is: Your limited time and resources mean you must prioritize.

2. An appropriate strategy

To successfully execute the technical aspects of a data migration, IT teams need to agree on a migration strategy and approach.

The two principal types of migration are big bang migrations, which involve executing the entire migration in a small, defined processing window (for example, a weekend), and trickle migrations, which take an incremental approach. Although the former is attractive for its speed, the latter typically is more advantageous because it eliminates downtime and maximizes quality.

In addition to a general approach, you’ll need to choose a specific migration method. Migration teams can choose from several data migration methods, each of which has unique pros and cons. Options to explore further include:

    • Array-based migrations, which use storage arrays to move data instead of servers;
    • Virtualized migrations, which move data using virtualization tools;
    • Fabric migrations, which move data through the storage fabric (SAN) itself; and
    • Host-based migrations, which leverage host-based software running on a server.

3. Clean data

The old IT adage is true: Garbage in, garbage out. All data has problems. If you move those problems (for example, invalid dates, missing codes, incorrectly captured information, duplicate fields) from the old system to the new one, you’ve wasted a valuable opportunity to improve not only the data itself, but also the business insights that can be gleaned from it.

To ensure the data you move can be leveraged to maximum effect in your new system, ask subject matter experts to review it prior to migration. With their input, you should correct inaccurate data, purge unnecessary data and merge duplicate data.

Keep in mind your old system will continue to be updated with new data until it is no longer being used, so data must be cleaned up continuously throughout the migration process.

4. A comprehensive communication plan

Although data migration is a technical exercise, “soft skills” such as communication are critical, ensuring potential problems are identified and solved as quickly as possible.

To ensure frequent communication, schedule periodic project meetings with internal as well as external stakeholders to assess the project status.

Keep in mind that data migrations often involve a mix of internal employees and external consultants, many of whom may be located across different countries and have experience using different technologies. Therefore, collaboration tools that use videoconferencing and screen sharing are helpful, ensuring mutual understanding of requirements and progress by all stakeholders and team members.


5. Well-defined business rules

It’s important to involve the business in data migration projects, particularly when it comes to business rules that might impact data integrity, mapping, utility and access. For example, the migration team should understand what definitions the business uses and which areas of the business should have access to which types of data. Even something as simple as what the company means by “customer”—who counts as a customer and who doesn’t—should be documented, agreed upon and applied to the data.

6. Clear data migration policies

To minimize impacts on the business, migration teams must have clear policies for when and how migration will take place. For example, policies often dictate that data migrations be restricted to evenings and weekends when employees typically don’t need access to the network.

7. Dedicated resources

Although it’s tempting to borrow temporary resources from existing departments, keep in mind that projects lacking sufficient resources are doomed from the start. Therefore, to minimize risks, make sure you have dedicated data-migration personnel. Keep in mind that data migration typically requires the same technologies and skills as areas such as data integration, consolidation, quality, system upgrades and enterprise data architecture. So you can realize efficiencies by addressing all of them within a shared organizational structure.

If permanent resources aren’t an option, consider hiring a contractor or consultant to manage and execute the project. This gives you all the benefits of a dedicated team without having to siphon off resources from elsewhere.

8. Strong testing procedures

Imagine buying a new couch, then discovering when it’s delivered that it doesn’t fit through your front door. The same thing can happen with data. Therefore, testing is one of the most important steps in the migration process.

To test successfully, start by migrating small batches of data and determining how the new system handles it. Then, gradually increase your sample size from there. This ensures bugs are discovered—and corrected—as early as possible.

Post-migration validation is equally important, as you need to make sure migrated data is correct once you’ve moved it. For example, check data volume: If you moved 100 records from the old system, make sure there are 100 records in the new system. Likewise, make sure migrated data is mapped to the correct fields. Do this type of validation after every load and again at the end of the project to make certain old data made it to the new system.

From proper resourcing to thorough testing, it all comes down to thorough planning and good governance. With both, your company won’t be bemoaning lost data after a migration; instead, it will be celebrating newfound business insights.