This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
By Mark Hosking, Head of Delivery at Datometry
The cloud migration story is not new. Years ago, many organizations issued a cloud mandate and have been diligently deploying and shifting technology ever since. For most large organizations, the cloud migration story is not over, as tough challenges with on-premises systems remain to be addressed. Migrating an enterprise data warehouse (EDW) poses one such substantial challenge.
An EDW cannot simply be lifted and shifted to the cloud. Part of a complex data ecosystem, an EDW includes ingesting, cleansing, and curating data for consumption across an organization, weaving patterns of dependencies. Migrating an EDW requires orchestration and coordination across legions of data engineers and application teams who rely on the EDW for their extract-transform-load (ETL) operations, business intelligence (BI), and analytics applications. Development of the EDW and its ecosystem usually has happened over a long period by multiple teams, working to evolving data architecture standards. After a decade or two, an honest ecosystem diagram resembles a bowl of noodles. Re-coding this ecosystem for a cloud data warehouse threatens to become an expensive quagmire.
What you need to know about successful EDW migrations
If you are looking to modernize your on-premises data warehouse to Azure Synapse Analytics, here are some top tips from successful migrations:
- Beware of vendors who tell you that rewriting ETL, BI and analytics applications can be done within a reasonable timeframe and budget. Often, this advice results in two expensive EDWs, increasing complexity and multiplying the reasons for wanting to migrate to begin with.
- Leverage database virtualization technology so the migration team can focus on data migration, performance optimization, and careful sequencing. This approach eliminates the need for complex requirement gathering, data engineering, testing, and change management, all required for a full-blown rewrite to the target database.
- Know your warehouse before the migration. Create an inventory of data objects and clean up the EDW. An EDW that has been developed over a long period probably has obsolete data and ETL applications that process data that is no longer consumed. We have encountered EDWs in which nearly 30% of data is not required.
- A divide and conquer approach is best. Stage your Azure Synapse Analytics system by performing parallel data ingestion and processing. Introduce workloads to Synapse in tranches. Bring on a portion of your overall workload, optimize it, and then bring on the next portion. Once you have your core ETL running and validated, you can decide which data consuming applications should be brought online first.
- Move fast. The on-premises EDW is a moving target, constantly changing. While operating one EDW is a challenge, synchronizing two EDWs for a long period proves nearly impossible and leads to a lot of scope creep. Scope creep explains why multi-year EDW migration projects have such a low success rate.
Moving EDW quickly to Azure Synapse while reducing time, cost, and risk
However, modernizing an EDW onto Microsoft Azure Synapse Analytics is highly rewarding. Getting into Azure Synapse Analytics unlocks a rich toolset, including advanced data integration, data analytics, and machine learning. These Azure products allow organizations to rapidly derive key insights and build the data-enabled applications to put those insights into operation, extending capabilities and reducing costs. Moving such a critical ecosystem onto Azure enables a level of flexibility in analytics that is unimaginable on-premises, making the EDW’s presence on Azure an asset.
Datometry, an IP co-sell incentivized Microsoft partner, has been working on data warehouse modernization longer than most companies have had cloud mandates. Datometry Hyper-Q database virtualization software has enabled many Fortune Global 500 organizations to migrate from Teradata to Azure Synapse, keeping their existing ETL, BI, and analytics applications without re-coding. Datometry customers have migrated their EDW platforms to Azure Synapse in a fraction of the time, at a fraction of the cost, and with a fraction of the risk when compared to a typical EDW cloud migration involving application rewrites or code conversion. In all cases, thousands of end users have seamlessly cut over to Azure Synapse, continuing to use their existing applications without disruption.
Datometry Hyper-Q performs on-the-fly query translation and feature emulation to allow current data integration and consumption applications to be used against Azure Synapse Analytics. This is all done with minimal performance overhead.
Proven expertise migrating complex data warehouses
Drawing on years of experience, Datometry knows how to make complex migrations a success and guide customers away from bad options. Two recent migrations demonstrate challenges overcome by Datometry.
A global container logistics and shipping company with 95,000 employees moved their EDW from an on-premises Teradata EDW to Azure Synapse Analytics in less than a year by using Datometry Hyper-Q. Migration required moving 165TB of compressed data, over 50,000 database objects, up to 1.5 million daily queries, and a comprehensive ecosystem for ETL, BI, and analytics. Over 5,000 users moved seamlessly to the new EDW on Azure Synapse Analytics without business disruption. Most users didn’t know that their core data platform had changed.
In another case, a United Kingdom cooperative with 65,000 employees moved their grocery business Teradata EDW to Azure Synapse Analytics in 10 months, using Datometry Hyper-Q technology. A small team was able to cut 1,000-plus users over to Azure Synapse Analytics, while realizing significant cost savings compared with the on-premises solution.
Cloud migration from on-premises EDW has historically bogged corporations down. If you want to move your organization forward in data processing and analytics, leaving your EDW on-premises and developing a new system in the cloud is no longer the best option.
If you’re interested in making it even easier to start migrating your EDW ecosystem, you can purchase Datometry Hyper-Q directly from the Microsoft Azure Marketplace. You can also learn more about Datometry’s expertise and partnership with Microsoft and see why an increasing number of Fortune Global 500 companies have already chosen Datometry to lift the weight that keeps so many EDW migrations stuck on the ground.