Just Shipped: Azure Data Factory template for bulk copy from database to Azure Data Explorer

If you want to copy large amounts of data from a database (SQL Server, Oracle, DB2, Google Big Query, MySQL, Netezza, ODBC, PostgreSQL, etc.) into Azure Data Explorer, leverage Azure Data Factory template for bulk copy from database to Azure Data Explorer. ADF templates are predefined Azure Data Factory pipelines that allow you to get started quickly with Data Factory and reduce development time for building data integration projects. The Bulk copy from database to Azure Data Explorer template is created using Lookup and ForEach activities. You can use the template to create many pipelines per database or table for faster copying of data.

Once you choose Bulk copy from database to Azure Data Explorer template you will be asked to fill few parameters:

• ControlTableDataset - control table indicates what data is copied from source to destination and where it will be placed in the destination. you can fill as many records as you want.
• SourceDataset – Linked service to source database.
• AzureDataExplorerTable - Azure Data Explorer dataset.

• once a pipeline is being created, define the Batch count in Command activity ForEachPartition. This parameter determines the number of pipelines that run in parallel (1..50) until the ControlTableDataset number of rows is reached.

Read more on Azure Data Factory template for bulk copy from database to Azure Data Explorer here 

 

Learn more about Azure Data Explorer (Kusto):

1. Azure Data Explorer
2. Documentation
3. Course – Basics of KQL
4. Query explorer
5. Azure Portal
6. User Voice
7. Cost Estimator

Join us to share questions, thoughts, or ideas about Azure Data Explorer (Kusto) and receive answers from the diverse and knowledgeable Azure Data Explorer community.

 

Azure Data Explorer product team

“Join the conversation on the Azure Data Explorer community”.