How to ingest historical data into ADX

This post has been republished via RSS; it originally appeared at: Azure Data Explorer Blog articles.

LightIngest is a command-line utility for ad-hoc data ingestion into Azure Data Explorer. To learn more about LightIngest, see Use LightIngest to ingest data into Azure Data Explorer.

LightIngest can be particularly useful to load historical data from an existing storage system to Azure Data Explorer. While you can build your own command using the list of Command-line arguments, this article shows you how to auto-generate this command through an ingestion wizard. In addition to creating the command, you can use this process to create a new table, and create schema mapping. This tool infers schema mapping from your data set.

This article shows you how to create a new table, create schema mapping, and generate a LightIngest command for one-time ingestion using the LightIngest tool.

 

 

How to Ingest more than 5000 files/blobs to Azure Data Explorer 

 

  1. Open Web Explorer 
  2. Select ingest new data or Ingest new data from container at Data Page 
  3. Select Cluster, database and table (new or existing) 
  4. Source tab - for more than 5,000 blobs select Historical data 

source-tab-lightingest.png

  1. Select a source or enter SAS URL to source 

 

source-tab-container-from-subscription.png

  1. Select "Advanced parameters" to support: 
    1. Creation Time 
    2. blob name pattern 
    3. Tag
    4. limited amount of files 
    5. Don't wait for the ingestion to be completed (recommended for big amount of data )
    6. 1.PNG3.PNG
  2. Review and adjust the auto-suggested schema structure 
  3. In the Data ingestion completed window, all three steps will be marked with green check marks.

    Copy the generated LightIngest command by clicking on the copy icon to the top right of the command box.

    summary-tab-copy-command.png
  4. To complete the ingestion process, you must run LightIngest using this copied command.

read more on generate lightingest command 

Recommended reading 

  1. "How to Ingest Container with less than 5000"  
  2. Azure Data Explorer Ingestion Overview 
  3. Azure Data Explorer Ingestion 

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.