How to change Ingestion batching Policy from 5 Minutes to 5 Seconds?

This post has been republished via RSS; it originally appeared at: Azure Data Explorer articles.

Azure Data Explorer has an aggregation (batching) policy for data ingestion, designed to optimize the ingestion process. The policy is configured to 5 minutes or 500 MB of data, by default. in case you would like to change ingestion batching policy (Ingestion that is done in blobs that hold much less data than the optimal size is non-optimal, and therefore in queued ingestion ADX will batch such small blobs together) , See Ingestion batching policy for aggregation options.

 

//reduce ingestion time table to 5 seconds 
.alter table myData policy ingestionbatching '{"MaximumBatchingTimeSpan": "00:00:05"}'

 

Event Hub ingestion includes Event Hub response time of 10 seconds or 1 MB for non-streaming ingestion.

 

Streaming ingestion reduce batching policy and Event Hub response to seconds (depends on data format, data size, etc.). Use streaming ingestion policy to Configure your table to support streaming and remove the lag in response time. 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.