Event Driven Architecture with Custom Event Trigger and Advanced Filtering

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Event-driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption, and reaction to events. Data integration scenarios often require Azure Data Factory customers to trigger ETL or ELT pipelines when certain events occur.


Azure Data Factory native integration with Azure Event Grid now covers custom topics: the application and third-party topics. Custom topic, created by the event publisher, provides an endpoint where source sends events. Azure Data Factory subscribes to the topic and triggers a pipeline in response to incoming events.


Additionally, Azure Data Factory now supports Advanced Filtering that gives users fine grain control over which events to respond to based on the values in the event payload. For instance, you can express logic such as, a ETL pipeline should run if and only if data.date field in list [1, 2, 3, 4, 5], data.month field not in list [11, 12], data.tag field contains any of ['Fiscal Year 2021', 'FiscalYear2021', 'FY2021'].




You can also implement Metadata Driven + Event Driven architecture, by passing event payload values to pipeline parameters, using prefix @triggerBody().event.data.<key_name>, and consumes these values in your pipeline definition. For information, please refer to Retrieve and Reference Trigger Metadata in Pipeline.



To create a Custom Event Trigger, event producer needs to define a custom topic in Event Grid (see Azure Event Grid Custom Event Topic tutorials). And by creating a Custom Event Trigger, Azure Data Factory will create a subscription against the aforementioned topic.


Click + New Trigger, specify the Event Grid Topic Name, Subject Begins WithSubject Ends With, and Event Types



Once published and activated, you can double check successful creation by browsing to the aforementioned event topic in Azure Portal and see a new subscription created by Azure Data Factory.

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.