This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .
Welcome to Azure Data Factory’s first monthly update blog! Here we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all of our updates at What's new in Azure Data Factory - Azure Data Factory | Microsoft Docs.
We’ll also be hosting our monthly livestream on Wednesday July 20th at 1:30 PM EST! Let us know what’d you like to see and join us to see live demos and to ask us your ADF questions!
Join the livestream here.
Table of Contents
Data flow
- Assert error handling
- Editing capabilities in source projection
- Fuzzy Join option in Join Transformation
- User Defined Functions [Generally Available]
Data movement
- Parameterization natively supported in additional 4 connectors
- SAP Change Data Capture (CDC) connector [Public Preview]
Integration runtime
Monitoring
User interface
Data flow
Assert Error Handling
Error handling has now been added to sinks following an assert transformation. Assert transformations enable you to build custom rules for data quality and data validation and now, you can log rows that fail your own custom assertions!
This example shows which rows failed your assertions:
To learn more about error handling, read Assert data transformation in mapping data flow.
Editing capabilities in source projection
New UI updates have been made to source projection editing in mapping data flows. You can now update source projection column names and column types with the click of a button.
To learn more about source projection editing, read Source transformation in mapping data flow.
Fuzzy Join option in Join Transformation
Fuzzy matching with a sliding similarity score option has been added to the Join transformation in Mapping Data Flows. You can create inner and outer joins on data values that are similar rather than exact matches! The sliding scale value goes from 60% to 100%, making it easy to adjust the similarity threshold of the match.
To learn more about fuzzy joins, read Join transformation in mapping data flow.
User Defined Functions [Generally Available]
We’re excited to announce that user defined functions (UDFs) are now Generally Available. With user-defined functions, you can create customized expressions that can be reused across multiple mapping data flows. You no longer have to use the same string manipulation, math calculations, or other complex logic several times. User-defined functions will be grouped in libraries to help developers group common sets of functions.
To learn more about user defined functions, read User defined functions in mapping data flows.
Data Movement
Parameterization natively supported in 4 additional connectors
We’ve added native UI support of parameterization for the following linked services:
- Azure Data Lake Gen1
- Azure PostgreSQL
- Google AdWords
- PostgreSQL
To learn more, read Parameterize linked services - Azure Data Factory & Azure Synapse | Microsoft Docs.
SAP Change Data Capture (CDC) connector [Public Preview]
SAP change data capture (CDC) in ADF is now in public preview! The new data connector can connect to all SAP systems that support ODP and can be used with predefined templates to help streamline the integration of SAP data within Azure services.
To learn more, see Announcing the Public Preview of the SAP CDC solution in Azure Data Factory (microsoft.com).
Integration Runtime
Time-To-Live in managed VNET [Public Preview]
We’ve introduced the public preview of time-to-live (TTL) in managed virtual network. With this, users can reserve computes for the TTL period, saving time and improving efficiency.
To learn more, read Announcing public preview of Time-To-Live (TTL) in managed virtual network - Microsoft Tech Community.
Monitoring
Rerun pipeline with new parameters
You can now change pipeline parameters when re-running a pipeline from the Monitoring page without having to return to the pipeline editor. After running a pipeline with new parameters, you can easily monitor the new run against the old ones without having to toggle between pages.
Note that by re-running a pipeline with new parameters, it will be considered a new pipeline run and will not show under re-run groupings.
To learn more about rerunning pipelines with new parameters, read Rerun pipelines and activities.
User Interface (UI)
Studio preview experience
If you are interested in trying new UI updates, then the Azure Data Factory studio preview experience is for you! When you enable the studio preview experience, you’ll be able to test UI changes in the ADF pipeline and mapping data flow experience and provide feedback to the team.
To learn more about the experience, read Managing Azure Data Factory Studio preview updates - Azure Data Factory | Microsoft Docs.
Let us know what your favorites are in the comments below!