Azure Data Factory September 2022 Monthly Update

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Welcome to Azure Data Factory’s September monthly update! Here, we’ll share the latest updates on what’s new in Azure Data Factory. You can also find all our updates on our What's New in ADF page

 

We’ll also be hosting our monthly livestream next week on October 11th at 11:30am PST/ 2:30pm EST! Join us to see some live demos and to ask us your ADF questions!

 

Join the livestream here.

 

 

Table of Contents

 

Data flow

Data movement

Monitoring

  • Additional monitoring improvements in Azure Data Factory [Blog]
    • Additional sorting columns in monitoring – sorting added for Pipeline name, Run End, and Status
    • Monitoring consumption report improvement – loading icon added to know when consumption report is fully calculated
    • Monitoring loading improvements - pipeline re-run groupings data fetched only when expanded
    • Pagination added to pipeline activity runs view to show all activity records in pipeline run
    • Time-zone settings now saved in monitoring
  • Gantt chart view now supported in IR monitoring [Doc]

Orchestration

User interface

 

Data flow

Amazon S3 source connector added

We’ve added a new source connector to mapping data flows! Users can now read their data from Amazon Simple Storage Service (S3). Note: This connector is only available for the Dataset Source Type.

 

Noelle_Li_0-1664846213600.png

 

To read more information on this new source connector, read Copy and transform data in Amazon Simple Storage Service (S3) - Azure Data Factory & Azure Synapse | Microsoft Learn.

 

Google Sheets REST-based connector added (Preview)

We’ve added a new REST-based connector to mapping data flows! Users can now read their data from Google Sheets. Note: This connector is only available when using inline datasets.

 

Noelle_Li_1-1664846213603.png

 

To read more information on the Google Sheets connector (Preview), read Transform data in Google Sheets (Preview) - Azure Data Factory & Azure Synapse | Microsoft Learn.

 

Maximum column optimization in mapping dataflow 

For CSV data sources, we’ve added a new setting that allows you to set the maximum number of columns. The default is 20480 so if you have more than 20480 columns, you can customize this value and optimize it for your dataset.  

 

Noelle_Li_2-1664846213610.png

 

To learn more about this setting, refer to this document, Mapping data flows - delimited text format in Azure Data Factory

 

SAP Change Data Capture capabilities in Mapping Data Flow (Preview) - Extract and transform data changes from SAP systems for a more efficient data refresh

We made an exciting step in our SAP CDC roadmap! The SAP CDC connector is now available in ADF mapping data flows, providing a simple, integrated experience to extract and transform the raw change data coming from SAP.

 

Noelle_Li_3-1664846213613.png

 

To read more information about these new capabilities, read Transform data from an SAP ODP source with the SAP CDC connector in Azure Data Factory or Azure Synapse Analytics - Azure Data Factory & Azure Synapse | Microsoft Learn

 

Writing data to a lookup field via alternative keys supported in Dynamics 365/CRM connectors for mapping data flows

Now, you can write data to a lookup field when using Dynamics 365/CRM connectors in mapping dataflows. With alternate keys, you can define an attribute in your entity to correspond to a unique identifier, which can be used to uniquely identify a record in place of the primary key. In mapping data flows, writing data to a lookup field via alternative keys is quick and simple – just update your sink mappings! Note: This is currently supported only when you use an inline dataset for your mapping data flow sink transformation.  

 

 

Noelle_Li_4-1664846213614.png

 

To learn more about this update, read Copy and transform data in Dynamics 365 (Microsoft Dataverse) or Dynamics CRM - Azure Data Factory & Azure Synapse | Microsoft Learn.

 

Data movement

Support to convert Oracle NUMBER type to corresponding integer in source

We have added new support to convert Oracle NUMBER type to corresponding integer in source. You will see a new property convertDecimalToInteger, which will allow you to convert Oracle NUMBER type from zero or an unspecified scale to the corresponding integer.  

 

To learn more about this update, read Copy data to and from Oracle by using Azure Data Factory or Azure Synapse Analytics.

 

Monitoring

Additional monitoring improvements in Azure Data Factory

We’ve released a new bundle of improvements to the monitoring experience based on community feedback. These improvements aim to provide a smoother and more comprehensive monitoring experience.  

 

Additional sorting columns in monitoring – sorting added for Pipeline name, Run End, and Status

We now have more columns available for sorting. Previously, you could only sort by Run start. Now, you can sort by Pipeline Name, Run End, and Status in addition to the Trigger Name and Status columns.

 

Noelle_Li_5-1664846213621.png

 

Monitoring consumption report improvement – loading icon added to know when consumption report is fully calculated

The pipeline consumption report now shows a loading icon when there are many activities in a pipeline to let you know when the consumption information has been fully calculated.  

 

Noelle_Li_0-1664847951460.png

 

Monitoring loading improvements - pipeline re-run groupings data fetched only when expanded

Previously, when pipelines were re-run, all re-runs were retrieved for the monitoring UI, even when the re-runs were not viewed. Now, pipeline groupings for re-runs will only fetch data when the re-run grouping is expanded. This will improve the performance of monitoring pages that have multiple pipeline re-run groupings. 

Noelle_Li_0-1664847734951.png

 

Pagination added to pipeline activity runs view to show all activity records in pipeline run

Previously, in pipelines with many activities, the list of records might have timed-out or returned an incomplete list of records. Now, when monitoring pipeline runs, all records for pipelines will be retrieved and can be viewed in different pages to help you navigate through all the activity records. 

 

Noelle_Li_1-1664847777442.png

 

 

Time-zone settings now saved in monitoring

Monitoring time zone settings are now saved in your browser cache with your configured columns so that you don’t have to reconfigure your settings each time!

 

Noelle_Li_9-1664846213641.png

 

To learn more about these monitoring updates, read Azure Data Factory monitoring improvements (microsoft.com).

 

Gantt chart view now supported in IR monitoring

You now have the option to view your activity runs with a Gantt chart in Integration Runtime monitoring. A Gantt chart is a view that allows you to see the run history over a time range. This will give you additional monitoring views of your integration runtimes.  

 

Noelle_Li_10-1664846213642.png

 

 

For more information, read Monitor integration runtime in Azure Data Factory - Azure Data Factory | Microsoft Learn. 

 

Orchestration

DELETE method in the Web activity now supports sending a body with HTTP request

We have added support for sending a body (optional) when using the DELETE method in Web activity. This is useful in the case where your web endpoint expects a body to be passed with the HTTP request DELETE method.

 

Noelle_Li_11-1664846213644.png

 

To learn more about this update, read Web Activity - Azure Data Factory & Azure Synapse | Microsoft Learn.

 

User interface

Native UI support of parameterization added for 6 additional linked services – SAP ODP, ODBC, Microsoft Access, Informix, Snowflake, and DB2

We have added native UI support of parameterization for 6 additional linked services! These linked services include SAP ODP, ODBC, Microsoft Access, Informix, Snowflake, and DB2. This provides a built-in parameterization experience with parameter options and dynamic content support when authoring linked services.

 

For more information, read Parameterize linked services - Azure Data Factory & Azure Synapse | Microsoft Learn.

 

Pipeline designer enhancements added to Default studio experience from Preview experience

We are moving some of our exciting preview UI updates from Preview Experience to the default UI designer!

 

You will see a new floating toolbox when adding activities to your pipeline. This will allow you to add the next activity in your workflow sequence without needing to drag in an activity from the mounted toolbox.

 

Noelle_Li_12-1664846213649.png

 

In addition to this update to adding pipeline activities, you will also see activity connectors on the right-hand side of the pipeline activity, allowing you to easily choose what kind of activity connector to add.

 

Noelle_Li_13-1664846213649.png

 

Lastly, you will have the ability to view iteration and conditional activities with the container view. This allows you to adjust your activity size to be able to see the activities contained within your iteration and conditional activities.

 

Noelle_Li_14-1664846213693.png

 

For more information on the UI updates, read Azure Data Factory Updated Pipeline Designer (microsoft.com).

 

 

We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in these blogs or livestreams. We love hearing your feedback!

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.