Read and Write Complex Data Types in ADF

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

ADF has connectors for Parquet, Avro, and ORC data lake file formats. However, datasets used by Copy Activity do not currently have support for those types. Here is how to read and write those complex columns in ADF by using data flows.

 

There is a description of this technique in each file format documentation page in the ADF online docs:

 

https://docs.microsoft.com/en-us/azure/data-factory/format-orc#dataset-properties

https://docs.microsoft.com/en-us/azure/data-factory/format-parquet#data-type-support

https://docs.microsoft.com/en-us/azure/data-factory/format-avro#data-flows 

 

Step 1: Make a new dataset and choose the file format type. In this example, I am using Parquet. Set NONE for schema:

complex1.png

Step 2: Make a data flow with this new dataset as the source:

complex2.png

Step 3: Go to Projection -> Import Projection

complex4.png

Step 4: You’ll see your data under Data Preview

complex3.png

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.