This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.
ADF has connectors for Parquet, Avro, and ORC data lake file formats. However, datasets used by Copy Activity do not currently have support for those types. Here is how to read and write those complex columns in ADF by using data flows.
There is a description of this technique in each file format documentation page in the ADF online docs:
https://docs.microsoft.com/en-us/azure/data-factory/format-orc#dataset-properties
https://docs.microsoft.com/en-us/azure/data-factory/format-parquet#data-type-support
https://docs.microsoft.com/en-us/azure/data-factory/format-avro#data-flows
Step 1: Make a new dataset and choose the file format type. In this example, I am using Parquet. Set NONE for schema:
Step 2: Make a data flow with this new dataset as the source:
Step 3: Go to Projection -> Import Projection
Step 4: You’ll see your data under Data Preview
