Parquet data format

CONNECT data services can output your data view in Apache Parquet, an open source, column-oriented data file format designed for efficient data storage and retrieval. This binary file format allows you to easily upload CONNECT data services data into data lakes and data warehouses. Parquet is a common format for working with data in technologies like Databricks, Snowflake, Azure, AWS, and others.

Parquet data serialization

When you request a data view in Parquet format, CONNECT data services serializes the data from text-based data to the Parquet binary file format, which is not human-readable. To read the Parquet file, you will need to open it with a programming language routine or application capable of deserialization, such as Databricks, Snowflake, Azure, AWS, or another compatible application.

Parquet file download

When you make requests against a dataview in the API console with the Parquet form selected, you can download the request as a Parquet file. Select Download Parquet File to download the file.

For more information on how to request a data view in the Parquet format, see Data views and the API Console.

API Console: Download Parquet File

Handling multiple data types

When you request a data view that includes multiple data types in a single column:

1. The data view converts two or more SDS data types into a compatible SDS data type wide enough to accommodate all types without losing information. This conversion occurs to accommodate the strongly-typed data in the Parquet format.

2. The data view converts the compatible SDS data type to a Parquet data type. This conversion occurs because the Parquet format does not support all data types natively.

For more information on the conversion, see Data view data type conversion.