Description
Vortex is an extensible columnar file format for compressed Apache Arrow-compatible data, designed for fast scans and random access. ClickHouse supports reading and writing Vortex files.Data types matching
The table below shows the Vortex data types and the corresponding ClickHouse data types inINSERT and SELECT queries.
Other types are not supported. In particular, Map,
Int128/UInt128/Int256/UInt256, IPv6
and Interval columns cannot be written to Vortex files.
String columns are written as
Binary because ClickHouse strings are
arbitrary byte sequences, while Vortex requires Utf8 values to be valid UTF-8. Vortex has no
fixed-size binary type, so FixedString is also written as Binary;
schema inference reads it back as String.
LowCardinality columns are written as their underlying type
(Vortex chooses dictionary and other encodings adaptively by itself).
DateTime columns are written as vortex.timestamp with second precision,
so they are read back as DateTime64 with scale 0.
IPv4 columns are written as U32 because Vortex has no type for IP addresses,
so schema inference reads them back as UInt32. Specify the type explicitly
to read such a column back as IPv4: SELECT * FROM file('data.vortex', Vortex, 'ip IPv4').
The data types of ClickHouse table columns do not have to match the corresponding Vortex data fields.
When inserting data, ClickHouse interprets data types according to the table above and then
casts the data to the data type set for the
ClickHouse table column.
Example usage
You can select data from a Vortex file:Format settings
As in other columnar formats, only the columns used by the query are read from the file, and
columns missing in the file are filled with default values.
Performance
A file is read in parallel. The scan splits it into row ranges (aligned to the chunk boundaries of the requested columns, at most 100 000 rows each) that are read, filtered and decoded concurrently: the decoding runs on up tomax_parsing_threads threads of the same pool the Parquet reader uses,
the reads on up to max_download_threads threads, and the conversion of a decoded chunk to
ClickHouse columns happens on the thread that decoded it. Chunks are returned as soon as they are
ready, so the row order is not guaranteed unless input_format_vortex_preserve_order is set. The
reads of segments that are close in the file are merged into one request (up to 4 MiB for local
files and 16 MiB for remote storage). Filter pushdown reduces the amount of data read and decoded
by selective queries.