> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-vortex-format.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Documentation for the Vortex format

# Vortex

| Input | Output | Alias |
| ----- | ------ | ----- |
| ✔     | ✔      |       |

<h2 id="description">
  Description
</h2>

[Vortex](https://vortex.dev/) is an extensible columnar file format for compressed Apache Arrow-compatible data,
designed for fast scans and random access. ClickHouse supports reading and writing Vortex files.

<h2 id="data-types-matching-vortex">
  Data types matching
</h2>

The table below shows the Vortex data types and the corresponding ClickHouse [data types](/reference/data-types/index)
in `INSERT` and `SELECT` queries.

| Vortex data type (`INSERT`) | ClickHouse data type                                                                      | Vortex data type (`SELECT`) |
| --------------------------- | ----------------------------------------------------------------------------------------- | --------------------------- |
| `Bool`                      | [Bool](/reference/data-types/boolean)                                                     | `Bool`                      |
| `I8`, `U8`                  | [Int8/UInt8](/reference/data-types/int-uint)                                              | `I8`, `U8`                  |
| `I16`, `U16`                | [Int16/UInt16](/reference/data-types/int-uint)                                            | `I16`, `U16`                |
| `I32`, `U32`                | [Int32/UInt32](/reference/data-types/int-uint)                                            | `I32`, `U32`                |
| `I64`, `U64`                | [Int64/UInt64](/reference/data-types/int-uint)                                            | `I64`, `U64`                |
| `F32`                       | [Float32](/reference/data-types/float)                                                    | `F32`                       |
| `F64`                       | [Float64](/reference/data-types/float)                                                    | `F64`                       |
| `Utf8`, `Binary`            | [String](/reference/data-types/string)                                                    | `Binary`                    |
| `Binary`                    | [FixedString](/reference/data-types/fixedstring)                                          | `Binary`                    |
| `Decimal`                   | [Decimal](/reference/data-types/decimal)                                                  | `Decimal`                   |
| `vortex.date`               | [Date32](/reference/data-types/date32)                                                    | `vortex.date`               |
| `vortex.timestamp`          | [DateTime](/reference/data-types/datetime)/[DateTime64](/reference/data-types/datetime64) | `vortex.timestamp`          |
| `vortex.time`               | [Time64](/reference/data-types/time64)                                                    | `vortex.time`               |
| `List`                      | [Array](/reference/data-types/array)                                                      | `List`                      |
| `Struct`                    | [Tuple](/reference/data-types/tuple)                                                      | `Struct`                    |
| `Null`                      | [Nullable(Nothing)](/reference/data-types/special-data-types/nothing)                     | `Null`                      |

Other types are not supported. In particular, [Map](/reference/data-types/map),
[Int128/UInt128/Int256/UInt256](/reference/data-types/int-uint), [IPv6](/reference/data-types/ipv6)
and [Interval](/reference/data-types/special-data-types/interval) columns cannot be written to Vortex files.
[String](/reference/data-types/string) columns are written as `Binary` because ClickHouse strings are
arbitrary byte sequences, while Vortex requires `Utf8` values to be valid UTF-8. Vortex has no
fixed-size binary type, so [FixedString](/reference/data-types/fixedstring) is also written as `Binary`;
schema inference reads it back as [String](/reference/data-types/string).
[LowCardinality](/reference/data-types/lowcardinality) columns are written as their underlying type
(Vortex chooses dictionary and other encodings adaptively by itself).
[DateTime](/reference/data-types/datetime) columns are written as `vortex.timestamp` with second precision,
so they are read back as [DateTime64](/reference/data-types/datetime64) with scale 0.
[IPv4](/reference/data-types/ipv4) columns are written as `U32` because Vortex has no type for IP addresses,
so schema inference reads them back as [UInt32](/reference/data-types/int-uint). Specify the type explicitly
to read such a column back as `IPv4`: `SELECT * FROM file('data.vortex', Vortex, 'ip IPv4')`.

The data types of ClickHouse table columns do not have to match the corresponding Vortex data fields.
When inserting data, ClickHouse interprets data types according to the table above and then
[casts](/reference/functions/regular-functions/type-conversion-functions#CAST) the data to the data type set for the
ClickHouse table column.

<h2 id="example-usage">
  Example usage
</h2>

You can select data from a Vortex file:

```sql theme={null}
SELECT * FROM file('data.vortex');
```

And write data to a Vortex file:

```sql theme={null}
SELECT * FROM numbers(3) INTO OUTFILE 'numbers.vortex' FORMAT Vortex;
```

<h2 id="format-settings">
  Format settings
</h2>

| Setting                                | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | Default         |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- |
| `input_format_vortex_filter_push_down` | Push translatable parts of the `WHERE` condition down into the scan, so that a selective query reads and decodes less data. Only the translated part of the condition is enforced inside the scan: a condition that is only partly translatable is widened to one that keeps at least every matching row, so rows that do not match can still be read and decoded, and ClickHouse reapplies the full filter afterwards. Pushdown currently supports top-level integer, floating-point, string/binary, `Bool`, `Date`, `Date32` and `DateTime64` columns. | `1`             |
| `input_format_vortex_preserve_order`   | Return the rows in file order. By default the row splits of a file are decoded in parallel and returned as soon as they are ready, so the row order is not guaranteed; with the setting a slow split holds back the ones after it.                                                                                                                                                                                                                                                                                                                       | `0`             |
| `max_parsing_threads`                  | The number of threads that decode Vortex files (shared by the files read in parallel by one query). `1` disables the thread pool: the file is then read inside `read`, on the thread of the query pipeline.                                                                                                                                                                                                                                                                                                                                              | number of cores |
| `max_download_threads`                 | The number of threads that read the file. `0` makes the reads share the decoding threads.                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `4`             |

As in other columnar formats, only the columns used by the query are read from the file, and
columns missing in the file are filled with default values.

<h2 id="performance">
  Performance
</h2>

A file is read in parallel. The scan splits it into row ranges (aligned to the chunk boundaries of
the requested columns, at most 100 000 rows each) that are read, filtered and decoded concurrently:
the decoding runs on up to `max_parsing_threads` threads of the same pool the `Parquet` reader uses,
the reads on up to `max_download_threads` threads, and the conversion of a decoded chunk to
ClickHouse columns happens on the thread that decoded it. Chunks are returned as soon as they are
ready, so the row order is not guaranteed unless `input_format_vortex_preserve_order` is set. The
reads of segments that are close in the file are merged into one request (up to 4 MiB for local
files and 16 MiB for remote storage). Filter pushdown reduces the amount of data read and decoded
by selective queries.
