KDB-X for parquet conversion

Community Support
1

Does KDB-x natively support KDB table to parquet conversion?

Also, is it possible to install arrowkdb on top of KDB-X?

We are trying to convert KDB tables into parquet format, please suggest here.

2

Native Parquet support is on the KDB-X Roadmap and will be released soon.

For now you can still use arrowkdb in a KDB-X process.

/Create a sample table
q)n:5
q)tl:"jsdtfi"!(n;`5;.z.d;.z.t;"f"$n;"i"$n)
q)ct:"jsdt"
q)gt:{flip (`$"col_",/:string til count x)!{n?tl x}each x}
q)t:gt ct

/Write it to a parquet file
q)\l arrowkdb.q
q)parquet_write_options:(enlist `PARQUET_VERSION)!(enlist `V2.0)
q).arrowkdb.pq.writeParquetFromTable["inferred_schema.parquet";t;parquet_write_options]

/Read the parquet file back
q)new_table:.arrowkdb.pq.readParquetToTable["inferred_schema.parquet";::]
3

Thanks, As per GitHub - KxSystems/arrowkdb: kdb+ integration with Apache Arrow and Parquet the installation of arrowkdb requires QHOME location which should contain a q.k file. Currently in KDB-x inside the .kx folder we are not able to find the q.k file

Loading arrowkdb.q using \l throws the following error. Could you please help with the steps for installing arrowkdb on kdb-x, with the pre-requisites specified (including licenses if any)?

image
image1382×559 46.7 KB

4

With QHOME not being used by default and a slightly different folder structure you can copy the two needed files manually:

mkdir arrowkdb
cd arrowkdb
wget https://github.com/KxSystems/arrowkdb/releases/download/1.4.1-rc.1/arrowkdb-linux-1.4.1-rc.1.tgz
tar -xvzf arrowkdb-linux-1.4.1-rc.1.tgz

mkdir ~/.kx/l64
cp lib/arrowkdb.so ~/.kx/l64/
cp q/arrowkdb.q ~/.kx/q/
5

Many thanks for your prompt response. I have tried the above steps, now i see error as libarrow.so not being available. Any suggestions in getting this resolved? Please advise

image
image1382×339 24.2 KB

6

Third party library installation is needed.

It directs you to the Apache site here.

For me on Ubuntu 20.04 I needed to run line by line:

sudo apt update
sudo apt install -y -V ca-certificates lsb-release wget
wget https://packages.apache.org/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb
sudo apt update
sudo apt install -y -V libarrow-dev=9.0.0-1
sudo apt install -y -V libparquet-dev=9.0.0-1

Now the library runs on this new VM:

$ q
KDB-X 0.1.0 2025.06.25 Copyright (C) 1993-2025 Kx Systems
l64/ 8()core 7862MB *** *** 127.0.1.1 EXPIRE 2025.09.30 ***@***.com COMMUNITY #***

Welcome to KDB-X Community Edition!
For Community support, please visit https://kx.com/slack
Tutorials can be found at https://github.com/KxSystems/tutorials
Ready to go beyond the Community Edition? Email preview@kx.com

q)\l arrowkdb.q
q)n:5
q)tl:"jsdtfi"!(n;`5;.z.d;.z.t;"f"$n;"i"$n)
q)ct:"jsdt"
q)gt:{flip (`$"col_",/:string til count x)!{n?tl x}each x}
q)t:gt ct
q)parquet_write_options:(enlist `PARQUET_VERSION)!(enlist `V2.0)
q).arrowkdb.pq.writeParquetFromTable["inferred_schema.parquet";t;parquet_write_options]
q)new_table:.arrowkdb.pq.readParquetToTable["inferred_schema.parquet";::]
q)new_table
col_0 col_1   col_2      col_3
-------------------------------------
4     "jecpa" 2004.02.01 05:18:45.828
0     "kfmoh" 2010.12.20 02:25:54.221
2     "lkklc" 2009.05.09 01:53:10.321
1     "kfifp" 2016.09.30 05:15:26.418
2     "fglgo" 2017.09.06 01:05:42.696