kxnoob
August 8, 2023, 12:00am
1
https://learninghub.kx.com/forums/topic/arrowkdb
I have tried installing ArrowKDB from source.
I followed the installation instructions and the instructions to build worked properly.
git clone https://github.com/KxSystems/arrowkdb.git cd arrowkdb
mkdir build cd build cmake --build . --config Release
cmake --build . --config Release --target install
I had to change somethings in the config and some environment variables to make it works but it builds successfully and it builds successfully.
$ cmake --build . --config Release --target install
However, when I try to load arrowkdb I get the following:
q arrowkdb.q
KDB+ 4.0 Cloud Edition 2022.09.30 Copyright (C) 1993-2022 Kx Systems
l64/ 32()core 253885MB <> EXPIRE 2024.03.14 <> KXCO KXCE <>
'<>/q/l64/arrowkdb.so: undefined symbol: _ZNK5arrow9ArraySpan24UnionMayHaveLogicalNullsEv
[2] <>/q/arrowkdb.q:6: dt.boolean:`arrowkdb 2:(`boolean;1);
Can someone point me to what might have gone wrong in the build process? I am building against arrow 9. Using the release version did not work.
Arrowkdb should work with earlier libarrow versions, say 5.0 (or maybe even 3.0) since your are building from source.
What is your distro?
If you’re able to install libarrow.so.800 you should also be able to install libparquet.so.800. It’ll be two separate yum commands similar to https://github.com/KxSystems/arrowkdb#linux (although that’s apt on ubuntu)
See https://arrow.apache.org/install/#c-and-glib-c-packages-for-debian-gnulinux-ubuntu-and-centos:
Something like this from centos 7:
sudo yum install -y epel-release || sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$( cut -d : -f5 /etc/system-release-cpe | cut -d . -f1 ) .noarch.rpm sudo yum install -y https://apache.jfrog.io/artifactory/arrow/centos/$( cut -d : -f5 /etc/system-release-cpe | cut -d . -f1 ) /apache-arrow-release-latest.rpm sudo yum install -y --enablerepo = epel arrow-devel # For C++ sudo yum install -y --enablerepo = epel parquet-devel # For Apache Parquet C++
kxnoob
August 8, 2023, 12:00am
3
I had to change somethings in the config and some environment variables to make it works but it builds successfully
Yeah - I had to change the CMakeLists to point to that libarrow12 (ARROW_LIBRARY and PARQUET_LIBRARY). I tried pointing to a libarrow9 version in a different miniconda environment. I also had to force it to use C++ 17.
Unfortunately, I am not able to install libarrow 9 on my system outside of conda. The most my system is able to yum install is libarrow8. It installs libarrow.so.800 to /usr/lib64 but not a libparquet.so.800.
Do you have any suggestions for any workarounds to this? I’m not an expert on CMake or building C++ packages and whatnot.
Hi,
I suspect you’re missing the libarrow dependencies.ldd $QHOME/l64/arrowkdb.so?
ldd arrowkdb.so
linux-vdso.so.1 (0x00007ffed21f9000)
libarrow.so.900 => /lib/x86_64-linux-gnu/libarrow.so.900 (0x00007f92d5ded000)
libparquet.so.900 => /lib/x86_64-linux-gnu/libparquet.so.900 (0x00007f92d5ac2000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f92d58e0000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f92d58c5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f92d56d3000)
libbrotlienc.so.1 => /lib/x86_64-linux-gnu/libbrotlienc.so.1 (0x00007f92d564a000)
libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007f92d563a000)
libutf8proc.so.2 => /lib/x86_64-linux-gnu/libutf8proc.so.2 (0x00007f92d55ed000)
libre2.so.5 => /lib/x86_64-linux-gnu/libre2.so.5 (0x00007f92d557c000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f92d5576000)
libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f92d529f000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f92d528c000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f92d5269000)
libsnappy.so.1 => /lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007f92d525e000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f92d5242000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f92d5199000)
libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f92d5107000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f92d4fb8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f92d4f93000)
/lib64/ld-linux-x86-64.so.2 (0x00007f92d7c87000)
libthrift-0.13.0.so => /lib/x86_64-linux-gnu/libthrift-0.13.0.so (0x00007f92d4edc000)
libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007f92d4eb9000)
libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f92d4e90000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f92d4e6f000)
librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f92d4e4d000)
libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f92d4ddf000)
libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f92d4dcc000)
libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f92d4d39000)
libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f92d4cec000)
libldap_r-2.4.so.2 => /lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f92d4c96000)
liblber-2.4.so.2 => /lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f92d4c83000)
libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f92d4b01000)
libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f92d492b000)
libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007f92d48f4000)
libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 (0x00007f92d48ba000)
libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f92d4834000)
libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f92d4757000)
libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f92d4726000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f92d471f000)
libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f92d4710000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f92d46f2000)
libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f92d46d5000)
libgssapi.so.3 => /lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f92d4690000)
libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f92d455a000)
libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f92d4544000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f92d453b000)
libheimntlm.so.0 => /lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f92d452f000)
libkrb5.so.26 => /lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f92d449c000)
libasn1.so.8 => /lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f92d43f5000)
libhcrypto.so.4 => /lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f92d43bd000)
libroken.so.18 => /lib/x86_64-linux-gnu/libroken.so.18 (0x00007f92d43a2000)
libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007f92d4396000)
libwind.so.0 => /lib/x86_64-linux-gnu/libwind.so.0 (0x00007f92d436c000)
libheimbase.so.1 => /lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f92d435a000)
libhx509.so.5 => /lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f92d430c000)
libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f92d41e3000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f92d41a6000)
kxnoob
August 8, 2023, 12:00am
5
Hello, thank you for your response!
I got the following output.
$ ldd $QHOME/l64/arrowkdb.so linux-vdso.so.1 (0x00007ffd3e3fc000) libarrow.so.1200 => <>/miniconda3/lib/python3.9/site-packages/pyarrow/libarrow.so.1200 (0x00007fbaf4ee9000) libparquet.so.1200 => <>/miniconda3/lib/python3.9/site-packages/pyarrow/libparquet.so.1200 (0x00007fbaf4760000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fbaf43cb000) libm.so.6 => /lib64/libm.so.6 (0x00007fbaf4049000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fbaf3e31000) libc.so.6 => /lib64/libc.so.6 (0x00007fbaf3a6c000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fbaf3868000) librt.so.1 => /lib64/librt.so.1 (0x00007fbaf3660000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbaf3440000) /lib64/ld-linux-x86-64.so.2 (0x00007fbaf7c8a000)