pykx incompatible with subprocess and multiprocess - core dumps

Community Support
1

Hi,

 

If I try to spawn a child python process after importing pykx the child will core dump.

 

$ python --version

Python 3.10.10

$ pip list

Package         Version

ms.version      2.9.0

numpy           1.24.3

pandas          1.5.3

pip             23.1.2

pykx            1.5.4

python-dateutil 2.8.2

pytz            2022.7.1

setuptools      65.5.0

six             1.16.0

 

 

$ cat parent.py

print(“parent start”)

import pykx

import subprocess

proc=subprocess.Popen([‘/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/bin/python’,‘child.py’],stdout=subprocess.PIPE,stderr=subprocess.PIPE);proc.communicate();print(proc.wait())

print(“parent end”)

import sys

sys.exit(69)

 

$ cat child.py

print(‘Child start’)

import sys

print(‘loading pykx’)

#import pykx

print(‘Child end’)

sys.exit(42)

 

$ python parent.py

parent start

42 #as expected

parent end

 

If I also import pykx in the child it fails:

$ cat child.py

print(‘Child start’)

import sys

print(‘loading pykx’)

import pykx

print(‘Child end’)

sys.exit(42)

 

$ python parent.py

parent start

-11

parent end

stracing the child reveals the coredump here:

lstat(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/lib/python3.10/site-packages/pykx/lib/l64/libq.so”, {st_mode=S_IFREG|0755, st_size=813136, …}) = 0

— SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} —

+++ killed by SIGSEGV (core dumped) +++

 

Stracing the parent shows what it is supposed to look like:

lstat(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/lib/python3.10/site-packages/pykx/lib/l64/libq.so”, {st_mode=S_IFREG|0755, st_size=813136, …}) = 0

pipe2([3, 4], O_CLOEXEC)                = 0

fstat(3, {st_mode=S_IFIFO|0600, st_size=0, …}) = 0

ioctl(3, TCGETS, 0x7ffe05620b40)        = -1 ENOTTY (Inappropriate ioctl for device)

lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)

ioctl(3, TCGETS, 0x7ffe05620bc0)        = -1 ENOTTY (Inappropriate ioctl for device)

pipe2([5, 6], O_CLOEXEC)                = 0

rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], , = 0

vfork()                                 = 349938

rt_sigprocmask(SIG_SETMASK, , NULL, = 0

close(6)                                = 0

close(4)                                = 0

read(5, “”, 50000)                      = 0

close(5)                                = 0

lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)

fstat(3, {st_mode=S_IFIFO|0600, st_size=0, …}) = 0

read(3, “”, 8192)                       = 0

close(3)                                = 0

wait4(349938, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 349938

open(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/lib/python3.10/site-packages/pykx/lib/l64/libq.so”, O_RDONLY|O_CLOEXEC) = 3

read(3, “\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\206\0\0\0\0\0\0”…, 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=813136, …}) = 0

mmap(NULL, 3684984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8e4ccd9000

Similarly if I switch from subprocess to multiprocess and set to spawn instead of fork it still fails.

 

This time we don’t even need to load pykx in the child:

$ cat child.py

print(‘Child start’)

import sys

print(‘loading pykx’)

#import pykx

print(‘Child end’)

sys.exit(42)

 

$ cat parentMP.py

import os

import sys

import multiprocessing

import pykx

def task():

    os.system(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/bin/python -u child.py>/tmp/hodgidav2/out 2>&1”)

 

if __name__ == ‘__main__’:

        multiprocessing.set_start_method(‘spawn’,force=True)

        print(“parent start”)

        # create the process

        process = multiprocessing.Process(target=task)

        # start the process

        process.start()

        # wait for the process to complete

        process.join()

        print(“parent end”)

        sys.exit(69)

 

$ python parentMP.py

parent start

parent end

 

$ cat /tmp/hodgidav2/out

cat: /tmp/hodgidav2/out: No such file or directory

 

If I remove pykx from parentMP.py then it works as expected:

$ cat parentMP.py

import os

import sys

import multiprocessing

#import pykx

def task():

    os.system(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/bin/python -u child.py>/tmp/hodgidav2/out 2>&1”)

 

if __name__ == ‘__main__’:

        multiprocessing.set_start_method(‘spawn’,force=True)

        print(“parent start”)

        # create the process

        process = multiprocessing.Process(target=task)

        # start the process

        process.start()

        # wait for the process to complete

        process.join()

        print(“parent end”)

        sys.exit(69)

 

$ python parentMP.py

parent start

parent end

 

$ cat /tmp/hodgidav2/out

Child start

loading pykx

Child end

 

2

This has to do with the fact that PyKX uses a few environment variables to stop certain pieces from being loaded twice if it is reimported. The parent process loads PyKX and the env vars are set, when the child process is created these env vars are passed along to the child process, which causes PyKX in the child process to only partially load and segfault.

This can be fixed by loading PyKX in the parent process after the child process is spawned.

parent.py

 

print("parent start") import subprocess proc=subprocess.Popen( ['/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/bin/python','child.py'], stdout=subprocess.PIPE, stderr=subprocess.PIPE ) import pykx # import PyKX after spawning the child process proc.communicate() print(proc.wait()) print("parent end") import sys sys.exit(69)

 

3

PyKX 1.6.1 has been released. Full details on the release can be found here.
Highlights:

  • Added sorted, grouped, parted, and unique. As methods off of Tables and Vectors.
  • Added PyKXReimport class to allow subprocesses to reimport PyKX safely. Details here.
  • Added environment variables to specify a path to libpython. Details here.
  • Fixed memory leaks in QConnection and PyKX as a server.
  • Fixed bug in Jupyter Notebook magic command.

Full details including many other fixes included here.

 
import pykx as kx with kx.PyKXReimport(): output = subprocess.run( (str(Path(sys.executable).as_posix()), ‘-c’, ‘import pykx; print(pykx.q(“til 10”))’), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, ).stdout.strip() assert output == “0 1 2 3 4 5 6 7 8 9”