# pykx incompatible with subprocess and multiprocess - core dumps
**URL:** https://forum.kx.com/t/pykx-incompatible-with-subprocess-and-multiprocess-core-dumps/12904
**Category:** Community Support
**Tags:** pykx
**Created:** [June 16, 2023, 4:30pm UTC](https://forum.kx.com/t/pykx-incompatible-with-subprocess-and-multiprocess-core-dumps/12904 "2023-06-16T16:30:00Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@hodgidav1](https://forum.kx.com/u/hodgidav1)
#### Post date: [June 16, 2023, 4:30pm UTC](https://forum.kx.com/t/pykx-incompatible-with-subprocess-and-multiprocess-core-dumps/12904/1 "2023-06-16T16:30:00Z")
Hi,
If I try to spawn a child python process after importing pykx the child will core dump.
$ python --version
Python 3.10.10
$ pip list
Package Version
* * *
ms.version 2.9.0
numpy 1.24.3
pandas 1.5.3
pip 23.1.2
pykx 1.5.4
python-dateutil 2.8.2
pytz 2022.7.1
setuptools 65.5.0
six 1.16.0
$ cat parent.py
print(“parent start”)
import pykx
import subprocess
proc=subprocess.Popen([‘/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/bin/python’,‘child.py’],stdout=subprocess.PIPE,stderr=subprocess.PIPE);proc.communicate();print(proc.wait())
print(“parent end”)
import sys
sys.exit(69)
$ cat child.py
print(‘Child start’)
import sys
print(‘loading pykx’)
#import pykx
print(‘Child end’)
sys.exit(42)
$ python parent.py
parent start
42 #as expected
parent end
If I also import pykx in the child it fails:
$ cat child.py
print(‘Child start’)
import sys
print(‘loading pykx’)
import pykx
print(‘Child end’)
sys.exit(42)
$ python parent.py
parent start
-11
parent end
stracing the child reveals the coredump here:
lstat(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/lib/python3.10/site-packages/pykx/lib/l64/libq.so”, {st\_mode=S\_IFREG|0755, st\_size=813136, …}) = 0
— SIGSEGV {si\_signo=SIGSEGV, si\_code=SEGV\_MAPERR, si\_addr=NULL} —
+++ killed by SIGSEGV (core dumped) +++
Stracing the parent shows what it is supposed to look like:
lstat(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/lib/python3.10/site-packages/pykx/lib/l64/libq.so”, {st\_mode=S\_IFREG|0755, st\_size=813136, …}) = 0
pipe2([3, 4], O\_CLOEXEC) = 0
fstat(3, {st\_mode=S\_IFIFO|0600, st\_size=0, …}) = 0
ioctl(3, TCGETS, 0x7ffe05620b40) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK\_CUR) = -1 ESPIPE (Illegal seek)
ioctl(3, TCGETS, 0x7ffe05620bc0) = -1 ENOTTY (Inappropriate ioctl for device)
pipe2([5, 6], O\_CLOEXEC) = 0
rt\_sigprocmask(SIG\_BLOCK, ~[RTMIN RT\_1], , = 0
vfork() = 349938
rt\_sigprocmask(SIG\_SETMASK, , NULL, = 0
close(6) = 0
close(4) = 0
read(5, “”, 50000) = 0
close(5) = 0
lseek(3, 0, SEEK\_CUR) = -1 ESPIPE (Illegal seek)
fstat(3, {st\_mode=S\_IFIFO|0600, st\_size=0, …}) = 0
read(3, “”, 8192) = 0
close(3) = 0
wait4(349938, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 349938
open(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/lib/python3.10/site-packages/pykx/lib/l64/libq.so”, O\_RDONLY|O\_CLOEXEC) = 3
read(3, “\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\>\0\1\0\0\0\300\206\0\0\0\0\0\0”…, 832) = 832
fstat(3, {st\_mode=S\_IFREG|0755, st\_size=813136, …}) = 0
mmap(NULL, 3684984, PROT\_READ|PROT\_EXEC, MAP\_PRIVATE|MAP\_DENYWRITE, 3, 0) = 0x7f8e4ccd9000
Similarly if I switch from subprocess to multiprocess and set to spawn instead of fork it still fails.
This time we dont even need to load pykx in the child:
$ cat child.py
print(‘Child start’)
import sys
print(‘loading pykx’)
#import pykx
print(‘Child end’)
sys.exit(42)
$ cat parentMP.py
import os
import sys
import multiprocessing
import pykx
def task():
os.system(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/bin/python -u child.py\>/tmp/hodgidav2/out 2\>&1”)
if \_\_name\_\_ == ‘\_\_main\_\_’:
multiprocessing.set\_start\_method(‘spawn’,force=True)
print(“parent start”)
# create the process
process = multiprocessing.Process(target=task)
# start the process
process.start()
# wait for the process to complete
process.join()
print(“parent end”)
sys.exit(69)
$ python parentMP.py
parent start
parent end
$ cat /tmp/hodgidav2/out
cat: /tmp/hodgidav2/out: No such file or directory
If I remove pykx from parentMP.py then it works as expected:
$ cat parentMP.py
import os
import sys
import multiprocessing
#import pykx
def task():
os.system(“/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/bin/python -u child.py\>/tmp/hodgidav2/out 2\>&1”)
if \_\_name\_\_ == ‘\_\_main\_\_’:
multiprocessing.set\_start\_method(‘spawn’,force=True)
print(“parent start”)
# create the process
process = multiprocessing.Process(target=task)
# start the process
process.start()
# wait for the process to complete
process.join()
print(“parent end”)
sys.exit(69)
$ python parentMP.py
parent start
parent end
$ cat /tmp/hodgidav2/out
Child start
loading pykx
Child end
---
### Author:  [@kshepherd1](https://forum.kx.com/u/kshepherd1)
#### Post date: [June 19, 2023, 4:00pm UTC](https://forum.kx.com/t/pykx-incompatible-with-subprocess-and-multiprocess-core-dumps/12904/2 "2023-06-19T16:00:00Z")
This has to do with the fact that PyKX uses a few environment variables to stop certain pieces from being loaded twice if it is reimported. The parent process loads PyKX and the env vars are set, when the child process is created these env vars are passed along to the child process, which causes PyKX in the child process to only partially load and segfault.
This can be fixed by loading PyKX in the parent process after the child process is spawned.
parent.py
```
print("parent start") import subprocess proc=subprocess.Popen( ['/a/stor164ncs2.new-york.ms.com/sc34770/s180992/hodgidav/myenv/bin/python','child.py'], stdout=subprocess.PIPE, stderr=subprocess.PIPE ) import pykx # import PyKX after spawning the child process proc.communicate() print(proc.wait()) print("parent end") import sys sys.exit(69)
```
---
### Author:  [@rocuinneagain1](https://forum.kx.com/u/rocuinneagain1)
#### Post date: [July 21, 2023, 11:58am UTC](https://forum.kx.com/t/pykx-incompatible-with-subprocess-and-multiprocess-core-dumps/12904/3 "2023-07-21T11:58:00Z")
PyKX 1.6.1 has been released. Full details on the release can be found here.
Highlights:
- Added `sorted`, `grouped`, `parted`, and `unique`. As methods off of `Tables` and `Vectors`.
- **Added `PyKXReimport` class to allow subprocesses to reimport `PyKX` safely. Details here.**
- Added environment variables to specify a path to `libpython`. Details here.
- Fixed memory leaks in QConnection and PyKX as a server.
- Fixed bug in Jupyter Notebook magic command.
Full details including many other fixes included here.
import pykx as kx with kx.PyKXReimport(): output = subprocess.run( (str(Path(sys.executable).as\_posix()), ‘-c’, ‘import pykx; print(pykx.q(“til 10”))’), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, ).stdout.strip() assert output == “0 1 2 3 4 5 6 7 8 9”