questions about tuning a NUMA server

Community Support
1

Hello everyone, 

i have some questions about tuning q processes on NUMA server (linux kernel version is above 3.x). 

  1. According to https://code.kx.com/q/kb/linux-production/  [1]  it is suggested to disable interleave and enable zone_reclaim, however https://code.kx.com/q/kb/cpu-affinity/#activating-numa [2]  suggests 

a) “numactl --interleave= all --physcpubind=0,1,2 q” 

Q1. Why allows the process to access all node’s memory instead of bind the process to local memory only? 

b) “echo 0 > /proc/sys/vm/zone_reclaim_mode”  which lets the process closes zone-reclaim and takes other numa nodes’s memory. 

Q2.  Is zone_reclaim_mode=1  the correct answer according to [1] ? 

The attached shows one of our q process,  on a numa-enabled server, binds to mem 0-3, 

Q3. Is it identical to “interleave=all” ? 

Q4. If i want this process access local mem only,  shall i execute  “numactl --interleave=0” ?

Q5. following up Q4,  How to start a q process and put numactl command’s to work in kx-platform?  which Reversed Parameter shall be edited? or, should i “run”/start a process first, then  execute system "numactl --interleave=0 --physcpubind=0 " inside process’s console ?

thank you and the best,

leguan, 

 

2

Hello,

Great questions for the Community! I will share them internally and come back with any feedback.

Many thanks

Luke

3

Hello Luke,

May i know if there is any update from tech-team? 

best,

leguan

4

Hello,

Thank you for your patience.

We have escalated it with the appropriate teams internally and will hopefully get back with an answer soon.

Many Thanks

Luke

5

Hello,

Reverting back with feedback for some of the questions:

Q1) Keeping the process within one node is the best option for performance. However, if the working set exceeds one node’s memory capacity, it then has to spill in to other nodes.

Q2) No, this rarely helps with recent kernels. This discussion will be removed altogether from code.kx.com. The default for modern systems is 0, which should be fine. (Regardless, this setting has no effect if confining the process to a single node)

Q3) No, the default NUMA policy attempts to allocate from the current node, and from others if no free memory is available. See https://man7.org/linux/man-pages/man2/set_mempolicy.2.html for more details.

Q4) You would usually express that as “numactl -N 0 -m 0” (allocate from node 0, and also run on the cores that belong to node 0)

I hope this information helps.

Many Thanks

Luke

6

Hello Luke , 

Thank you very much for the above helpful answers. 

I would like to ask how to start a q instance/task in kx-platform with specified numactrl command? 

best,

 

leguan

7

Hello,

You could try “numactl --cpubind=0 --membind=0,1 <command to start the q process>”. Let me know if this works.

Many Thanks

Luke

8

Hello Luck,

Thank you for the answer.

“numactl --cpubind=0 --membind=0,1 q blablabla…” This is a good practice to start a raw-kdb, to my understanding.

Well, what i am trying to ask is : 

when we wanna start a task of an instance in kx-platform(w r using v4.5.0), we can  right-click on the task and select “RUN”.   Therefore which instance’s RESERVED PARAM(e.g host/port/taskSet/enableIPSharding/memLimit …) i can edit so that when we “RUN” the task, it works just as start a raw-kdb process in the way as you suggested.

the best and wish you a great weekend!

leguan

9

Following up, i started a q instance as suggested, but it could not be recognized and managed by kx-platform, i guess it fails to register with the DC. 

best, 

10

Hello Luke and everyone,

I tried again  this morning but it still does not work.  I executed the following line:

“numactl -membind=0,1 /mykxPlatform_home/delta-bin/softeware/KDBPlus_myKdbVersion/l64/q q/prcl.q_ -process myInstanceName -prhost myControlFQDN -prhost portOfControl  -name CentralConfig-PidOfControl -logfile /pathOfLogdir/myProcessName.out -o 8”.

The result is: the process is seemingly UP, i can see its status is “running” in “Process Status Library” page, however after i right-click the process and select “connect”, i got a busy process which is unable to interact with, and there is no log file of it.

I would like to trouble you all to let me know how to start a process/task and making system command like “numactl” works on that process.
This is significant to us, given i re-installed whole platform last week, finding there is one server has high numa_miss, albeit the server has exactly the same hardware,even firmware version, with the other 3.
I would like to manage processes on-demand to see if we can reduce numa_miss and boost up rdb’s speed.
Thank you all!

best,

leguan