Hi,
I’m extending some functionality of Kdb with a shared library which subscribes to quote feeds from within kdb (instead of using an external feed handler).
This means that the event processing is being done in real time from inside the shared library, and I’d like to update an existing table in kbd each time a new quote arrives. Is this even possible without callbacks or IPC?
Thanks,
Tiago Rodrigues
Hi,
just found the answer to my question on kov’s post ‘Question about callbacks from C-lib to q via C-function sd1()’.
I can use the following, with handle 0, for in process requests to kdb:
K result=k(0,“0N!”,ki(42),(K)0);
Best regards,
Tiago Rodrigues
On Monday, 25 May 2015 14:23:55 UTC+1, Tiago Rodrigues wrote:
Hi,
I’m extending some functionality of Kdb with a shared library which subscribes to quote feeds from within kdb (instead of using an external feed handler).
This means that the event processing is being done in real time from inside the shared library, and I’d like to update an existing table in kbd each time a new quote arrives. Is this even possible without callbacks or IPC?
Thanks,
Tiago Rodrigues
Well, it seems I still have a problem after all. I can run queries in q using the k(0, …), but since I’m calling from a slave thread I only have read-only access to kdb variables.
Any hints on how can I manipulate a kdb variable from inside a thread in the shared lib?
Many thanks,
Tiago Rodrigues
There are a few key points to adhere to:
use k(0,..) from the kdb+ main thread only.
use sd1 to register a callback from the kdb+ main thread.
free objects in the thread in which they were allocated.
do not share objects between threads.
to pass objects between threads, use serialization.
internalize strings (via ss or ks) in the main thread only, or call setm(1) to allow internalization from other threads.
Some may choose to ignore the above claiming that it seems to work ok anyway ;-) But they may be setting themselves up for tracking down nasty race conditions with random crashes sometime later.
Hope this helps,
Charlie
I setm(I f) is part of the sharedlib c-api; it can be called to activate locks around the interning of symbols in threads other than the kdb+ main thread. It should only be called when you can guarantee that no threads could be inside sn at the time of the call. e.g. call it from the kdb+ main thread on startup. It returns the previous setting.
Thanks, the callback works nicely. I was making things harder than they needed to be.
Tiago
Charles:
what is the meaning of the argument for setm(I f)?
…
setm(1);
ks(“asymbolinternedforever”);
setm(0); //???
…
> what is the meaning of the argument for setm(I f)?
The argument f is to activate locks around sym interning. If you need to intern from threads other than main, you should set this just once, from the main thread, and never reset it.
>- if k(0,…) and q function it calls do not write, can’t they be run safely on non-main threads?
If the q function does not read any global variables, it may be ok.
No, it just frees whatever was allocated for that thread.
>>- if k(0,…) and q function it calls do not write, can’t they be run safely on non-main threads?
>If the q function does not read any global variables, it may be ok.
Actually, not even then. There is still potential for ref count anomalies in the vm, which could lead to a crash after some time.
kdb+ itself handles these issues differently during peach/multithreaded input mode, but this is not available to the capi directly.
I was wondering if there was a reference manual for the sharedlib API (beyond whats on the integrating/extending with C on the wiki). That would be really helpful to better understand the internals and avoid some head-banging on the wall :)
there are some additional docs floating around
The internals of kdb+ are really quite complicated and subject to change; we try to hide that with the simple c-api.
when I read setm() above I thought thread-safety for ks()/ss().
if read-only access from non-main thread is not safe, the little descriptions for setm() and m9() are confusing… documentation does *not* say something like
you can launch threads with callbacks to main.
call setm() in the main once (and only once) before any threads use ks()/ss().
call m9() inside threads before exiting.
ok I am probably misreading mails and docs and second guessing too much!
another vote for reference doc. (lately i’ve been learning about a few new functions not appearing in wiki pages.)
something I’ve been wondering about: c api doesn’t seem to include a dicionary lookup support.
Hi Charles,
I really appreciate your help so far and hope I’m not abusing of your good will, but I still have some questions.
For instance, this works on the main thread, but results on a kdb crash when in a thread:
char *CALLBACK = NULL;
K callback() {
K a=kp(“C”);
k(0,CALLBACK,a,(K)0);
R (K)0; }
Z K1(test) {
CALLBACK=x->s;
sd1(0,callback);
R (K)0; }
The callback function in Q:
upd:{ update state:x from `.mbt.c}
Table definition:
c:([name:0#`] s:0#0N; state:0#`; server:0#`)
Also, there seems to be a 1-2 second delay on the update from the callback. Does it have some kind of deferred execution?
Here is an example:
q).mbt.c
name| s state server
----| --------------
qt | 6 N
q).mbt.test[`upd]
q).mbt.c <------ About 1 second after .mbt.test call
name| s state server
----| --------------
qt | 6 N
q).mbt.c <------ About 3 seconds after .mbt.test call
name| s state server
----| --------------
qt | 6 C
This also means that if I call r0(a) after the call to k(0,…) it causes a sigsegv (I assume its freeing ‘a’ before finishing the update?)
q).mbt.c
name| s state server
----| --------------
qt | 6 N
q).mbt.test[`upd]
q).mbt.c
name| s state server
----| --------------
qt | 6 N
q).mbt.c
name| s state server
----| --------------
qt | 6 .
q)
rlwrap: warning: q crashed, killed by SIGSEGV.
Am I completely misunderstanding the correct way to do this?
Tiago
the first arg to sd1 should be a file descriptor that the main kdb+ loop can monitor for readable activity via the system select call. For example, see the section using eventfds at http://code.kx.com/wiki/Cookbook/InterfacingWithC
I don’t see the sense in using stdin (fd 0) for the sd1 call sd1(0,…; I think that can only confuse kdb+ since it is already monitoring stdin.
For a system which is not under load, the latency should just be how long the select call takes to return - so a few uS. The callback is not deferred.
For these kinds of issues, it is really helpful to have a snippet of code that will fully compile and yet reproduces the issue.
Also please include (most of) the kdb+ startup banner as that has relevant details for the OS, kdb+ version/release etc.
Thanks Charlie
We’ve updated that document recently and added some source code : https://github.com/AquaQAnalytics/kdb-c-interface
We’ve been building some feed handlers in C for clients recently and created a feed handler tutorial document which is probably also relevant. It walks through how to build an example feed handler in c as both a shared object loaded into the q process and a standalone executable (https://github.com/AquaQAnalytics/kdb-feedhandler-tutorial)
Charlie, you are right, using the stdin was causing the delay (it waited for some input on q console before triggering the callback) which as you said didn’t make sense. The example feed handler Andrew linked to was quite helpful in understanding the way to follow and I’ve managed to get it working.
Andrew, if you won’t mind me asking, is there any special reason for your choice of using sockets instead of a pipe to communicate to the main process or is it just a matter of personal preference?