re: [personal kdb+] reducing k() overhead

Community Support
,
1

Hi,

Instead of calling it for single atoms, create vector versions of functions.
Overhead remains the same but is spread over the 1000’s of data points.

In your case this would become:
\ts do[1; f M#0]

  • Ryan

From : “Jay Han” <jayhan@gmail.com>
Sent : Tuesday, November 27, 2012 7:26 AM
To : personal-kdbplus@googlegroups.com
Subject : [personal kdb+] reducing k() overhead

I have a shared library in c that calls q with k(). I notice that k() is relatively expensive.

Is there any way to make it faster? Thanks!

: air:q ; cat b.c

// derived from http://kx.com/q/c/c/a.c

// clang -Wall -Wextra -m32 -bundle -undefined dynamic_lookup -L./m32 b.c -o m32/b.so

#include <stdio.h>

#include"k.h"

K f(K x){return ki(x->i+1);}  // q calls c

K g(K x){return k(0,“1+”,r1(x),0);} // c calls q

K h(K x){return k(0,“fq”,r1(x),0);} // ditto; fq:{:1+x} in b.q

: air:q ; cat b.q

f:b 2:(f;1)

g:b 2:(g;1)

fq:{:1+x}

h:b 2:(h;1)

M:1000*1000

\ts do[M; f 0]

\ts do[M; ff:f 0]

\ts do[M; g 0]

\ts do[M; gg:g 0]

\ts do[M; h 0]

\ts do[M; hh:h 0]

\

: air:q ; q b.q

KDB+ 2.8 2012.09.02 Copyright (C) 1993-2012 Kx Systems

m32/ 2()core 4096MB hjh air.local 10.0.1.2 PLAY 2012.12.01 

404 592j

661 704j

5810 592j / c calling q is not cheap

5019 704j

5194 592j

6442 704j

: air:q ; 

–
Submitted via Google Groups

2

That’s is one possible answer.  However, an event loop or data points from unknown future timestamps wouldn’t easily yield to this approach. (Well, it’s not impossible – binning functions by  latency/timeout and applying them to a segment of data is possible workaround, but a bit convoluted. :-)

kx.com/FD folks: any comment on the 10x overhead of k() would be much appreciated!

3

Hi Jay,

> comment on the 10x overhead of k()

One contributor to the overhead is calling a function in a shared
object. You can discover the overhead by using pure c:

======== a.c
f(void);
main()
{
return f();
}
======== b.c
f()
{
return 39;
}

Compare the run time of two cases where you statically link (cc a.c
b.c -o c) and where you dynamically link in f(). I’m pretty sure
you’ll see a big difference.

Ta, Jack.