C to Q bulk insert

Community Support
1

Hello Experts,

 

I have trolled the archives looking for a solution, to no avail. Apologies if this has been answered before.

 

I am attempting a bulk insert from C to Q.

 

here is the Q script:

quote:( date:date$(); sym:$();time:time$(); mkt:$(); bidpx: long$(); bidsize:int$(); bidcount:int$(); askpx:long$(); asksize:int$(); askcount:int$(); mkttimestamp:`timestamp$())
.u.upd:{[t;d] t insert d}
…

 

In C++, I do:

 

…

 

enum FieldList
 {
  DATE,
  SYM,
  TIME,
   …
  FieldCount
 };

 

 K q = ktn(0, FieldCount);
 QuoteList::size_type count = quotes.size();
 K date = kK(q)[DATE] = ktn(KD, count);
 K sym = kK(q)[SYM] = ktn(KS, count);
 K time = kK(q)[TIME] = ktn(KT, count);

…

 QuoteList::const_iterator s = quotes.begin(), f = quotes.end();
 int i = 0;
 for(;s!=f;++s,++i)
 {
  const KdbQuote& quote = (*s);
  kI(date)[i] = kLocalDate(quote.eventTime);
  kS(sym)[i] = ss((char*)quote.symbol.c_str());

…

 }

 

//finally

K r = k(0,(char*)“.u.upd”, ks(ss((char*)“quote”)), q, (K)0 );

 

… the problem is …

 

Here, q throws a segv in r0()

 

Program received signal SIGSEGV, Segmentation fault.
0x00000000004156eb in r0 ()
(gdb) where
#0  0x00000000004156eb in r0 ()
#1  0x0000000000415866 in r0 ()
#2  0x0000000000415866 in r0 ()
#3  0x0000000000419c5a in k ()
#4  0x00002aaaafc599b3 in Accumulator::pub (this=0x7fffffffd9c0, quotes=…) at /home/n596524/workspace/FXGATEWAYS/trunk/kdb/Accumulator.cpp:123

 

On the surface it appears like a reference count problem, but I’m not sure what the issue is. Perhaps the inner lists need their references incremented?

 

Thanks.

 

 

 

 

 

 

 

2

the snippet that you’ve posted looks ok.

k(c,“f”,x,y,z,0) will decrement the ref count on x,y,z during the k call.

Are you calling r0 explicitly anywhere on these objects? From that code, the only object you’ll need to r0 is the return value from the k call, r.

3

The r0 is being called from k() within the k(0, “.u.upd”… call. I’m not at the point where I have to r0(r). The segfault is happening in the call to k(0, …)

 

Thanks

 

 

 

On Thursday, March 6, 2014 1:27:41 PM UTC-5, Arvind Narayan wrote:

Hello Experts,

 

I have trolled the archives looking for a solution, to no avail. Apologies if this has been answered before.

 

I am attempting a bulk insert from C to Q.

 

here is the Q script:

quote:( date:date$(); sym:$();time:time$(); mkt:$(); bidpx: long$(); bidsize:int$(); bidcount:int$(); askpx:long$(); asksize:int$(); askcount:int$(); mkttimestamp:`timestamp$())
.u.upd:{[t;d] t insert d}
…

 

In C++, I do:

 

…

 

enum FieldList
 {
  DATE,
  SYM,
  TIME,
   …
  FieldCount
 };

 

 K q = ktn(0, FieldCount);
 QuoteList::size_type count = quotes.size();
 K date = kK(q)[DATE] = ktn(KD, count);
 K sym = kK(q)[SYM] = ktn(KS, count);
 K time = kK(q)[TIME] = ktn(KT, count);

…

 QuoteList::const_iterator s = quotes.begin(), f = quotes.end();
 int i = 0;
 for(;s!=f;++s,++i)
 {
  const KdbQuote& quote = (*s);
  kI(date)[i] = kLocalDate(quote.eventTime);
  kS(sym)[i] = ss((char*)quote.symbol.c_str());

…

 }

 

//finally

K r = k(0,(char*)“.u.upd”, ks(ss((char*)“quote”)), q, (K)0 );

 

… the problem is …

 

Here, q throws a segv in r0()

 

Program received signal SIGSEGV, Segmentation fault.
0x00000000004156eb in r0 ()
(gdb) where
#0  0x00000000004156eb in r0 ()
#1  0x0000000000415866 in r0 ()
#2  0x0000000000415866 in r0 ()
#3  0x0000000000419c5a in k ()
#4  0x00002aaaafc599b3 in Accumulator::pub (this=0x7fffffffd9c0, quotes=…) at /home/n596524/workspace/FXGATEWAYS/trunk/kdb/Accumulator.cpp:123

 

On the surface it appears like a reference count problem, but I’m not sure what the issue is. Perhaps the inner lists need their references incremented?

 

Thanks.

 

 

 

 

 

 

 

4

Hi Charles,

 

The r0() is getting called inside the k() call. Further, it appears that creating a list of lists as above using ktn(0, count) and knk(count, …) seem to produce different looking data structures. The ones produced with knk() seem to get inserted properly, but the ones created with ktn seem to break as above.

 

Since knk has the 8 column limit, I need to use the above approach. Have you any ideas on what might cause the break?

 

Thanks

On Thursday, March 6, 2014 1:49:30 PM UTC-5, Charles Skelton wrote:

the snippet that you’ve posted looks ok.

k(c,“f”,x,y,z,0) will decrement the ref count on x,y,z during the k call.

Are you calling r0 explicitly anywhere on these objects? From that code, the only object you’ll need to r0 is the return value from the k call, r.

On Thu, Mar 6, 2014 at 7:27 PM, Arvind Narayan <arvind…@gmail.com> wrote:

Hello Experts,

 

I have trolled the archives looking for a solution, to no avail. Apologies if this has been answered before.

 

I am attempting a bulk insert from C to Q.

 

here is the Q script:

quote:( date:date$(); sym:$();time:time$(); mkt:$(); bidpx: long$(); bidsize:int$(); bidcount:int$(); askpx:long$(); asksize:int$(); askcount:int$(); mkttimestamp:`timestamp$())
.u.upd:{[t;d] t insert d}
…

 

In C++, I do:

 

…

 

enum FieldList
 {
  DATE,
  SYM,
  TIME,
   …
  FieldCount
 };

 

 K q = ktn(0, FieldCount);
 QuoteList::size_type count = quotes.size();
 K date = kK(q)[DATE] = ktn(KD, count);
 K sym = kK(q)[SYM] = ktn(KS, count);
 K time = kK(q)[TIME] = ktn(KT, count);

…

 QuoteList::const_iterator s = quotes.begin(), f = quotes.end();
 int i = 0;
 for(;s!=f;++s,++i)
 {
  const KdbQuote& quote = (*s);
  kI(date)[i] = kLocalDate(quote.eventTime);
  kS(sym)[i] = ss((char*)quote.symbol.c_str());

…

 }

 

//finally

K r = k(0,(char*)“.u.upd”, ks(ss((char*)“quote”)), q, (K)0 );

 

… the problem is …

 

Here, q throws a segv in r0()

 

Program received signal SIGSEGV, Segmentation fault.
0x00000000004156eb in r0 ()
(gdb) where
#0  0x00000000004156eb in r0 ()
#1  0x0000000000415866 in r0 ()
#2  0x0000000000415866 in r0 ()
#3  0x0000000000419c5a in k ()
#4  0x00002aaaafc599b3 in Accumulator::pub (this=0x7fffffffd9c0, quotes=…) at /home/n596524/workspace/FXGATEWAYS/trunk/kdb/Accumulator.cpp:123

 

On the surface it appears like a reference count problem, but I’m not sure what the issue is. Perhaps the inner lists need their references incremented?

 

Thanks.

--

Submitted via Google Groups

5

the ktn approach should work fine. It’s quite a standard approach.
Are your iterators working correctly such that it really does only insert count items in the loop?

DATE really does have an enum value of 0?

If the knk approach works, could it be a problem with the other columns that you could not test with the knk approach?

I would suspect something in your code is overwriting the header of one of your k objects; e.g. running off the end of an array.

6

e.g. if you are using the wrong accessor (for the underlying vector) with an index, that could send you off the end of allocated space.