c++ interfacing - handling dictionaries of tables

Community Support
1

Hi,
I am trying to process some data in c++. I first got my rawData on kdb+.

Then I converted it to a dictionary of tables for easier processing (not sure if this would be the best approach)

applyParamsR: {

dictR:: ()!();

tmp: raze addDictR each (select distinct date from rawData)`date;

dictR

};

applyParamsR

Then I just downloaded this in to c++ and stored in a K object say flip.

Then i created another variable processedFlip and initialized to flip.

Now I have a function called processData which essentially takes each table in this dictionary, processes it and then returns the modified table and i assign the processedFlip key to this modified table.

The trouble is when i try to access the flip again, it is now behaving like processedFlip. Instead of raw data, it now seems to have processed data. Is this expected behaviour? My raw data had 10 columns, on second run of applyParamC, there are only 8(as in processed data).


Some code below.

K applyParamC(K& allD, appParams& apar)

{

cout << “in appPC \n”;

K keys, data, dat1, dat1s, dat2s, procFlip;

int ndicts, row;

keys = kK(allD)[0];

data = kK(allD)[1];

ndicts = data->n;

procFlip = allD;

cout << “ndicts:” << ndicts << “\n”;

for(row=0; row<ndicts; ++row)

{ cout << "dictprocessed: " << row << “\n”;

dat1 = kK(data)[row];

dat1s = kK(dat1->k)[1];

dat2s = applyParamSin(dat1s, apar);

cout << “dat2s returned \n”;

kK(kK(procFlip)[1])[row] = dat2s;

//r0(dat2s);

}

return(procFlip);

}

main ----

K flip = getRawData();

K appFlip;

for(i=0; i<2; i++)

appFlip = applyParamC(flip, apar);

appParams is just a normal struct. applyParamC runs fine for the first time but when i call it again it gives a segmentation fault coz the code tries to access 9th column but now the data has only 8 columns.

2

Sharma,

> The trouble is when i try to access the flip again, it is now behaving like processedFlip. Instead of raw data, it now seems to have processed data. Is
> this expected behaviour? My raw data had 10 columns, on second run of applyParamC, there are only 8(as in processed data).

Yes, it’s expected behaviour. Please bear in mind that K type is just a pointer to k0 struct, so after you do procFlip=allD, procFlip and allD point to the same memory, no values are copied. The data is overwritten in this line of code:
kK(kK(procFlip)[1])[row] = dat2s;
because it’s the same as writing kK(kK(allD)[1])[row].

To work around this problem you would need to make a 2-level copy of allD K object and update reference counts appropriately.

HTH,
Pawel

2013/10/22 <sharma.naveen888@gmail.com>

Hi,
I am trying to process some data in c++. I first got my rawData on kdb+.

Then I converted it to a dictionary of tables for easier processing (not sure if this would be the best approach)

applyParamsR: {

dictR:: ()!();

tmp: raze addDictR each (select distinct date from rawData)`date;

dictR

};

applyParamsR

Then I just downloaded this in to c++ and stored in a K object say flip.

Then i created another variable processedFlip and initialized to flip.

Now I have a function called processData which essentially takes each table in this dictionary, processes it and then returns the modified table and i assign the processedFlip key to this modified table.

The trouble is when i try to access the flip again, it is now behaving like processedFlip. Instead of raw data, it now seems to have processed data. Is this expected behaviour? My raw data had 10 columns, on second run of applyParamC, there are only 8(as in processed data).


Some code below.

K applyParamC(K& allD, appParams& apar)

{

cout << “in appPC \n”;

K keys, data, dat1, dat1s, dat2s, procFlip;

int ndicts, row;

keys = kK(allD)[0];

data = kK(allD)[1];

ndicts = data->n;

procFlip = allD;

cout << “ndicts:” << ndicts << “\n”;

for(row=0; row<ndicts; ++row)

{ cout << "dictprocessed: " << row << “\n”;

dat1 = kK(data)[row];

dat1s = kK(dat1->k)[1];

dat2s = applyParamSin(dat1s, apar);

cout << “dat2s returned \n”;

kK(kK(procFlip)[1])[row] = dat2s;

//r0(dat2s);

}

return(procFlip);

}

main ----

K flip = getRawData();

K appFlip;

for(i=0; i<2; i++)

appFlip = applyParamC(flip, apar);

appParams is just a normal struct. applyParamC runs fine for the first time but when i call it again it gives a segmentation fault coz the code tries to access 9th column but now the data has only 8 columns.

–

Submitted via Google Groups

3

Hi Pawel, 
I am a little confused about this, not sure what you mean by 2 level copy and reference count. can you please explain a little bit and give an example if possible.

4

Or the alternate could be I populate procFlip as I go along. but not sure there’s an easy way to do this. Can I put these returned tables into a mixed list and create a dictionary later using original dictionary keys?
Is there a way to use knk in a loop setting?

5

That alternate approach seems better to me.

When using knk() you need to know at the time of writing the code the number and all the elements of a mixed list. In your case you would like to use ktn(0,n) to create a mixed list with n elements and then populate it with values (tables returned by applyParamSin() function).

You can use original dictionary keys list, but you need to increase its reference count using r1() function, otherwise KDB will be giving you segfaults. Make sure to do r1() for every K object you reuse.

More on memory management and reference counting can be found here: http://code.kx.com/wiki/Cookbook/InterfacingWithC#Managing_memory_and_Reference_Counting

2013/10/22 <sharma.naveen888@gmail.com>

Or the alternate could be I populate procFlip as I go along. but not sure there’s an easy way to do this. Can I put these returned tables into a mixed list and create a dictionary later using original dictionary keys?
Is there a way to use knk in a loop setting?

6

Thanks a bunch Pawel. somehow I had missed ktn(0, n). Surprisingly I didn’t do any r0 or r1 but everything still worked fine.