Re: qJava and Publisher

Community Support
1

if you want to insert more than one row, you’ll need to do it a little differently

.u.upd:{:dsPricing upsert flip iddateopenclosehighlowvolume!y;}

but as we discussed in another thread, you should really append to splayed tables instead.

2

Thanks for all your help ( since now I know the CTO for the company is helping me!). I think you’re right that I’m not doing this correctly and still getting the error
nfoCode:1,Date:Thu Feb 17 00:00:00 EST 1994,Open:73.0,close:73.0,High:76.0,Low:73.0,Volume:2223000.0
InfoCode:1,Date:Wed Feb 16 00:00:00 EST 1994,Open:76.0,close:76.0,High:77.899994,Low:75.0,Volume:3167000.0
com.exxeleron.qjava.QException: type
    at com.exxeleron.qjava.QReader.readError(QReader.java:409)
    at com.exxeleron.qjava.QReader.readObject(QReader.java:189)
    at com.exxeleron.qjava.QReader.read(QReader.java:99)
    at com.exxeleron.qjava.QBasicConnection.sync(QBasicConnection.java:159)
    at PublisherTask.run(Publisher.java:84)
    at java.lang.Thread.run(Thread.java:744)

I was wondering if there is a tutorial out there for Java that I can read some historical data for about 200,000 securities and insert that into a file ( obviously not enough memory). How should I approach this?
I assume the SQL query needs to be done in Java.

3

> read some historical data for about 200,000 

Are you extracting that from another database using java (and their SQL)? Which database vendor? If you have a jdbc driver for it, and the SQL is fairly simple, you could consider http://code.kx.com/wiki/Babel

I’m not familiar with the devnet qjava, but with the Kx http://kx.com/q/c/kx/c.java it looks like this (maybe you can tell if there are type diffs here).

public static void main(Stringargs){

  try{

    c c=new c(“”,5001);

    c.k(“addData”,new Object{new int{1,1},

                               new Date{Date.valueOf(“1994-2-17”),Date.valueOf(“1994-2-16”)},

                               new double{73.,76.},

                               new double{73.,76.},

                               new double{76.,77.899994},

                               new double{73.,75.},

                               new int{2223000,3167000}});

    c.close();

  }

  catch(Exception e){

    e.printStackTrace();

  }

}

And then in kdb+, create an empty table on disk, and define our upsert function

q):dsPricing set ([id:int$(); date:date$()] open:float$();close:float$();high:float$();low:float$();volume:int$())

`:dsPricing

q)addData:{:dsPricing upsert flip iddateopenclosehighlowvolume!x;}

q)select from `:dsPricing

id date| open close high low volume

-------| --------------------------

then run the java program

$java kx/c

then confirm data is present

q)select from `:dsPricing

id date      | open close high     low volume 

-------------| -------------------------------

1  1994.02.17| 73   73    76       73  2223000

1  1994.02.16| 76   76    77.89999 75  3167000

With 32bit kdb+, you’ll probably want to run with the -g 1 cmd line option to try to keep mem usage low.

4

and with that as a splayed table - just remove the key, and append / to the tablename

q):dsPricing/ set ([]id:int$(); date:date$(); open:float$();close:float$();high:float$();low:float$();volume:int$())

`:dsPricing/

q)addData:{:dsPricing/ upsert flip iddateopenclosehighlowvolume!x;}

q)select from `:dsPricing

id date       open close high     low volume 

1  1994.02.17 73   73    76       73  2223000

1  1994.02.16 76   76    77.89999 75  3167000

5

There are some minor differences between Kx c.java and DEVnet qJava in this example, namely:

  • q date type is represented in qJava as instances of QDate class,
  • QConnection.sync( String query, Object… parameters ) method uses Java varargs. In case your q function takes only a single array parameter, the array in qJava has to be explicitly cast to Object.
    E.g.: q.sync(“addData”, (Object) data);

Please take a look at sample code:

  

public static void main( final String[] args ) throws IOException { final QConnection q = new QBasicConnection("localhost", 5001, "", ""); try { q.open(); Object[] data = new Object[] { new int[] { 1, 1 }, new QDate[] { QDate.fromString("1994.02.17"), QDate.fromString("1994.02.16") }, new double[] { 73., 76. }, new double[] { 73., 76. }, new double[] { 76., 77.899994 }, new double[] { 73., 75. }, new int[] { 2223000, 3167000 } }; q.sync("addData", (Object) data); } catch ( final Exception e ) { System.err.println(e); } finally { q.close(); } }

Best regards

Maciej Lach