q/kdb+ api for getting market and financial data from IEX

Figured I would post this here as many of you might find this useful.

This weekend, I wrote a library to get IEX data in kdb. You can get all sorts of data including EOD summary (high, low, open, close, vwap), earnings (actual vs consensus EPS), financials and news. 

Code: https://github.com/himoacs/iex\_q

Examples: http://www.enlistq.com/qkdb-api-getting-market-financial-data-iex/

P.S. Through this project, I learnt that q has an operator for parsing json messages which I wasn’t expecting. I just wish it had a more descriptive name than -29!. :)

thanks Himanshu,

json support is available in .j

 http://code.kx.com/q/ref/dotj/

nice work on the iex library.

What is the best way to “unnest” t?

I would like a 4 column table with cols: symmcappeebitda

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Menlo; color: #d0d2da; background-color: #161821; background-color: rgba(22, 24, 33, 0.95)}span.s1 {font-variant-ligatures: no-common-ligatures}

q)t:.j.k .Q.hg`$“https://api.iextrading.com/1.0/stock/market/batch?symbols=aapl,googl,amzn,fb&types=quote,stats&filter=marketCap,peRatio,EBITDA

q)

q)t

     | quote                                  stats                 

---- | -------------------------------------------------------------

AAPL | marketCappeRatio!8.983506e+11 19.88  (,`EBITDA)!,7.6569e+10

GOOGL| marketCappeRatio!7.340723e+11 35.37  (,`EBITDA)!,3.2714e+10

AMZN | marketCappeRatio!5.715004e+11 301.02 (,`EBITDA)!,1.4021e+10

FB   | marketCappeRatio!5.311237e+11 35.22  (,`EBITDA)!,2.0304e+10

q)

q)type t

99h

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Menlo; color: #d0d2da; background-color: #161821; background-color: rgba(22, 24, 33, 0.95)}span.s1 {font-variant-ligatures: no-common-ligatures} p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Menlo; color: #d0d2da; background-color: #161821; background-color: rgba(22, 24, 33, 0.95)}span.s1 {font-variant-ligatures: no-common-ligatures} p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Menlo; color: #d0d2da; background-color: #161821; background-color: rgba(22, 24, 33, 0.95)}span.s1 {font-variant-ligatures: no-common-ligatures}

Here’s one way to do it:<o:p></o:p>

<o:p> </o:p>

q)symmcappeebitda xcol ( sym:key t),'exec (quote,'stats) from value t<o:p></o:p>

sym   mcap         pe     ebitda<o:p></o:p>

------------------------------------<o:p></o:p>

AAPL  8.983506e+11 19.88  7.6569e+10<o:p></o:p>

GOOGL 7.340723e+11 35.37  3.2714e+10<o:p></o:p>

AMZN  5.715004e+11 301.02 1.4021e+10<o:p></o:p>

FB    5.311237e+11 35.22  2.0304e+10<o:p></o:p>

<o:p> </o:p>

Hope this helps<o:p></o:p>

Jonathon<o:p></o:p>

<o:p> </o:p>

From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of ag
Sent: 27 November 2017 08:14
To: Kdb+ Personal Developers <personal-kdbplus@googlegroups.com>
Subject: [personal kdb+] Re: q/kdb+ api for getting market and financial data from IEX<o:p></o:p>

<o:p> </o:p>

What is the best way to “unnest” t?<o:p></o:p>

I would like a 4 column table with cols: symmcappeebitda<o:p></o:p>

<o:p> </o:p>

q)t:.j.k .Q.hg`$“https://api.iextrading.com/1.0/stock/market/batch?symbols=aapl,googl,amzn,fb&types=quote,stats&filter=marketCap,peRatio,EBITDA”<o:p></o:p>

q)<o:p></o:p>

q)t<o:p></o:p>

     | quote                                  stats                 <o:p></o:p>

---- | -------------------------------------------------------------<o:p></o:p>

AAPL | marketCappeRatio!8.983506e+11 19.88  (,`EBITDA)!,7.6569e+10<o:p></o:p>

GOOGL| marketCappeRatio!7.340723e+11 35.37  (,`EBITDA)!,3.2714e+10<o:p></o:p>

AMZN | marketCappeRatio!5.715004e+11 301.02 (,`EBITDA)!,1.4021e+10<o:p></o:p>

FB   | marketCappeRatio!5.311237e+11 35.22  (,`EBITDA)!,2.0304e+10<o:p></o:p>

q)<o:p></o:p>

q)type t<o:p></o:p>

99h<o:p></o:p>


Submitted via Google Groups

Amazing! This is great. I am reading through the exec documentation but there isn’t a full description, just examples.
It seems to pull out the contents from the dictionary key - is that correct?
http://code.kx.com/q/ref/qsql/#exec

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Menlo; color: #d0d2da; background-color: #161821; background-color: rgba(22, 24, 33, 0.95)}span.s1 {font-variant-ligatures: no-common-ligatures}

q)foo: flipab`c!flip(0 0 0;1 2 3;2 4 6)

q)foo

a b c

-----

0 0 0

1 2 3

2 4 6

q)exec a from value foo

'type

[0] exec a from value foo

^

q)exec a from foo

0 1 2

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px Menlo; color: #d0d2da; background-color: #161821; background-color: rgba(22, 24, 33, 0.95)}span.s1 {font-variant-ligatures: no-common-ligatures}

( sym:key t),'exec quote from value t

On Monday, November 27, 2017 at 3:40:38 AM UTC-8, Jonathon McMurray wrote:

Here’s one way to do it:

 

q)symmcappeebitda xcol ( sym:key t),'exec (quote,'stats) from value t

sym   mcap         pe     ebitda

------------------------------------

AAPL  8.983506e+11 19.88  7.6569e+10

GOOGL 7.340723e+11 35.37  3.2714e+10

AMZN  5.715004e+11 301.02 1.4021e+10

FB    5.311237e+11 35.22  2.0304e+10

 

Hope this helps

Jonathon

 

From: personal…@googlegroups.com [mailto:personal…@googlegroups.com] On Behalf Of ag
Sent: 27 November 2017 08:14
To: Kdb+ Personal Developers <personal…@googlegroups.com>
Subject: [personal kdb+] Re: q/kdb+ api for getting market and financial data from IEX

 

What is the best way to “unnest” t?

I would like a 4 column table with cols: symmcappeebitda

 

q)t:.j.k .Q.hg`$“https://api.iextrading.com/1.0/stock/market/batch?symbols=aapl,googl,amzn,fb&types=quote,stats&filter=marketCap,peRatio,EBITDA

q)

q)t

     | quote                                 stats                 

---- | -------------------------------------------------------------

AAPL | marketCappeRatio!8.983506e+11 19.88 (,`EBITDA)!,7.6569e+10

GOOGL| marketCappeRatio!7.340723e+11 35.37 (,`EBITDA)!,3.2714e+10

AMZN | marketCappeRatio!5.715004e+11 301.02 (,`EBITDA)!,1.4021e+10

FB   | marketCappeRatio!5.311237e+11 35.22 (,`EBITDA)!,2.0304e+10

q)

q)type t

99h


Submitted via Google Groups

So basically the object “t” from your previous example (the result of JSON parsing by .j.k) is a dictionary where the “values” are dictionaries - and remember, in KDB a list of dictionaries is a table:

q)value t
quote                                  stats

marketCappeRatio!8.938324e+11 19.78  (,EBITDA)!,7.6569e+10 marketCappeRatio!7.448348e+11 35.89&nbsp; (,EBITDA)!,3.2714e+10
marketCappeRatio!5.762372e+11 303.51 (,EBITDA)!,1.4021e+10 marketCappeRatio!5.318502e+11 35.27&nbsp; (,EBITDA)!,2.0304e+10

So this is what the “exec” statement in my example is operating on. exec extracts columns from a table - if you exec a single column, the return will be a list. In my previous example, I use join-each (,') to combine the two dictionary columns and exec them as one - therefore the return is a list. As these columns are dictionary columns, when we exec this list, we get a list of dictionaries, i.e. a table

q)exec (quote,'stats) from value t

marketCap    peRatio EBITDA


8.938324e+11 19.78   7.6569e+10

7.448348e+11 35.89   3.2714e+10

5.762372e+11 303.51  1.4021e+10

5.318502e+11 35.27   2.0304e+10

>From here it’s fairly trivial to use join-each to combine this with a table containing the sym values, and xcol is used finally to rename the columns as you requested

Hope that helps

Jonathon


From: personal-kdbplus@googlegroups.com <personal-kdbplus@googlegroups.com> on behalf of ag <anuj.goyal@gmail.com>
Sent: 27 November 2017 17:19
To: Kdb+ Personal Developers
Subject: Re: [personal kdb+] Re: q/kdb+ api for getting market and financial data from IEX
 

Amazing! This is great. I am reading through the exec documentation but there isn’t a full description, just examples.
It seems to pull out the contents from the dictionary key - is that correct?
http://code.kx.com/q/ref/qsql/#exec

q)foo: flipab`c!flip(0 0 0;1 2 3;2 4 6)

q)foo

a b c

-----

0 0 0

1 2 3

2 4 6

q)exec a from value foo

'type

[0] exec a from value foo

^

q)exec a from foo

0 1 2

( sym:key t),'exec quote from value t

On Monday, November 27, 2017 at 3:40:38 AM UTC-8, Jonathon McMurray wrote:

Here?s one way to do it:

 

q)symmcappeebitda xcol ( sym:key t),'exec (quote,'stats) from value t

sym   mcap         pe     ebitda

------------------------------------

AAPL  8.983506e+11 19.88  7.6569e+10

GOOGL 7.340723e+11 35.37  3.2714e+10

AMZN  5.715004e+11 301.02 1.4021e+10

FB    5.311237e+11 35.22  2.0304e+10

 

Hope this helps

Jonathon

 

From: personal…@googlegroups.com [mailto:personal…@googlegroups.com] On Behalf Of ag
Sent: 27 November 2017 08:14
To: Kdb+ Personal Developers <personal…@googlegroups.com>
Subject: [personal kdb+] Re: q/kdb+ api for getting market and financial data from IEX

 

What is the best way to “unnest” t?

I would like a 4 column table with cols: symmcappeebitda

 

q)t:.j.k .Q.hg`$“https://api.iextrading.com/1.0/stock/market/batch?symbols=aapl,googl,amzn,fb&types=quote,stats&filter=marketCap,peRatio,EBITDA

q)

q)t

     | quote                                  stats                 

---- | -------------------------------------------------------------

AAPL | marketCappeRatio!8.983506e+11 19.88  (,`EBITDA)!,7.6569e+10

GOOGL| marketCappeRatio!7.340723e+11 35.37  (,`EBITDA)!,3.2714e+10

AMZN | marketCappeRatio!5.715004e+11 301.02 (,`EBITDA)!,1.4021e+10

FB   | marketCappeRatio!5.311237e+11 35.22 (,`EBITDA)!,2.0304e+10

q)

q)type t

99h


Submitted via Google Groups

Hello Himanshu,

Execellent work on IEX.

I am trying to run one of the function from iex_q.q on my windows 10 laptop. getting below error.

it seems some setup to be done before we run this. please help me out.

q)get_last_trade[IBM] 'conn. OS reports: The requested protocol has not been configured into the system, or no implementation for it exists. &nbsp; [2]&nbsp; C:\q\w32\iex\_q.q:19: get\_data:{[main\_url;suffix;prefix;char\_delta;identifier] &nbsp; result: ($“:https://”,main_url) suffix," ",prefix;
          ^
  (char_delta + first result ss identifier) _ result
q))\

Hi Anand<o:p></o:p>

<o:p> </o:p>

Looks like you need to set up OpenSSL. You can download the installer for Windows from here: https://slproweb.com/products/Win32OpenSSL.html (you probably want to download “Win32 OpenSSL v1.1.0g Light”)<o:p></o:p>

<o:p> </o:p>

You’ll also need a certificate file – you can download this from here: https://curl.haxx.se/ca/cacert.pem<o:p></o:p>

<o:p> </o:p>

Finally in your command prompt, before running q, you need to run this command to set an environment variable:<o:p></o:p>

<o:p> </o:p>

set SSL_CA_CERT_FILE=C:\path\to\cacert.pem<o:p></o:p>

<o:p> </o:p>

(replacing path with correct path to wherever you downloaded cacert.pem)<o:p></o:p>

<o:p> </o:p>

Note the above “set” command will only be valid for the current command prompt session. To set it permanently, use the following instead:<o:p></o:p>

<o:p> </o:p>

setx SSL_CA_CERT_FILE C:\path\to\cacert.pem<o:p></o:p>

<o:p> </o:p>

This won’t set the variable in the current session, but will set it for all future sessions.<o:p></o:p>

<o:p> </o:p>

After doing these setup steps, HTTPS downloads should function correctly on Windows<o:p></o:p>

<o:p> </o:p>

Regards<o:p></o:p>

Jonathon<o:p></o:p>

<o:p> </o:p>

From: personal-kdbplus@googlegroups.com [mailto:personal-kdbplus@googlegroups.com] On Behalf Of Anand C
Sent: 03 January 2018 10:18
To: Kdb+ Personal Developers <personal-kdbplus@googlegroups.com>
Subject: [personal kdb+] Re: q/kdb+ api for getting market and financial data from IEX<o:p></o:p>

<o:p> </o:p>

Hello Himanshu,<o:p></o:p>

<o:p> </o:p>

Execellent work on IEX.<o:p></o:p>

<o:p> </o:p>

I am trying to run one of the function from iex_q.q on my windows 10 laptop. getting below error.<o:p></o:p>

<o:p> </o:p>

it seems some setup to be done before we run this. please help me out.<o:p></o:p>

<o:p> </o:p>

q)get_last_trade[IBM] 'conn. OS reports: The requested protocol has not been configured into the system, or no implementation for it exists. &nbsp; [2]&nbsp; C:\q\w32\iex\_q.q:19: get\_data:{[main\_url;suffix;prefix;char\_delta;identifier] &nbsp; result: ($“:https://”,main_url) suffix," ",prefix;
          ^
  (char_delta + first result ss identifier) _ result
q))<o:p></o:p>

On Monday, November 13, 2017 at 12:00:15 PM UTC, Himanshu Gupta wrote:<o:p></o:p>

Figured I would post this here as many of you might find this useful.<o:p></o:p>

<o:p> </o:p>

This weekend, I wrote a library to get IEX data in kdb. You can get all sorts of data including EOD summary (high, low, open, close, vwap), earnings (actual vs consensus EPS), financials and news. <o:p></o:p>

<o:p> </o:p>

Code: https://github.com/himoacs/iex_q<o:p></o:p>

Examples: http://www.enlistq.com/qkdb-api-getting-market-financial-data-iex/<o:p></o:p>

<o:p> </o:p>

P.S. Through this project, I learnt that q has an operator for parsing json messages which I wasn’t expecting. I just wish it had a more descriptive name than -29!. :)<o:p></o:p>


Submitted via Google Groups

I’ve tried following these instructions as well as the kx SSL cookbook several times but cannot get this working for the life of me. I can’t get past the following error:

OS reports: The requested protocol has not been configured into the system, or no implementation for it exists.

I have tried many versions of openssl and verified they work from the command line before running q, made sure the required windows DLLs are correctly on the PATH.  I also have tried setting SSL_VERIFY_SERVER to “NO”. Nothing works, I keep getting this same error.

I’m on Windows 10. 

Does anyone know how I might be able to proceed to debug things from here to get this working?

justin, are you using the 32 or 64bit version of kdb+? Please can you include the startup banner in your email?

You’ll need to install 32bit ssl libs if using 32bit kdb+, and 64bit ssl libs if using 64bit kdb+.

hth,

Charlie

I’m using the 32 bit version, banner is as follows:

Welcome to kdb+ 32bit edition

For support please see http://groups.google.com/d/forum/personal-kdbplus

Tutorials can be found at http://code.kx.com/wiki/Tutorials

To exit, type \

To remove this startup msg, edit q.q

I just tried uninstalling all versions of openssl and then installing it fresh, version “Win32 OpenSSL v1.1.0i” but am still getting the same error. 

ok, as per note from https://code.kx.com/q/cookbook/ssl/#suitability-and-restrictions

kdb+ does not yet work with openssl 1.1, you’ll need an older version -

https://slproweb.com/download/Win32OpenSSL-1_0_2p.exe


btw, if you prefer the 64bit kdb+, you can apply for a kod personal license from https://ondemand.kx.com

That did it, everything’s working now, thanks so much!

May I also get some help please with setting-up SSL:

 

q).Q.hg"https://www.google.com" 'conn. OS reports: The requested protocol has not been configured into the system, or no implementation for it exists. [0] .Q.hg"https://www.google.com" ^ q))

 

 

I’ve 64-bit kdb+ installed and I’ve installed 64-bit OpenSSL version:Win64 OpenSSL v3.2.0 Light
I have also downloaded the certificate from here:https://curl.haxx.se/ca/cacert.pem 
And I’ve set the Environment Variable to be the path of above cacert.pem: SSL_CA_CERT_FILE:C:\Users\newbie\OneDrive\Desktop\kdb+\cacert.pem

I’ve also added 2 paths to Path variable: 
C:\Program Files\OpenSSL-Win64\bin\libcrypto-3-x64.dll
C:\Program Files\OpenSSL-Win64\bin\libssl-3-x64.dll

In fact, I get this:

q)(-26!) 'Could not initialize openssl. Error was The specified module could not be found. [0] (-26!) ^ q)

It still doesn’t work, may I have more hints on what can be checked, thank you!