Deep neural networks and kdb

Community Support
1

Hello,Has any used one of the popular deep learning frameworks like TensorFlowfor training neural networks with data from kdb? If so, what’s yourwork flow like? Do you export data to CSV files? Interface directlywith C libraries? Use PyQ? Any suggestions would be helpful before Istart experimenting myself.Thanks,Rob

2

charset=“utf-8”

X-Mailer: Microsoft Outlook 14.0
Thread-Index: AQGNU5FpXbL47O38lZOjorJq3gdaL6G0glwA
Content-Language: de

>>> Has any used one of the popular deep learning frameworks like =
TensorFlow for training neural networks with data from kdb?
No, usually it is hard to find someone who knows kdb+ and machine =
learning.

>>> Do you export data to CSV files
No, don=E2=80=99t do it. CSV is not reliable and it shows that you are =
not an expert in kdb+.

>>> Interface directly with C libraries
Yes, please check Qml and Bml as an example.

>>> Use PyQ?
Yes, you can do it. There are a lot of plugins outside to execute python =
code inside kdb+.

>>> Any suggestions would be helpful before I start experimenting =
myself.
It really depends what you want to do or achieve. IMHO if your dataset =
is small like say smaller than 100mb you don=E2=80=99t need to use kdb+. =
You can easily interface kdb+ with python or R and use the libraries in =
the language. These libraries are usually from high quality and they are =
implemented from experts. Moreover both languages also support =
visualization something you cannot find in kdb+ yet. (You cannot do =
machine learning without visualization.) And kdb+ does not support =
visualization directly, not yet. There will be something awesome in the =
future where you can do visualization directly from kdb+ but it is not =
ready yet. Don=E2=80=99t think in dashboard but a nice library in kdb+.=20

So if your dataset is big, say more than 12gb like the dataset from =
https://www.kaggle.com/c/bosch-production-line-performance then you =
cannot use the libraries in python or R natively. Then it makes sense to =
implement a machine learning libraries in kdb+ that can be used directly =
in a hdb. Check xgboost https://github.com/kimtang/xgb as an example.

Kim

-----Urspr=C3=BCngliche Nachricht-----
Von: personal-kdbplus@googlegroups.com =
[mailto:personal-kdbplus@googlegroups.com] Im Auftrag von Rob Smith
Gesendet: Donnerstag, 5. Januar 2017 17:56
An: Kdb+ Personal Developers
Betreff: [personal kdb+] Deep neural networks and kdb

Hello,

Has any used one of the popular deep learning frameworks like TensorFlow =
for training neural networks with data from kdb? If so, what’s your =
work flow like? Do you export data to CSV files? Interface directly =
with C libraries? Use PyQ? Any suggestions would be helpful before I =
start experimenting myself.

Thanks,
Rob

–
You received this message because you are subscribed to the Google =
Groups “Kdb+ Personal Developers” group.
To unsubscribe from this group and stop receiving emails from it, send =
an email to personal-kdbplus+unsubscribe@googlegroups.com.
To post to this group, send email to personal-kdbplus@googlegroups.com.
Visit this group at https://groups.google.com/group/personal-kdbplus.
For more options, visit https://groups.google.com/d/optout.

3

Follow up on this question… I can’t import tensorflow or keras in pyq. Is there any solution that I can use Python deep learning packages from q?

Thanks,

Kaili

4

Use the python interface instead of PyQ

5

sorry, should’ve been:

https://code.kx.com/q/ml/embedpy/

6

I was trying:
\l p.q

.p.e “import tensorflow”

It didn’t work…Modu_leNotFoundError…_

7

Is the module usable with your python environment (not via q invocation) in the first place?

8

Kaili, if you’re using embedPy, the reason for your error is likely the syntax, give this a try for importing Tensorflow;

\l p.q

tf:.p.import[`tensorflow]

If you want to take a look at an example of a Neural Network developed using embedPy and JupyterQ which uses Keras you might be interested in these notebooks and associated blog series. The notebook outlines an installation procedure for tensorflow if that’s potentially an issure. I’ve attached the Neural Network blog as a sample;

https://github.com/KxSystems/mlnotebooks/tree/master/notebooks

https://kx.com/blog/neural-networks-in-kdb-2/

If you’re still having issues with the installation of embedPy and JupyterQ there is an outline of how to install both in both a docker environment and via anaconda, which can be found here.

https://code.kx.com/q/ml/setup/

Conor