Loading a database from GCS storage

Community Support
1

Hi 

I am trying to load a database using GCS. 

  • I have read and followed  this documentation
  • I have created a database (by the example): 

d:2021.09.01+til 20
{[d;n]sv[;.Q.par[:test/db/;d;trade],]set .Q.en[:test/;([]sym:$'n?.Q.A;time:.z.P+til n;price:n?100f;size:n?50)];}[;10000]each d

 

  • I have uploaded it to GCS.

This is how my bucket looks: 

lior_kdb/db/

  • I have set environment variable GCLOUD_PROJECT_ID to the correct project.

 

  • This is my local folder:

 

  • The contents of par.txt 

$ cat par.txt
gs://lior_kdb/db

 

 

  • When I try to load this way:

q)\l db

Thee is no error.

 

  • When I try to access the data, I get an error

q)t:get$":gs://lior\_kdb/db/2021.09.01/trade/" 'gs://lior\_kdb/db/2021.09.01/trade/. OS reports: No such file or directory [0] t:get$“:gs://lior_kdb/db/2021.09.01/trade/”

 

All other attempts to access the database also fail.

I am using KDB+ 4.0 2023.08.11 installed on Ubuntu 20.04

 

Could you please assist,

Thank you,

Lior

 

 

 

 

 

 

 

2

Can you try again after setting this environment variable?

export KX_TRACE_OBJSTR=1

Does it print any additional info?

3

Hi, 

Thanks for your reply. 

I am getting the same results: 

 

q)\l db
q)t:get$":gs://lior\_kdb/db/2021.09.01/trade/" 'gs://lior\_kdb/db/2021.09.01/trade/. OS reports: No such file or directory [0] t:get$“:gs://lior_kdb/db/2021.09.01/trade/”

q)select from $":gs://lior\_kdb/db/2021.09.01/trade/" 'gs://lior\_kdb/db/2021.09.01/trade/. OS reports: No such file or directory [0] select from $“:gs://lior_kdb/db/2021.09.01/trade/”

Just to confirm that the data is there I double checked

$gsutil ls gs://lior_kdb/db/2021.09.01/trade/
gs://lior_kdb/db/2021.09.01/trade/.d
gs://lior_kdb/db/2021.09.01/trade/price
gs://lior_kdb/db/2021.09.01/trade/size
gs://lior_kdb/db/2021.09.01/trade/sym
gs://lior_kdb/db/2021.09.01/trade/tim

4

Are you using the Insights Core version of kdb+ instead of the vanilla one (which lacks the cloud-related features such as object storage)?

5

Hi I am using the version available at GCP

KDB+ 4.0 Cloud Edition 2021.03.12 Copyright (C) 1993-2021 Kx Systems
l64/ 8()core 64297MB   kx-insights-1-vm 10.128.0.4 EXPIRE 2024.06.02 654962966702 GCP KXCE #5014615

 

6

Previously you mentioned using KDB+ 4.0 2023.08.11 installed on Ubuntu 20.04. But in your latest reply the release date is 2021.03.12. So you were probably using two different versions for two different tests. Can you try using the latest version of Cloud Edition as it should have some bug fixes related to object storage?

7

I am using cloud edition for production and integration.

I have another installation for development only. 

The results are the same on both installations. 

I will create another cloud instance with the latest edition and report.

Edit: I can can confirm that the current version available in the market:

KDB+ 4.0 Cloud Edition 2023.01.20 Copyright (C) 1993-2023 Kx Systems

solves the issue. 

 

Thanks for your help!

L.