Climate Data

Community Support
,
1

https://learninghub.kx.com/forums/topic/climate-data

I’m curious if kdb+/q would be an appropriate solution for spatial time series data. Climate data is typically 4-D and can be quite large. Would like be able to do some in memory computations and also query at much faster speeds than current standard structures (grib or netcdf files with python xarray).

2

Yes it could be used.

 

To test you could look at PyKX for an easy Python interface.

A 2 minute example of passing in a Dataset to q in shown below.

PyKX allows Registering Custom Conversions so you could create a function to pass the Dataset in exactly the form you wish to q instead of passing it all as a dictionary in my example.

 

import pykx as kx 
import xarray as xr 
import numpy as np 
import pandas as pd 
ds = xr.Dataset(     
      {"foo": (("x", "y"), np.random.rand(5, 5))},     
      coords={         "x": [10, 20, 30, 40, 50],         
                       "y": pd.date_range("2000-01-01", periods=5),         
                       "z": ("x", list("abcde")),     }, ) 
kx.q['ds'] = kx.toq(ds.to_dict()) 
kx.q('ds') 
pykx.Dictionary(pykx.q(' 
coords   | `x`y`z!+`dims`attrs`data!((,`x<span class=""hljs-comment"">;,`y;,`x);(()!();()!();()!());(10 20..</span> attrs    | ()!() dims     | `x`y!<span class=""hljs-number"">5</span> <span class=""hljs-number"">5</span> dat<span class=""hljs-built_in"">a_vars</span>| (,`foo)!+`dims`attrs`data!(,`x`y<span class=""hljs-comment"">;,()!();,(0.7412575 0.2054306 0.10..</span> ')) 

kx.q('flip ds[`coords;;`data]') 
pykx.Table(pykx.q(' 
x  y                             z 
---------------------------------- 
<span class=""hljs-number"">10 2000.01</span>.<span class=""hljs-number"">01D00:00:00</span>.<span class=""hljs-number"">000000000</span> a 
<span class=""hljs-number"">20 2000.01</span>.<span class=""hljs-number"">02D00:00:00</span>.<span class=""hljs-number"">000000000</span> b 
<span class=""hljs-number"">30 2000.01</span>.<span class=""hljs-number"">03D00:00:00</span>.<span class=""hljs-number"">000000000</span> c 
<span class=""hljs-number"">40 2000.01</span>.<span class=""hljs-number"">04D00:00:00</span>.<span class=""hljs-number"">000000000</span> d 
<span class=""hljs-number"">50 2000.01</span>.<span class=""hljs-number"">05D00:00:00</span>.<span class=""hljs-number"">000000000</span> e ')) 

kx.q('ds[`data_vars;`foo;`data]') 
pykx.List(pykx.q(' <span class=""hljs-number"">0.7412575</span> <span class=""hljs-number"">0.2054306</span>   <span class=""hljs-number"">0.1009393</span> <span class=""hljs-number"">0.8792678</span> <span class=""hljs-number"">0.04105999</span> <span class=""hljs-number"">0.1811459</span> <span class=""hljs-number"">0.01659637</span>  <span class=""hljs-number"">0.2406029</span> <span class=""hljs-number"">0.4900055</span> <span class=""hljs-number"">0.551788</span>   <span class=""hljs-number"">0.6303767</span> <span class=""hljs-number"">0.0702013</span>   <span class=""hljs-number"">0.6831359</span> <span class=""hljs-number"">0.5961667</span> <span class=""hljs-number"">0.3722388</span>  <span class=""hljs-number"">0.9255059</span> <span class=""hljs-number"">0.9202499</span>   <span class=""hljs-number"">0.5055902</span> <span class=""hljs-number"">0.9767793</span> <span class=""hljs-number"">0.7440498</span>  <span class=""hljs-number"">0.7331576</span> <span class=""hljs-number"">0.003197568</span> <span class=""hljs-number"">0.4939932</span> <span class=""hljs-number"">0.5433492</span> <span class=""hljs-number"">0.01175784</span> '))