[TorQ] .gw.syncexec[] select servers by procname or attribute

Community Support
1

How can I make .gw.syncexec work like .servers.getservers? I have separate hdbs for each stock market. How can I query for prc_stocks but not hk_stocks? .servers.getservers supports selecting servers by procname or attributes. thx

proctype,procname,attribute

hdb,hdb_prc,(enlist product)!(enlist prc_stocks)

hdb,hdb_hk,(enlist product)!(enlist hk_stocks)

2

Should I make 1 proctype for each stock market? What is the intended usage? Thx.

proctype,procname,attribute

hdb_prc,hdb_prc1,(enlist product)!(enlist prc_stocks)

hdb_prc,hdb_prc2,(enlist product)!(enlist prc_stocks)

hdb_hk,hdb_hk1,(enlist product)!(enlist hk_stocks)

hdb_hk,hdb_hk2,(enlist product)!(enlist hk_stocks)

3

What if I have 20 nodes holding hk_stocks and prc_stocks for 2000-2009? 1 node per product per year. .proc.getattributes returns date=Jan 01 to Dec 31. The query is asking for the data of 1 product on 1 day on 1 minute. Would you recommend any design on the proctype/procname/attributes/nodes/instances? Thx.

4

Hi Yan Yan

I think in general if a service has a different capability then it should have a different proctype. The underlying .servers code supports the different attributes capabilities but the current gateway code doesn’t. So you would assume that everything with the same proctype is homogeneous and the gateway will load balance across replicated instances. I think this works reasonably well for most cases eg you have rdb+hdb, or split it a bit more granularly such as by region and/or asset class. 

There comes a point though where it becomes a bit hard to manage - striped databases is one case, especially if the striping changes from time to time in rdbs. It also puts a burden on the client to know which services it has to access - probably not to bad in the rdb/hdb case but difficult in the striped case. 

We have another version of the gateway which we plan to roll into into the main TorQ release. It uses the attributes concept and is backwardly compatible with the current proctype usage. Attributes can be anything though I imagine in most cases they will be a combination of symbol and date. The gateway tries to route the query so it hits as few processes as possible to fulfil the client request in the case where the target processes have overlapping subsets of data. Example queries are at the bottom of the script. 

https://github.com/AquaQAnalytics/TorQ/blob/gateway_reload/code/processes/gateway.q

It’s not in the main release yet as there are a couple of things we need to  work through around how it interacts with the rdb/hdb/wdb processes around the end of day event (i.e. when a database process has to update its attributes). The code is there but not fully signed off. As a gateway to query across processes using attributes it should be good to go. I think this approach should make it easier for clients - for example it’s easier for a client application to know which instruments and dates it is interested in, rather than having to know which processes it needs to get them from. 

Thanks

Jonny

5

ty