[TorQ] Creating new processes

How can I create a new real-time subscriber process that works like rdb or ctp? I would expect it to show up in http://192.168.86.78:6009/.non?monitorui and receive data at the upd function.

I am getting this at the out_xxx.log. The access error is gone if I remove “-U config/passwords/accesslist.txt” from discovery1 in start_torq_demo.sh.

connection to :centos67:6001 failed: access


/ newnode.q

\d .servers

CONNECTIONS:`tickerplant /- list of connections to make at start up

STARTUP:1b                    /- create connections

\d .hb

enabled:1b                    /- disable heartbeating

Can’t exactly remember how I got this to work. Do you need to add a new line to accesslist.txt with mynewprocess:pass ?

Hi,

The access error is thrown because the process you are creating does not have login credentials to connect to the discovery and tickerplant processes.

These login credentials are stored in .txt files in the config/password directory.

Basically you need to have a file (default.txt, <proctype>.txt or <procname>.txt) that has a username:password that is present in accesslist.txt

You can find more info in section 5.3 of the TorQ documentation (https://github.com/AquaQAnalytics/TorQ/blob/master/AquaQTorQ.pdf?raw=true). 

Regards,

Paul

ty

How do I subscribe for quotes? I called .sub.subscribe[;;0b;0b] each .sub.getsubscriptionhandles[tickerplanttypes;();()!()]. But I am just getting a heartbeat every 30 sec.

Hi,

It should work so long as you do something like below.

You need to makes sure that .servers.startup is called to open the connections and the variable tickerplanttypes is defined (should be tickerplant).

Regards,

Paul

q).servers.CONNECTIONS:`tickerplant / - define connections to make

q).servers.startup / - open connections to discovery and .servers.CONNECTIONS

2015.08.14D12:47:31.589032000|localhost.localdomain|rte|rte1|INF|conn|attempting to connect to discovery services

2015.08.14D12:47:31.589082000|localhost.localdomain|rte|rte1|INF|conn|attempting to open handle to :localhost.localdomain:6001

2015.08.14D12:47:31.590058000|localhost.localdomain|rte|rte1|INF|conn|connection to :localhost.localdomain:6001 successful

2015.08.14D12:47:31.590729000|localhost.localdomain|rte|rte1|INF|conn|registering with discovery services

2015.08.14D12:47:31.590753000|localhost.localdomain|rte|rte1|INF|conn|requesting processes from discovery service

2015.08.14D12:47:31.590768000|localhost.localdomain|rte|rte1|INF|conn|querying discovery services for processes of types tickerplant

2015.08.14D12:47:31.592386000|localhost.localdomain|rte|rte1|INF|conn|attempting to open handle to :localhost.localdomain:6000

2015.08.14D12:47:31.592777000|localhost.localdomain|rte|rte1|INF|conn|connection to :localhost.localdomain:6000 successful

q)upd:{show x;show y} / - define upd to print out the table name and data

q)tickerplanttypes:`tickerplant

q).sub.getsubscriptionhandles[tickerplanttypes;();()!()]

procname proctype w

--------------------------

tickerplant1 tickerplant 5

q).sub.subscribe[;;0b;0b] each .sub.getsubscriptionhandles[tickerplanttypes;();()!()]

2015.08.14D12:50:39.382114000|localhost.localdomain|rte|rte1|INF|subscribe|attempting to subscribe to quote,trade on handle 5

2015.08.14D12:50:39.382149000|localhost.localdomain|rte|rte1|INF|subscribe|getting details from the server

2015.08.14D12:50:39.382300000|localhost.localdomain|rte|rte1|INF|subscribe|subscription successful

subtables tplogdate

---------------------

quote trade

q)`trade

time sym price size stop cond ex

-----------------------------------------------------------

2015.08.14D13:50:39.403411000 GOOG 69.92 45 0 C N

2015.08.14D13:50:39.403411000 AIG 19.18 0 0 Z O

2015.08.14D13:50:39.403411000 MSFT 22.62 7 0 B N

2015.08.14D13:50:39.403411000 IBM 59.28 7 0 B N

2015.08.14D13:50:39.403411000 GOOG 70.09 62 0 P N

2015.08.14D13:50:39.403411000 DELL 20.14 29 0 T N

2015.08.14D13:50:39.403411000 AAPL 101.71 77 0 T N

2015.08.14D13:50:39.403411000 MSFT 22.62 42 0 J N

2015.08.14D13:50:39.403411000 HPQ 42.86 23 0 L O

2015.08.14D13:50:39.403411000 IBM 59.13 75 0 K N

2015.08.14D13:50:39.403411000 GOOG 70.12 45 0 N N

2015.08.14D13:50:39.403411000 GOOG 70.05 45 0 8 N

2015.08.14D13:50:39.403411000 DELL 20.15 40 0 Z N

`trade

time sym price size stop cond ex

----------------------------------------------------------

2015.08.14D13:50:39.602416000 AIG 19.17 68 0 8 O

2015.08.14D13:50:39.602416000 HPQ 42.8 28 0 P O

2015.08.14D13:50:39.602416000 HPQ 42.77 34 0 O O

2015.08.14D13:50:39.602416000 AMD 25.09 76 0 C N

2015.08.14D13:50:39.602416000 AIG 19.16 19 0 A O

2015.08.14D13:50:39.602416000 GOOG 70.07 14 0 A N

2015.08.14D13:50:39.602416000 MSFT 22.6 93 0 N N

2015.08.14D13:50:39.602416000 INTC 46.82 73 0 C N

2015.08.14D13:50:39.602416000 GOOG 70.08 56 0 A N

> q).servers.CONNECTIONS:`tickerplant / - define connections to make


How did you open the interactive q session before this line? What scripts and parameters have you passed to the q executable? Thx.

On Friday, August 14, 2015 at 8:56:34 PM UTC+8, Paul Loughran wrote:

Hi,

It should work so long as you do something like below.
You need to makes sure that .servers.startup[] is called to open the connections and the variable tickerplanttypes is defined (should be tickerplant).

Regards,
Paul

q).servers.CONNECTIONS:`tickerplant / - define connections to make
q).servers.startup[] / - open connections to discovery and .servers.CONNECTIONS
2015.08.14D12:47:31.589032000|localhost.localdomain|rte|rte1|INF|conn|attempting to connect to discovery services
2015.08.14D12:47:31.589082000|localhost.localdomain|rte|rte1|INF|conn|attempting to open handle to :localhost.localdomain:6001
2015.08.14D12:47:31.590058000|localhost.localdomain|rte|rte1|INF|conn|connection to :localhost.localdomain:6001 successful
2015.08.14D12:47:31.590729000|localhost.localdomain|rte|rte1|INF|conn|registering with discovery services
2015.08.14D12:47:31.590753000|localhost.localdomain|rte|rte1|INF|conn|requesting processes from discovery service
2015.08.14D12:47:31.590768000|localhost.localdomain|rte|rte1|INF|conn|querying discovery services for processes of types tickerplant
2015.08.14D12:47:31.592386000|localhost.localdomain|rte|rte1|INF|conn|attempting to open handle to :localhost.localdomain:6000
2015.08.14D12:47:31.592777000|localhost.localdomain|rte|rte1|INF|conn|connection to :localhost.localdomain:6000 successful
q)upd:{show x;show y} / - define upd to print out the table name and data
q)tickerplanttypes:`tickerplant
q).sub.getsubscriptionhandles[tickerplanttypes;();()!()]
procname proctype w
--------------------------
tickerplant1 tickerplant 5
q).sub.subscribe[`;`;0b;0b] each .sub.getsubscriptionhandles[tickerplanttypes;();()!()]
2015.08.14D12:50:39.382114000|localhost.localdomain|rte|rte1|INF|subscribe|attempting to subscribe to quote,trade on handle 5
2015.08.14D12:50:39.382149000|localhost.localdomain|rte|rte1|INF|subscribe|getting details from the server
2015.08.14D12:50:39.382300000|localhost.localdomain|rte|rte1|INF|subscribe|subscription successful
subtables tplogdate
---------------------
quote trade
q)`trade
time sym price size stop cond ex
-----------------------------------------------------------
2015.08.14D13:50:39.403411000 GOOG 69.92 45 0 C N
2015.08.14D13:50:39.403411000 AIG 19.18 0 0 Z O
2015.08.14D13:50:39.403411000 MSFT 22.62 7 0 B N
2015.08.14D13:50:39.403411000 IBM 59.28 7 0 B N
2015.08.14D13:50:39.403411000 GOOG 70.09 62 0 P N
2015.08.14D13:50:39.403411000 DELL 20.14 29 0 T N
2015.08.14D13:50:39.403411000 AAPL 101.71 77 0 T N
2015.08.14D13:50:39.403411000 MSFT 22.62 42 0 J N
2015.08.14D13:50:39.403411000 HPQ 42.86 23 0 L O
2015.08.14D13:50:39.403411000 IBM 59.13 75 0 K N
2015.08.14D13:50:39.403411000 GOOG 70.12 45 0 N N
2015.08.14D13:50:39.403411000 GOOG 70.05 45 0 8 N
2015.08.14D13:50:39.403411000 DELL 20.15 40 0 Z N
`trade
time sym price size stop cond ex
----------------------------------------------------------
2015.08.14D13:50:39.602416000 AIG 19.17 68 0 8 O
2015.08.14D13:50:39.602416000 HPQ 42.8 28 0 P O
2015.08.14D13:50:39.602416000 HPQ 42.77 34 0 O O
2015.08.14D13:50:39.602416000 AMD 25.09 76 0 C N
2015.08.14D13:50:39.602416000 AIG 19.16 19 0 A O
2015.08.14D13:50:39.602416000 GOOG 70.07 14 0 A N
2015.08.14D13:50:39.602416000 MSFT 22.6 93 0 N N
2015.08.14D13:50:39.602416000 INTC 46.82 73 0 C N
2015.08.14D13:50:39.602416000 GOOG 70.08 56 0 A N



I can’t get the tick data at the upd function. Any idea? Thx.

user01@centos67 ~/torq

$ find | grep -vP ‘#|log|~’ | grep -iP ‘rtsub|accesslist|process.csv’ | xargs head -1000

==> ./code/processes/rtsub.q <==

\d .rtsub

\d .

.servers.CONNECTIONS:`tickerplant;

.servers.startup;

upd:{show x;show y};

tickerplanttypes:`tickerplant;

.sub.getsubscriptionhandles[tickerplanttypes;();()!()];

.sub.subscribe[;;0b;0b] each .sub.getsubscriptionhandles[tickerplanttypes;();()!()];


==> ./config/passwords/accesslist.txt <==

discovery:pass

feed:pass

gateway:pass

hdb:pass

housekeeping:pass

kill:pass

monitor:pass

rdb:pass

reporter:pass

sort:pass

tickerplant:pass

wdb:pass

chainedtp:pass

rtsub:pass

admin:admin


==> ./config/passwords/rtsub.txt <==

rtsub:pass


==> ./config/nontorqprocess.csv <==

host,port,proctype,procname


==> ./config/process.csv <==

host,port,proctype,procname

localhost,{KDBBASEPORT}+1,discovery,discovery1

localhost,{KDBBASEPORT},tickerplant,tickerplant1

localhost,{KDBBASEPORT}+2,rdb,rdb1

localhost,{KDBBASEPORT}+3,hdb,hdb1

localhost,{KDBBASEPORT}+4,hdb,hdb2

localhost,{KDBBASEPORT}+5,wdb,wdb1

localhost,{KDBBASEPORT}+6,sort,sort1

localhost,{KDBBASEPORT}+7,gateway,gateway1

localhost,{KDBBASEPORT}+8,kill,killtick

localhost,{KDBBASEPORT}+9,monitor,monitor1

localhost,{KDBBASEPORT}+10,tickerlogreplay,tpreplay1

localhost,{KDBBASEPORT}+11,housekeeping,housekeeping1

localhost,{KDBBASEPORT}+12,reporter,reporter1

localhost,{KDBBASEPORT}+13,compression,compression1

localhost,{KDBBASEPORT}+14,feed,feed1

localhost,{KDBBASEPORT}+15,chainedtp,chainedtp1

localhost,{KDBBASEPORT}+16,rtsub,rtsub1



user01@centos67 ~/torq

$ head -v -n1000 start_torq_demo.sh

==> start_torq_demo.sh <==

# Load the environment

. ./setenv.sh


# sets the base port for a default TorQ installation

export KDBHDB=${TORQHOME}/hdb/database

export KDBBASEPORT=6000

export KDBSTACKID=“-stackid ${KDBBASEPORT}”

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$KDBLIB/l32


##### EMAILS #####

# this is where the emails will be sent to

# export DEMOEMAILRECEIVER=user@torq.co.uk


# also set the email server configuration in config/settings/default.q

##### END EMAILS #####


# launch the discovery service

echo ‘Starting discovery proc…’

q torq.q -load code/processes/discovery.q ${KDBSTACKID} -proctype discovery -procname discovery1 -U config/passwords/accesslist.txt -localtime </dev/null >$KDBLOG/torqdiscovery.txt 2>&1 &


# launch the tickerplant, rdb, hdb

echo ‘Starting tp…’

q code/processes/tickerplant.q database ${TORQHOME}/hdb ${KDBSTACKID} -proctype tickerplant -procname tickerplant1 -U config/passwords/accesslist.txt -localtime </dev/null >$KDBLOG/torqtp.txt 2>&1 &


echo ‘Starting rdb…’

q torq.q -load code/processes/rdb.q ${KDBSTACKID} -proctype rdb -procname rdb1 -U config/passwords/accesslist.txt -localtime -g 1 -T 30 </dev/null >$KDBLOG/torqrdb.txt 2>&1 &


echo ‘Starting ctp…’

q torq.q -load code/processes/chainedtp.q ${KDBSTACKID} -proctype chainedtp -procname chainedtp1 -U config/passwords/accesslist.txt -localtime </dev/null >$KDBLOG/torqchainedtp.txt 2>&1 &


echo ‘Starting hdb1…’

q torq.q -load ${KDBHDB} ${KDBSTACKID} -proctype hdb -procname hdb1 -U config/passwords/accesslist.txt -localtime -g 1 -T 60 -w 4000 </dev/null >$KDBLOG/torqhdb1.txt 2>&1 &


echo ‘Starting hdb2…’

q torq.q -load ${KDBHDB} ${KDBSTACKID} -proctype hdb -procname hdb2 -U config/passwords/accesslist.txt -localtime -g 1 -T 60 -w 4000 </dev/null >$KDBLOG/torqhdb2.txt 2>&1 &


# launch the gateway

echo ‘Starting gw…’

q torq.q -load code/processes/gateway.q ${KDBSTACKID} -proctype gateway -procname gateway1 -U config/passwords/accesslist.txt -localtime -g 1 -w 4000 </dev/null >$KDBLOG/torqgw.txt 2>&1 &


# launch the monitor

echo ‘Starting monitor…’

q torq.q -load code/processes/monitor.q ${KDBSTACKID} -proctype monitor -procname monitor1 -localtime </dev/null >$KDBLOG/torqmonitor.txt 2>&1 &


# launch the reporter

echo ‘Starting reporter…’

q torq.q -load code/processes/reporter.q ${KDBSTACKID} -proctype reporter -procname reporter1 -U config/passwords/accesslist.txt -localtime </dev/null >$KDBLOG/torqreporter.txt 2>&1 &


# launch housekeeping

echo ‘Starting housekeeping proc…’

q torq.q -load code/processes/housekeeping.q ${KDBSTACKID} -proctype housekeeping -procname housekeeping1 -U config/passwords/accesslist.txt -localtime </dev/null >$KDBLOG/torqhousekeeping.txt 2>&1 &


# launch sort processes

echo ‘Starting sorting proc…’

q torq.q -load code/processes/wdb.q ${KDBSTACKID} -proctype sort -procname sort1 -U config/passwords/accesslist.txt -localtime -g 1 </dev/null >$KDBLOG/torqsort.txt 2>&1 & # sort process


# launch wdb

echo ‘Starting wdb…’

q torq.q -load code/processes/wdb.q ${KDBSTACKID} -proctype wdb -procname wdb1 -U config/passwords/accesslist.txt -localtime -g 1 </dev/null >$KDBLOG/torqwdb.txt 2>&1 & # pdb process


# launch compress

echo ‘Starting compression proc…’

q torq.q -load code/processes/compression.q ${KDBSTACKID} -proctype compression -procname compression1 -localtime </dev/null >$KDBLOG/torqcompress1.txt 2>&1 & # compression process


# launch feed

echo ‘Starting feed…’

q torq.q -load code/tick/feed.q ${KDBSTACKID} -proctype feed -procname feed1 -localtime </dev/null >$KDBLOG/torqfeed.txt 2>&1 &


# rtsub

echo ‘Starting rtsub…’

q torq.q -load code/processes/rtsub.q ${KDBSTACKID} -proctype rtsub -procname rtsub1 -U config/passwords/accesslist.txt -localtime -g 1 -T 30 </dev/null >$KDBLOG/torqrtsub.txt 2>&1 &


user01@centos67 ~/torq

$ tail logs/*rtsub1.log

==> logs/err_rtsub1.log <==


==> logs/out_rtsub1.log <==

# #

################################################################################

2015.08.17D11:38:34.659732000|centos67.yanyan.com|rtsub|rtsub1|INF|conn|attempting to connect to discovery services

2015.08.17D11:38:34.659768000|centos67.yanyan.com|rtsub|rtsub1|INF|conn|attempting to open handle to :centos67.yanyan.com:6001

2015.08.17D11:38:34.670402000|centos67.yanyan.com|rtsub|rtsub1|INF|conn|connection to :centos67.yanyan.com:6001 successful

2015.08.17D11:38:34.672815000|centos67.yanyan.com|rtsub|rtsub1|INF|conn|registering with discovery services

2015.08.17D11:38:34.672858000|centos67.yanyan.com|rtsub|rtsub1|INF|conn|requesting processes from discovery service

2015.08.17D11:38:34.672915000|centos67.yanyan.com|rtsub|rtsub1|INF|conn|querying discovery services for processes of types tickerplant

2015.08.17D11:41:03.940222000|centos67.yanyan.com|rtsub|rtsub1|INF|conn|attempting to open handle to :centos67.yanyan.com:6000

2015.08.17D11:41:03.940536000|centos67.yanyan.com|rtsub|rtsub1|INF|conn|connection to :centos67.yanyan.com:6000 successful


==> logs/usage_rtsub1.log <==

2015.08.17D11:56:33.396779000|373|0N|ps|rtsub|rtsub1|"b"|-1062709672i|user01|0i|“.hb.publishheartbeat”|2645632 67108864 67108864 0 0|0N|“”

2015.08.17D11:56:33.397087000|373|114|ps|rtsub|rtsub1|"c"|-1062709672i|user01|0i|“.hb.publishheartbeat”|2645792 67108864 67108864 0 0|10|“”

2015.08.17D11:56:36.597051000|374|0N|ps|rtsub|rtsub1|"b"|-1062709672i|user01|0i|“.hb.checkheartbeat”|2645888 67108864 67108864 0 0|0N|“”

2015.08.17D11:56:36.597708000|374|204|ps|rtsub|rtsub1|"c"|-1062709672i|user01|0i|“.hb.checkheartbeat”|2646048 67108864 67108864 0 0|10|“”

2015.08.17D11:56:36.598934000|375|0N|ps|rtsub|rtsub1|"b"|-1062709672i|user01|0i|“.sub.checksubscriptions”|2646144 67108864 67108864 0 0|0N|“”

2015.08.17D11:56:36.599440000|375|351|ps|rtsub|rtsub1|"c"|-1062709672i|user01|0i|“.sub.checksubscriptions”|2646304 67108864 67108864 0 0|10|“”

2015.08.17D11:56:46.596283000|376|0N|ps|rtsub|rtsub1|"b"|-1062709672i|user01|0i|“.sub.checksubscriptions”|2646400 67108864 67108864 0 0|0N|“”

2015.08.17D11:56:46.596497000|376|35|ps|rtsub|rtsub1|"c"|-1062709672i|user01|0i|“.sub.checksubscriptions”|2646560 67108864 67108864 0 0|10|“”

2015.08.17D11:56:46.797315000|377|0N|ps|rtsub|rtsub1|"b"|-1062709672i|user01|0i|“.hb.checkheartbeat”|2646656 67108864 67108864 0 0|0N|“”

2015.08.17D11:56:46.797558000|377|66|ps|rtsub|rtsub1|"c"|-1062709672i|user01|0i|“.hb.checkheartbeat”|2646816 67108864 67108864 0 0|10|“”

Hi Yan 

I think the problem is that when you add the real time subscriber to the bottom it is trying to connect but fails due to a timeout, but the log message you are seeing is the automatic reconnect on a timer.  A bit further up the log file there is this:

2015.08.17D08:54:22.413968000|aquaq-dev|rtsub|rtsub1|INF|fileload|loading code/processes/rtsub.q

2015.08.17D08:54:22.414913000|aquaq-dev|rtsub|rtsub1|INF|conn|attempting to connect to discovery services

2015.08.17D08:54:22.414972000|aquaq-dev|rtsub|rtsub1|INF|conn|attempting to open handle to :aquaq-dev:6001

2015.08.17D08:54:24.420410000|aquaq-dev|rtsub|rtsub1|INF|conn|connection to :aquaq-dev:6001 failed: timeout

You can do add a sleep in the start script before the process is launched, and/or increase the hopen timeout.  Better though might be to do something similar to the RDB and WDB processes - to block the process until a tickerplant connection is successfully created.  This means that the processes can start up in any order, and the process will not appear active/queryable until it has subscribed to the TP. 

So the code might look something like this: 

\d .rtsub

\d .

.servers.CONNECTIONS:`tickerplant;

/ - function to check that the tickerplant is connected and subscription has been setup

upd:{show x;show y};

tickerplanttypes:`tickerplant;

/-check if the tickerplant has connected, block the process until a connection is established

while[0 = count select from .sub.SUBSCRIPTIONS where proctype in tickerplanttypes, active;

        /-while no connected make the process sleep for X seconds and then run the subscribe function again

        .os.sleep 2;

        /-run the servers startup code again (to make connection to discovery)

        .servers.startup;

        .sub.subscribe[;;0b;0b] each .sub.getsubscriptionhandles[tickerplanttypes;();()!()]] 

Thanks 

Andrew

thx andrew. sleep 5 failed but sleep 10 worked. why doesnt it retry as described in page 28 chapter 5.1?

# ./start_torq_demo.sh

launch rtsub

sleep 10

netstat -anp | grep LISTEN | grep -iP --color ‘600[01]’

echo ‘Starting rtsub…’

q torq.q -load code/processes/rtsub.q ${KDBSTACKID} -proctype rtsub -procname rtsub1 -U config/passwords/accesslist.txt -localtime -g 1 -T 30 </dev/null >$KDBLOG/torqrtsub.txt \

2>&1 &

Hi Yan,
It does retry the connection but not the subscriptions. It has to make the subscription before it is able to retry it. The code I posted above is a more reliable way as it can retry the subscription until it is successful unlike using sleep.

Thanks

Andrew 

  1. connected

  2. subscribed

  3. network failure

  4. network resumed

  5. connection retry OK

  6. will subscription be retried and resumed?

If you set .sub.AUTORECONNECT variable to true, then it will automatically resubscribe when it remakes the connection.  It does this by modifying the .servers.connectcustom method - this is invoked whenever a connection is (re-)established.

However, we’ve just been testing this and noticed a bug if the symbol list is set to ` (it works fine for defined subscription lists).  We’ll get that fixed up this afternoon.

Also note though that you may have to do something custom if you can recover data e.g. from a TP log file - maybe on a re-subscription you might need to dump the current data set and replay it all from the TP log file.

Andrew

Hi Yan,

The bug is fixed now. So it should work fine now as laid out before.

Thanks

Andrew