HI all, we are working on a system where some data need to publish from KDB to external system, as far as I aware, there are 2 approaches of getting that done
Any comments about under which approach is better at what situation? and Why?
This is a little too general a question IMO.
However, in a nutshell consider
How many subscribers are there?
Do they each have their own topic list, or do they all receive the same data? e.g. point to point, or point to multipoint.
Do they already have a preferred bus to receive data via?
Do they require guaranteed delivery of data?
Do they need to consume data from other publishers?
(If you want to keep it simple, use kdb+ pub/sub)
Depending on what the external system is, and the volume of data, you might also consider round tripping through files.
Thanks a lot Charles, below is more detail of my situation:
How many subscribers are there?
It should be around 10 subscribers
Do they each have their own topic list, or do they all receive the same data? e.g. point to point, or point to multipoint.
It is point to point
Do they already have a preferred bus to receive data via?
No.
Do they require guaranteed delivery of data?
Yes
Do they need to consume data from other publishers?
Yes
(If you want to keep it simple, use kdb+ pub/sub)
Yes, that also what I think, I would like to use kdb+ pub/sub, one concern is about the compression of kdb+, the network is not really reliable at my environment, will kdb+ compression less efficiency then, say, rv?
Regards,
Carfield
kdb+ ipc compression can be tested with -18!x.
e.g.
q)count -18!10000#0
682
versus no compression
q)count -8!10000#0
80014
you can of course time both serliaztion and deserialization to get an idea of the cpu overhead
q)v:10000#0
q)\ts do[100;-18!v]
12 196864
q)\ts do[100;-8!v]
1 131328
q)z:-18!v
q)\ts do[100;-9!z]
12 262400
q)z:-8!v
q)\ts do[100;-9!z]
1 131328
I don’t know whether or how much rv compresses.
kdb+ compresses data over tcp/ip connections if