Slow Consumer .z.W

Community Support
1

Hi All,

Can someone help me understand the outbound queue in .z.W a bit more.  If a tickerplant is using .u.pub which is async pub to all handles, can someone help me understand how a slow consumer affects other subscribers?  So if there is a build up in bytes in the outbound queue in .z.W, then how does the tickerplant read off of that queue? Does it read off the top and then continue pub to all subscribers, or will the tickerplant try to pub all messages in the outbound queue for the slow consumer before trying to pub to other subscribers?

Thanks!

2

Hi Roni,

Hope you’re doing great! :)

Please check the stack-overflow link  - https://stackoverflow.com/questions/60797066/can-slow-real-time-subscriber-kill-tickerplant-in-kdb

Thanks

Rohit Sharma

3

Hi,

A slow subscriber does not directly affect publishing to other subscribers (this is why async communication is used, otherwise the performance of the tickerplant as a whole will be throttled to that of the slow subscriber), but depending on the state of the receive buffer on the slow subscriber & some other settings, the tickerplant may not send anymore data to that subscriber, causing that data to be instead buffered on the tickerplant. The tickerplant heap is now effectively being controlled by the slow subscriber & if this is not addressed, eventually the heap limit will be reached and the tickerplant will deliberately abort.

kdb+ processes at most one message from each socket per spin of the main loop in order of the socket descriptor value on Unix

Some more info 

https://stackoverflow.com/questions/1997691/what-happens-when-tcp-udp-server-is-publishing-faster-than-client-is-consuming 

https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Flow_control

Jason

4

Thanks for this Jason,

quick question, you said depending on the state of the receive buffer on the slow consuer and “some other settings” can elaborate a bit on the other settings?

Also, are there any other buffer options available in kdb or its just limited to the heap limit, right?

Thanks!

5

Hi Rohit, yes thanks have seen that.  Was wondering a little more about how the messages were read off the outbound queue.  Jason’s answer explained.<o:p></o:p>

<o:p> </o:p>

Thanks<o:p></o:p>

<o:p> </o:p>

From: personal-kdbplus@googlegroups.com <personal-kdbplus@googlegroups.com> On Behalf Of Rohit Kumar
Sent: April 22, 2020 12:32 PM
To: Kdb+ Personal Developers <personal-kdbplus@googlegroups.com>
Subject: [personal kdb+] Re: Slow Consumer .z.W<o:p></o:p>

<o:p> </o:p>

Hi Roni,<o:p></o:p>

<o:p> </o:p>

Hope you’re doing great! :)<o:p></o:p>

<o:p> </o:p>

Please check the stack-overflow link  - https://stackoverflow.com/questions/60797066/can-slow-real-time-subscriber-kill-tickerplant-in-kdb<o:p></o:p>

<o:p> </o:p>

Thanks<o:p></o:p>

Rohit Sharma