CPU intensive Redis listeners
So the reason ScanDataListener (F5), Flint and the writer show unusually high CPU usage is due to the frantic XREAD loop in bliss.config.streaming.DataStreamReader._read_task_main
. This was diagnosed in #2319 (closed). It reads everything that is available in the subscribed Redis streams, queue's that data for the consumer greenlet to process and calls XREAD again. This loop is in fact too fast, so it reads a small amount of data lots of times.
The proposed solution is the reduce the rate at which data is published in Redis by buffering the data on the Bliss side (using the Redis pipeline mechanism). The question is, what are we going to use to determine the publishing rate:
- time: publish buffered data when a fixed time period has passed (!3162 (closed))
- number of stream event: publish buffered data when one channel has buffered x points or more (remember each stream can contain a maximum of 2048 events)
- data size: publish buffered data when the total buffered size has reached x MB or more
At first I thought option 2 was best (because of the Redis stream limit). However this made me question the way data is buffered on the Bliss side. When publishing channel data, we should publish only 1 event per channel. So the buffering is done in the events, not in the pipeline execution. And in that case option 3 might be the best. In either case, option 1 is easy to implement but error prone imo.