Skip to content

Rotating pipeline causes missing data

Wout De Nolf requested to merge 2675-ct-seems-a-bit-slow-2 into master

While reviewing !4694 (closed) I noticed a bug in the rotating proxy manager: the rotation lock was not used when actually rotating! This could cause the rotation to happen while still adding commands to the pipeline. As a result the events added to the pipeline after a rotation will not be send to Redis. This is because the next flush will be on the new pipeline and the old pipeline simply gets garbage collected with buffered commands.

In this MR I fix the bug and add tests

pytest tests/config/test_redis.py::test_rotating_async_proxy

Some time ago I noticed in my Redis benchmark scripts that the reader sometimes doesn't always get the number of expected points when using a rotating pipeline. The benchmark tests redis reading/writing without the actual scans, so pure Redis reading and writing which goes much faster of course. I ran the benchmark again with this fix and the issue is solved.

Closes #2675 (closed) In addition this fix also seems to fix a time overhead issue (!4694 (comment 192016)).

Edited by Wout De Nolf

Merge request reports