Change data_handler arg npoints to shape
This is to be a bit more generic for the multi scan case
Merge request reports
Activity
requested review from @denolf
What you are anticipating is
shape = (nscans, npoints)
right? But the buffers only knownpoints
so they can never pass(nscans, npoints)
to the handlers.nscans
will be equal to the number ofDatasetMsg
messages you receive. But in fact you knownscans
even before that, just from the task inputs.What you are anticipating is
shape = (nscans, npoints)
right?Yes, right.
My idea is to have the data handlers populating a slice of the datasets instead of the whole dataset. But for this, I need to initialize the dataset at the first
DatasetMsg
message received, hence the need for the full shape.But you are right, this might not work if buffers cannot send the full shape
Edited by Loic HuderYes the buffers don't know the full shape. They can only pass
npoints
. The code that receivesDatasetMsg
knows the number of scans so it scan create the fullshape = (nscans, npoints)
from the firstDatasetMsg.npoints
.Edited by Wout De NolfThat would only work for single-detector, no ? Since there will be as
n_detectors * n_scans
processes, right ?Edited by Loic HuderIt depends on how we will design the new
DatasetWriter
API. For examplewriter.add_points(DatasetMsg.sendid+1, data)
.Edited by Wout De NolfBut that is not very nice for the single scan case so probably a
detectorid
is better. Perhaps use a more generic name likedestinationid
.Edited by Wout De NolfA
destinationid
sounds good to me, even in the single scan case. It follows my thinking that the coupling between thesendid
and the key identifying theoutput_handler
should be removed (since we will have multiple processes (scans) for the same output_handler (detectors))Edited by Loic Huder
mentioned in merge request !51 (merged)
Closing since I will follow !50 (comment 251120)