LimaCCDs crash
On id15a, we get crashes and frames missed when running at high speed rated (>800Hz).
Missed frames:
[48]: abscorct()
Dataset set to '0047'
Data path: /data/visitor/ihma408/id15a/20230919/M50_60deg_70wk_xrdct/M50_60deg_70wk_xrdct_0047
Number of unconfirmed ICAT dataset registrations: 2
sheh3 was CLOSED and is now OPEN
X - Rotation Motor : hrrz
start pos = 0.0000 deg
stop pos = 280101.0000 deg
velocity = 60.00 deg/sec
step size = 0.3000 deg
acc size = 10.0100 deg
Y - Translation Motor : hry
--------- ---- ---------
first pos [mm] -12.5125
step size [mm] 0.0325
npoints 771
time sec 4668.35
--------- ---- ---------
Rotation movement during y-steps:
delta size = 3.3 deg
delta time = 0.055 sec
Acquisition :
acq size = 0.1800 deg
acq time = 0.003000 sec
dead time = 0.002000 sec
nb points = 925200 (X:1200 * Y:771)
Total time = 4669.017 sec
Scan 1 Thu Sep 21 12:51:02 2023 /data/visitor/ihma408/id15a/20230919/M50_60deg_70wk_xrdct/M50_60deg_70wk_xrdct_0047/M50_60deg_70wk_xrdct_0047.h5 eh3 user = opid15
aeroystepscan
Preparing: hrrz 3050.7299 hry -12.2525
Homing hrrz ...
Preparing: hrrz -10.0100 hry -12.51255
Opening FastShutter ...
Running: hrrz 20.7421 hry 12.5125 trig 925200 pcoedgehs 920652 (S +0)
-----> Stuck with many frames missed.
Crashes
#5 lima::CtSaving::_getCommonHeader (this=0x565276466610, header=...) at /usr/local/src/conda/lima-core-1.9.22/control/src/CtSaving.cpp:1484
#6 0x00007f2721311606 in lima::CtSaving::_takeHeader (this=0x565276466610, headerIter=..., header=..., keep_in_map=false) at /usr/local/src/conda/lima-core-1.9.22/control/src/CtSaving.cpp:1491
#7 0x00007f2721313450 in lima::CtSaving::_compressionFinished (this=0x565276466610, aData=..., stream=...) at /usr/local/src/conda/lima-core-1.9.22/control/src/CtSaving.cpp:1905
#8 0x00007f272131365a in lima::CtSaving::Stream::compressionFinished (this=0x565276795c10, data=...) at /usr/local/src/conda/lima-core-1.9.22/control/src/CtSaving.cpp:538
#9 0x00007f27213137b7 in lima::CtSaving::Stream::_CompressionCBK::finished (this=0x56527777bdd0, aData=...) at /usr/local/src/conda/lima-core-1.9.22/control/src/CtSaving.cpp:128
#10 0x00007f273f21d94c in PoolThreadMgr::_run (arg=0x565277778390) at /usr/local/src/conda/processlib-1.8.6/core/src/PoolThreadMgr.cpp:244
#11 0x00007f27442e5609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00007f27440a4133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
#5 0x00007f692400fac9 in PoolThreadMgr::Lock::lock (
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /usr/local/src/conda/processlib-1.8.6/core/include/processlib/PoolThreadMgr.h:72
#6 PoolThreadMgr::Lock::Lock (aLockFlag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
aLock=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /usr/local/src/conda/processlib-1.8.6/core/include/processlib/PoolThreadMgr.h:66
#7 SinkTaskBase::setEventCallback (this=0x7f68e040f0b0, aEventCbk=0x7f5d0fc5ac20)
at /usr/local/src/conda/processlib-1.8.6/core/src/SinkTask.cpp:50
#8 0x00007f690616f220 in lima::SoftOpExternalMgr::addTo (this=0x55f774ddb610, aTaskMgr=..., begin_stage=0,
last_link_task=@0x7f6914348344: -1, last_sink_task=@0x7f6914348348: 2)
at /usr/local/src/conda/lima-core-1.9.22/control/software_operation/src/SoftOpExternalMgr.cpp:263
#9 0x00007f6906110a53 in lima::CtControl::newFrameReady (this=0x55f773d48970, fdata=...)
at /usr/local/src/conda/lima-core-1.9.22/control/src/CtControl.cpp:1093
#10 0x00007f690611ff45 in lima::CtBufferFrameCB::newFrameReady (this=0x7f68c4005740, frame_info=...)
at /usr/local/src/conda/lima-core-1.9.22/control/src/CtBuffer.cpp:57
#11 0x00007f69060c1377 in lima::HwFrameCallbackGen::newFrameReady (this=this@entry=0x55f773d48518, frame_info=...)
at /usr/local/src/conda/lima-core-1.9.22/hardware/src/HwFrameCallback.cpp:80
#12 0x00007f69060c4ccc in lima::BufferCtrlMgr::acqFrameReady (this=0x55f773d48518, acq_frame_info=...)
at /usr/local/src/conda/lima-core-1.9.22/hardware/src/HwBufferMgr.cpp:714
#13 0x00007f69060c4de9 in lima::BufferCtrlMgr::AcqFrameCallback::newFrameReady (this=0x55f773d48588, finfo=...)
at /usr/local/src/conda/lima-core-1.9.22/hardware/src/HwBufferMgr.cpp:737
#14 0x00007f69060c1377 in lima::HwFrameCallbackGen::newFrameReady (this=this@entry=0x55f773d48460, frame_info=...)
at /usr/local/src/conda/lima-core-1.9.22/hardware/src/HwFrameCallback.cpp:80
#15 0x00007f69060caaac in lima::StdBufferCbMgr::newFrameReady (this=0x55f773d48460, frame_info=...)
at /usr/local/src/conda/lima-core-1.9.22/hardware/src/HwBufferMgr.cpp:455
#16 0x00007f69042a5c8f in lima::Pco::Camera::_AcqThread::threadFunction_Edge_clhs (this=0x55f774dd3170)
at /users/blissadm/local/lima-camera-pco.git/src/PcoCameraLin.cpp:1352
#17 0x00007f69042a3fef in lima::Pco::Camera::_AcqThread::threadFunction (this=0x55f774dd3170)
at /users/blissadm/local/lima-camera-pco.git/src/PcoCameraLin.cpp:171
#18 0x00007f690607c835 in lima::Thread::staticThreadFunction (data=0x55f774dd3170)
at /usr/local/src/conda/lima-core-1.9.22/common/src/ThreadUtils.cpp:318
#19 0x00007f69290d6609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x00007f6928e95133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
#0 0x00007f6997470d8b in _int_malloc (av=av@entry=0x7f6944000020, bytes=bytes@entry=64) at malloc.c:3608
#1 0x00007f6997473299 in __GI___libc_malloc (bytes=64) at malloc.c:3066
#2 0x00007f6995895a70 in operator new (sz=64) at ../../../../libstdc++-v3/libsupc++/new_op.cc:50
#3 0x00007f6992674d3b in std::__new_allocator<SinkTaskBase**>::allocate (
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
__n=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /opt/conda/conda-bld/processlib_1666351313488/_build_env/x86_64-conda-linux-gnu/include/c++/12.2.0/bits/stl_algobase.h:254
#4 std::allocator_traits<std::allocator<SinkTaskBase**> >::allocate (
__n=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
__a=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /opt/conda/conda-bld/processlib_1666351313488/_build_env/x86_64-conda-linux-gnu/include/c++/12.2.0/bits/alloc_traits.h:464
#5 std::_Deque_base<SinkTaskBase*, std::allocator<SinkTaskBase*> >::_M_allocate_map (
__n=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, this=<optimized out>)
at /opt/conda/conda-bld/processlib_1666351313488/_build_env/x86_64-conda-linux-gnu/include/c++/12.2.0/bits/stl_deque.h:597
#6 std::_Deque_base<SinkTaskBase*, std::allocator<SinkTaskBase*> >::_M_initialize_map (this=this@entry=0x7f69451c9978,
__num_elements=__num_elements@entry=0)
at /opt/conda/conda-bld/processlib_1666351313488/_build_env/x86_64-conda-linux-gnu/include/c++/12.2.0/bits/stl_deque.h:646
#7 0x00007f6992675b21 in std::_Deque_base<SinkTaskBase*, std::allocator<SinkTaskBase*> >::_Deque_base (this=0x7f69451c9978,
this@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /opt/conda/conda-bld/processlib_1666351313488/_build_env/x86_64-conda-linux-gnu/include/c++/12.2.0/bits/stl_deque.h:516
#8 std::deque<SinkTaskBase*, std::allocator<SinkTaskBase*> >::deque (this=0x7f69451c9978,
this@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /opt/conda/conda-bld/processlib_1666351313488/_build_env/x86_64-conda-linux-gnu/include/c++/12.2.0/bits/stl_deque.h:855
#9 TaskMgr::Task::Task (this=0x7f69451c9970,
this@entry=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /usr/local/src/conda/processlib-1.8.6/core/include/processlib/TaskMgr.h:42
#10 TaskMgr::addSinkTask (this=0x7f69455b2b90, aStage=1, aNewTask=<optimized out>)
at /usr/local/src/conda/processlib-1.8.6/core/src/TaskMgr.cpp:171
#11 0x00007f69747d4196 in lima::SoftOpBpm::addTo (this=<optimized out>, aMgr=...,
stage=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /usr/local/src/conda/lima-core-1.9.22/control/software_operation/src/SoftOpId.cpp:119
#12 0x00007f69849ab344 in ?? ()
#13 0x00007f69747d217a in lima::SoftOpExternalMgr::addTo (
this=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, aTaskMgr=...,
begin_stage=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
last_link_task=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>,
last_sink_task=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>)
at /usr/local/src/conda/lima-core-1.9.22/control/software_operation/src/SoftOpExternalMgr.cpp:244
#5 lima::CtSaving::_getCommonHeader (this=0x55de120f9060, header=...) at /users/blissadm/local/lima.git/control/src/CtSaving.cpp:1481
#6 0x00007f7bb4011746 in lima::CtSaving::_takeHeader (this=0x55de120f9060, headerIter=..., header=..., keep_in_map=<optimized out>) at /users/blissadm/local/lima.git/control/src/CtSaving.cpp:1488
#7 0x00007f7bb4013491 in lima::CtSaving::_compressionFinished (this=0x55de120f9060, aData=..., stream=...) at /users/blissadm/local/lima.git/control/src/CtSaving.cpp:1905
#8 0x00007f7bb40136aa in lima::CtSaving::Stream::compressionFinished (this=0x55de12c0b1c0, data=...) at /users/blissadm/local/lima.git/control/src/CtSaving.cpp:538
#9 0x00007f7bb4013807 in lima::CtSaving::Stream::_CompressionCBK::finished (this=0x55de13bf1170, aData=...) at /users/blissadm/local/lima.git/control/src/CtSaving.cpp:128
#10 0x00007f7bd1f1594c in PoolThreadMgr::_run (arg=0x55de13bed730) at /usr/local/src/conda/processlib-1.8.6/core/src/PoolThreadMgr.cpp:244
0x00007fe653d68dd7 in TaskSinkWrap::process (this=0x7fe61105a960) at /usr/local/src/conda/processlib-1.8.6/core/src/TaskMgr.cpp:76
76 /usr/local/src/conda/processlib-1.8.6/core/src/TaskMgr.cpp: No such file or directory.
(gdb) bt
#0 0x00007fe653d68dd7 in TaskSinkWrap::process (this=0x7fe61105a960)
at /usr/local/src/conda/processlib-1.8.6/core/src/TaskMgr.cpp:76
#1 0x00007fe653d6794c in PoolThreadMgr::_run (arg=0x5639c9a296a0)
at /usr/local/src/conda/processlib-1.8.6/core/src/PoolThreadMgr.cpp:244