Out of memory crashes during prepareAcq
I understood this is a known bug. Here are the steps to get it to reproduce at ID11, I just verified it still crashes since upgrading to bliss 1.8. Here is a plot of the percent memory + cpu versus time from top:
This needs to be run a few times from when the bliss nscope session is restarted to kill the Lima server:
# /users/opid11/bliss_scripts/eigertimeout.py
def eigertimeout1():
# go between 32/16 bit and a couple vs many frames
time.sleep(5)
eiger.camera.auto_summation = 'OFF'
limatake(0.1,2)
t = 0.002 # go to 16 bit to 32 bit
limatake(t, 120*1024*1024*1024//(2162*2068*2),save=False)
time.sleep(5)
eiger.camera.auto_summation = 'ON'
limatake(0.1,2)
print("!"*80)
print("####### control-c the next scan please, when collecting frames")
print("!"*80)
limatake(t, 120*1024*1024*1024//(2162*2068*4),save=False) - on lid11eh3 start nscope session, run the test script and control-c when it is collecting frames in the scan that is tagged:
NSCOPE [1]: user_script_load('eigertimeout.py')
Loading [/users/opid11/bliss_scripts/eigertimeout.py]
Merged [user] namespace in session.
NSCOPE [2]: user.eigertimeout1()
acquisition chain
\u2514\u2500\u2500 eiger
Scan 18 Thu Aug 12 11:11:29 2021 nscope user = opid11
limatake 0.1000 2
Preparing eiger ...
Running ...
eiger acq #2
Finished (took 0:00:05.300922)
acquisition chain
\u2514\u2500\u2500 eiger
Scan 19 Thu Aug 12 11:11:35 2021 nscope user = opid11
limatake 0.0020 14409
Preparing eiger ...
Running ...
eiger acq #14409
Finished (took 0:01:15.016661)
acquisition chain
\u2514\u2500\u2500 eiger
Scan 20 Thu Aug 12 11:12:56 2021 nscope user = opid11
limatake 0.1000 2
Preparing eiger ...
Running ...
eiger acq #2
Finished (took 0:00:02.628463)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
####### control-c the next scan please, when collecting frames
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
acquisition chain
\u2514\u2500\u2500 eiger
Scan 21 Thu Aug 12 11:12:58 2021 nscope user = opid11
limatake 0.0020 7204
Preparing eiger ...
Running ...
^Ceiger acq #4436
Finished (took 0:00:47.220614)
!!! === ScanAbort: === !!! ( for more details type cmd 'last_error' )- exit bliss and repeat step 1, it should crash on the second or third run.
Over on lid11eiger2lima you can watch the memory grow with: top -d 1 -b | grep -e 1430951 -e ^top
tagging @papillon. This bug was previously being chased on the bliss side bliss/bliss!3888 (merged), I am moving it here as it seems to be the lima eiger server that is crashing.