Skip to content

Overall performance is degraded when saving at high frame rates

The PSI/Eiger detector can continuously work at 8 kHz in 4-bit mode, but the Lima performance is degraded when data is saved. Even after applying several optimizations:

  • CPU idle Power Management disabled: sudo bash -c 'for f in /sys/devices/system/cpu/cpu*/cpuidle/state{1..3}/disable; do echo 1 > ${f}; done'

  • CPU frequency scaling performance governor: sudo bash -c 'for f in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > ${f}; done

  • Ping CPU & memory affinity to single NUMA node

the newFrameReady call in control layer takes several tens of microseconds, degrading the performance of the plugin acquisition thread loop. In addition, the overall saving performance is degraded when the frame rate increases.

The following benchmarks are performed with a simulator reading images from a PSI/Eiger-500k detector on a Dectris Eiger2 Lima computer:

# opid17@lid17eiger2lima
$ (cd ~blissadm/local/Git/lima/camera/simulator && 
     numactl --cpunodebind=1 --membind=1 \
       build/test/test_simulator --debug-type-flags=$(printf "%d" 0x7) \
                                 --cam-mode=LOADER_PREFETCH \
                                 --cam-file-pattern=/nobackup/deg_du_data/test_eiger/psi_eiger_500k_00000.edf --cam-nb-prefetched-frames=500 \
                                 --buffer-max-memory=20 \
                                 --acq-expo-time=0.5e-6 --acq-latency-time=0 --acq-nb-frames=100000 \
                                 --saving-directory=/data/id13/inhouse/test/psi_eiger_500k/test_eiger/simulator \
                                 --saving-prefix=test_simulator_ --saving-frames-per-file=10000 --saving-format=HDF5BS \
                                 --proc-nb-threads=10 --video-active=1 \
                                 --test-rm-files-before-acq=1 --test-image-status-callback=1)
[2024/01/04 09:27:06.262477] 7f84fb9d6c80             *Test*Simulator::TestApp::getCtControl (A Exception::102)-Always: Camera: m_pars->cam_mode=LOADER_PREFETCH
[2024/01/04 09:27:06.262569] 7f84fb9d6c80             *Test*Simulator::TestApp::getCtControl (A Exception::119)-Always: m_pars->cam_file_pattern=/nobackup/deg_du_data/test_eiger/psi_eiger_500k_00000.edf
[2024/01/04 09:27:06.262651] 7f84fb9d6c80             *Test*Simulator::TestApp::getCtControl (A Exception::121)-Always: m_pars->cam_nb_prefetched_frames=500
[2024/01/04 09:27:06.264035] 7f84fb9d6c80         *Test*Control::CtTestApp::init (control/src/CtTestApp.cpp:267)-Always: m_pars->acq_mode=Single, m_pars->acq_expo_time=5e-07, m_pars->acq_lat_time=1e-08
[2024/01/04 09:27:06.264315] 7f84fb9d6c80         *Test*Control::CtTestApp::init (control/src/CtTestApp.cpp:290)-Always: nb_exec_threads=0
[2024/01/04 09:27:06.264344] 7f84fb9d6c80         *Test*Simulator::TestApp::configureAcq (A Exception::140)-Always: effective_dim=<1030x514x1-Bpp8>
[2024/01/04 09:27:06.264444] 7f84fb9d6c80             *Test*Control::CtTestApp::execCmd (control/src/CtTestApp.cpp:327)-Always: executing cmd=rm -f /data/id13/inhouse/test/psi_eiger_500k/test_eiger/simulator/test_simulator_*.h5
[2024/01/04 09:27:06.270128] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:349)-Always: acq_idx=0, nb_frames_idx=0, acq_nb_frames=100000
[2024/01/04 09:27:27.677832] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:363)-Always: acq prepared
[2024/01/04 09:27:27.677872] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:367)-Always: max_nb_buffers=76073
[2024/01/04 09:27:27.677894] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:370)-Always: acq started
[2024/01/04 09:27:27.777974] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:838, saved: 799
[2024/01/04 09:27:27.878071] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:1699, saved: 1665
[2024/01/04 09:27:27.978158] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:2558, saved: 2528
[2024/01/04 09:27:28.078246] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:3389, saved: 3364
[2024/01/04 09:27:28.178336] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:4246, saved: 4225
[2024/01/04 09:27:28.278419] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:5102, saved: 5084
[2024/01/04 09:27:28.378513] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:5928, saved: 5913
[2024/01/04 09:27:28.478598] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:6782, saved: 6768
[2024/01/04 09:27:28.578680] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:7503, saved: 7491
[2024/01/04 09:27:28.678771] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:8329, saved: 8314
[2024/01/04 09:27:28.778866] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:9179, saved: 9165
...
[2024/01/04 09:27:43.392844] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:98834, saved: 98815
[2024/01/04 09:27:43.492952] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqRunning, acquired:99685, saved: 99669
[2024/01/04 09:27:43.593057] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:397)-Always: AcqReady, acquired:99999, saved: 99999
[2024/01/04 09:27:43.593095] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:406)-Always: all frames acquired: elapsed=15.9152, frame_rate=6283.3
[2024/01/04 09:27:43.593117] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:426)-Always: saving finished
[2024/01/04 09:27:43.593124] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:434)-Always: acq finished
[2024/01/04 09:27:43.593130] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:451)-Always: elapsed=15.9152
[2024/01/04 09:27:43.849468] 7f84fb9d6c80         *Test*Control::CtTestApp::runAcq (control/src/CtTestApp.cpp:464)-Always: Saving statistics (MByte/s): statistics_size=100000, incoming_speed=3512.57, compression_speed=3442.35, compression_ratio=5.44358, saving_speed=1515.68

The following limatake scan shows the same kind of degradation with a PSI/Eiger-500k detector:

TEST_EIGER [123]: limatake(10e-6, 100000, psi_eiger_500k, latency_time=(1 / 2e3 * int(psi_eiger_500k.camera.pixel_depth) / 16 - 10e-6), save=True)
Scan 178 Fri Jan 05 13:34:02 2024 /data/id13/inhouse/test/psi_eiger_500k/test_eiger/data.h5 test_eiger user = opid13
limatake 0.0000 100000
psi_eiger_500k acq #100000 save #100000  
Finished (took 0:01:06.030725)

       Out [123]: Scan(number=178, name=limatake, path=/data/id13/inhouse/test/psi_eiger_500k/test_eiger/data.h5)

TEST_EIGER [124]: limastat(psi_eiger_500k)

        camera    history        incoming    compression     compression          write
                     size           speed          ratio           speed          speed
--------------  ---------  --------------  -------------  --------------  -------------
psi_eiger_500k     100000  1432.74 MB/sec          5.442  1193.75 MB/sec  816.79 MB/sec

The incoming frame rate should be 8 kHz (4 GB/sec); it is degraded to ~3 kHz.

Edited by Alejandro Homs Puron