ucberkeleyseti / commensal-automator Goto Github PK
View Code? Open in Web Editor NEWAutomation for Breakthrough Listen's commensal observing
License: MIT License
commensal-automator's People
Contributors
Watchers
Forkers
danielczechcommensal-automator's Issues
Mismatch between input and output directories
From automator logs:
[2022-10-18 06:22:55,830 - INFO - proc_hpguppi.py:54] Creating output directory...
[2022-10-18 06:23:14,437 - INFO - proc_hpguppi.py:65] Processing:
inputdir: /buf0ro/20221014/0014/Unknown/GUPPI
outputdir: /scratch/data/20221017/0022/hpguppi_beamformer
[2022-10-18 06:23:14,438 - INFO - proc_hpguppi.py:69] Processing file: guppi_59867_42817_000358_J1830-3602_0001.0000.raw
Seems like inputdir and outputdir are supposed to be the same, though.
buf0 is completely empty.
An odd series of events earlier:
[2022-10-18 05:20:57,528 - INFO - proc_hpguppi.py:153] Incomplete agreement of PROCSTAT across hosts. Retrying in 1s.
[2022-10-18 05:20:58,535 - WARNING - proc_hpguppi.py:160] 3 straggler(s).
[2022-10-18 05:20:58,535 - INFO - proc_hpguppi.py:161] Gathered proc status: END
[2022-10-18 05:20:58,535 - INFO - proc_hpguppi.py:120] Upchanneliser/beamformer at END
[2022-10-18 05:20:58,536 - INFO - automator.py:437] New recording processed by hpguppi_proc. Output data are available in /scratch/data/20221017/0022
[2022-10-18 05:20:58,536 - INFO - automator.py:439] Proceeding to processing-complete
[2022-10-18 05:20:58,536 - INFO - automator.py:198] New state: processing-complete
[2022-10-18 05:20:58,537 - INFO - automator.py:471] ['srun', '-w', 'blpn0 blpn1 blpn2 blpn3 blpn4 blpn5 blpn6 blpn7 blpn8 blpn9 blpn10 blpn11 blpn12 blpn13 blpn14 blpn15 blpn16 blpn17 blpn18 blpn19 blpn20 blpn21 blpn22 blpn23 blpn24 blpn25 blpn26 blpn27 blpn28 blpn29 blpn30 blpn31 blpn32 blpn33 blpn34 blpn35 blpn36 blpn37 blpn38 blpn39 blpn40 blpn41 blpn42 blpn43 blpn44 blpn45 blpn46 blpn47 blpn48 blpn49 blpn50 blpn51 blpn52 blpn53 blpn54 blpn55 blpn56 blpn57 blpn58 blpn59 blpn60 blpn61 blpn62 blpn63', 'bash', '-c', '/home/obs/bin/cleanmybuf0.sh --force']
[2022-10-18 05:21:44,657 - INFO - automator.py:503] Resetting nshot after processing: coordinator:trigger_mode coordinator:trigger_mode:array_1:nshot:1
[2022-10-18 06:20:04,255 - INFO - automator.py:198] New state: capture-stop
[2022-10-18 06:20:04,256 - INFO - automator.py:264] Ignoring irrelevant state for array_1
[2022-10-18 06:20:14,356 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 06:20:20,320 - INFO - automator.py:198] New state: capture-done
[2022-10-18 06:20:20,320 - INFO - automator.py:264] Ignoring irrelevant state for array_1
[2022-10-18 06:21:43,671 - INFO - automator.py:198] New state: deconfigure
[2022-10-18 06:21:43,671 - INFO - automator.py:348] array_1 deconfigured. Proceeding to processing.
[2022-10-18 06:21:43,671 - INFO - automator.py:198] New state: processing
I'm not sure if it is supposed to be able to go from state processing-complete -> capture-stop -> not-tracking -> capture-done -> deconfigure -> processing - i would have expected it to gather data sometime between the two processing phases.
Reduce FBH5 output volume
Reduce FBH5 output volume by generating this diagnostic output only once every ten scans.
hpguppi_proc timeout leaves us in "processing" state
I think this will leave the automator wedged so that it can't do anything any more?
[2022-10-18 07:23:37,478 - INFO - proc_hpguppi.py:69] Processing file: guppi_59867_42817_000358_J1830-3602_0001.0000.raw
[2022-10-18 07:27:52,555 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 07:29:38,426 - INFO - automator.py:198] New state: tracking
[2022-10-18 07:29:38,426 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
[2022-10-18 07:31:42,562 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 07:31:42,562 - INFO - automator.py:380] No active recording for endof track for array_1
[2022-10-18 07:32:02,162 - INFO - automator.py:198] New state: tracking
[2022-10-18 07:32:02,162 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
[2022-10-18 07:33:37,548 - ERROR - proc_hpguppi.py:124] Timeout of 600 seconds exceeded
[2022-10-18 07:33:37,548 - ERROR - proc_hpguppi.py:74] Timed out, processing has not started.
[2022-10-18 07:33:37,548 - ERROR - automator.py:432] hpguppi_proc timed out. Stopping automator for debugging.
[2022-10-18 08:02:06,161 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 08:02:06,161 - INFO - automator.py:380] No active recording for endof track for array_1
[2022-10-18 08:02:24,961 - INFO - automator.py:198] New state: tracking
[2022-10-18 08:02:24,961 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
proc_script flag does nothing
I tried to totally remove it but got circus errors. Daniel claims it's super easy for him to fix
Double entry into "processing" state
Right after startup we get:
[2022-10-18 23:19:04,970 - INFO - automator.py:200] New state: not-tracking
[2022-10-18 23:19:04,972 - INFO - subarray.py:38] New subarray object for array_1
[2022-10-18 23:19:04,972 - INFO - automator.py:251] Initialised new subarray object: array_1 in state not-tracking
[2022-10-18 23:19:04,972 - INFO - automator.py:388] No active recording for endof track for array_1
[2022-10-18 23:19:18,795 - INFO - automator.py:200] New state: tracking
[2022-10-18 23:19:18,796 - INFO - automator.py:311] array_1 in tracking state with nshot = 0
[2022-10-18 23:19:18,796 - INFO - automator.py:313] Preparing for processing.
[2022-10-18 23:19:18,796 - INFO - automator.py:319] ['blpn0/0', 'blpn1/0', 'blpn2/0', 'blpn3/0', 'blpn4/0', 'blpn5/0', 'blpn6/0', 'blpn7/0', 'blpn8/0', 'blpn9/0', 'blpn10/0', 'blpn11/0', 'blpn12/0', 'blpn13/0', 'blpn14/0', 'blpn15/0', 'blpn16/0', 'blpn17/0', 'blpn18/0', 'blpn19/0', 'blpn20/0', 'blpn21/0', 'blpn22/0', 'blpn23/0', 'blpn24/0', 'blpn25/0', 'blpn26/0', 'blpn27/0', 'blpn28/0', 'blpn29/0', 'blpn30/0', 'blpn31/0', 'blpn32/0', 'blpn33/0', 'blpn34/0', 'blpn35/0', 'blpn36/0', 'blpn37/0', 'blpn38/0', 'blpn39/0', 'blpn40/0', 'blpn41/0', 'blpn42/0', 'blpn43/0', 'blpn44/0', 'blpn45/0', 'blpn46/0', 'blpn47/0', 'blpn48/0', 'blpn49/0', 'blpn50/0', 'blpn51/0', 'blpn52/0', 'blpn53/0', 'blpn54/0', 'blpn55/0', 'blpn56/0', 'blpn57/0', 'blpn58/0', 'blpn59/0', 'blpn60/0', 'blpn61/0', 'blpn62/0', 'blpn63/0']
[2022-10-18 23:19:18,809 - INFO - automator.py:321] 300.0
[2022-10-18 23:19:18,809 - INFO - automator.py:325] Starting tracking timer for 310.0 seconds
[2022-10-18 23:21:22,570 - INFO - automator.py:200] New state: not-tracking
[2022-10-18 23:21:36,582 - INFO - automator.py:200] New state: tracking
[2022-10-18 23:21:36,582 - INFO - automator.py:311] array_1 in tracking state with nshot = 0
[2022-10-18 23:21:36,583 - INFO - automator.py:313] Preparing for processing.
Subsequently there are more errors but I think this is the first point at which things seem troubled. We shouldn't start processing twice here. And FWIW at the end of all this there are no files on buf0.
Use the /opt/virtualenv/bluse3.9/ environment
3.5 is end-of-lifed, some libraries don't support it any more. The target selector is already using the 3.9 environment so this would also make the overall situation simpler.
Stop for debugging does not seem to actually stop for debugging
[2022-10-18 06:33:15,327 - ERROR - proc_hpguppi.py:124] Timeout of 600 seconds exceeded
[2022-10-18 06:33:15,327 - ERROR - proc_hpguppi.py:74] Timed out, processing has not started.
[2022-10-18 06:33:15,327 - ERROR - automator.py:432] hpguppi_proc timed out. Stopping automator for debugging.
[2022-10-18 06:33:16,417 - INFO - automator.py:113] Starting Automator:
Redis endpoint: 127.0.0.1:6379
Processing script: /home/obs/bin/processing_example.sh
[2022-10-18 06:34:29,844 - INFO - automator.py:198] New state: configure
[2022-10-18 06:34:29,844 - INFO - automator.py:264] Ignoring irrelevant state for array_1
[2022-10-18 06:35:31,356 - INFO - automator.py:198] New state: conf_complete
[2022-10-18 06:35:31,357 - INFO - subarray.py:38] New subarray object for array_1
[2022-10-18 06:35:31,357 - INFO - automator.py:249] Initialised new subarray object: array_1 in state conf_complete
Maybe circus is restarting it? I also wonder whether we should be ignoring the conf_complete state, it doesn't appear like we currently do anything with that at all.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.