pcdshub / engineering_tools

A repository of scripts, configuration useful for the PCDS team

engineering_tools

A repository of scripts, configuration useful for the PCDS team

Push updates

git push -u origin master

Add Tag

git tag -a R{tag} -m '{comment}'

git push -u origin R{tag}

Creating a new release

# Clone the source code into a new folder
git clone https://github.com/pcdshub/engineering_tools.git R{tag}
# Enter repository
cd R{Tag}
# checkout tag number
git checkout tags/R{tag}

Updating latest

# Go to latest checkout
cd engineering_tools
# Pull latest from master branch
git pull origin master

The scripts

ami_offline_psana	usage: ami_offline_psana options We will run ami_offline OPTIONS: -u user (needs to be able to log into the psananeh/feh, if not on psana already) -e EXPNUMBER -R rebinning (binned to 640x640) -n no timetool plugin
archive-status	usage: archive-status [-h] PV Return the status of the specified PV in the archiver. OPTIONS: -h, --help: Show the help message and exit.
camViewer	usage: /reg/g/pcds/engineering_tools/latest/scripts/camViewer options start the viewer for controls cameras OPTIONS: -c|--cam camera name as in camera list or gige # -m|--main bring up the edm screen -r|--reboot reboot the IOC -l|--list print list of cameras -w|--wait # (wait for # hours to ask to renew, default 2 and 12 for GIGEs) -u|--update # update rate limit (default 5) -H|--hutch: use a specific hutches camviewer config file -e|--enable disable camera ioc -d|--disable disable camera ioc -a|--acquire start acquiring images -s|--stop stop acquiring images -n|--cycle cycles acquisition (first stops then starts)
check_host	Usage: /reg/g/pcds/engineering_tools/latest/scripts/check_host HOSTNAME Display host info and run some checks.
configdb_readxtc	usage: configdb_readxtc options We will run configdb_readxtc OPTIONS: -u user (needs to be able to log into the psananeh/feh) -e expnumber
daq_control	daq_control COMMAND TARGET COMMAND : { start, stop, restart, status } TARGET : { daq, ami } COMMAND : ami TARGET : { [0], 1 }
daq_waitwin	Waits for the LCLS-I daq windows to load, then exits.
detector_totals.py	Generates a report for the detector group. Reports contains the number of events per detector type gathered from all experiments in a run period. For example, detector_totals.py --run_period 20 generates the detector total report for run 20
dev_conda	Source this to activate a pcds conda environment. By default, this activates the latest environment. Use export PCDS_CONDA_VER=VERSION before running to pick a different env. Pick up EPICS environment variable settings just in case user did not
eloggrabber	usage: eloggrabber options start the eloggrabber, by default look at current exp OPTIONS: -e pass in an experiment to look at -x instrument logbook -c controls logbook -u username
epicsArchChecker	usage: epicsArchChecker [-h] [-w] [-s] filepath Checks epicsArch files for mismatches of PVs and aliases, missing files, and unconnected PVs. positional arguments: filepath Full filepath of the file to check e.g /reg/g/pcds/dist/pds/xpp/misc/epicsArch.txt optional arguments: -h, --help show this help message and exit -w, --warnings Displays: -Pvs and Aliases duplicated. -Pvs with no alias and aliases no Pvs. -s, --status Displays Pvs not connected.
get_curr_exp	usage: get_curr_exp options OPTIONS: -l add live status -i/H information for hutch (override autodetection)
get_hutch_name	Returns the hutch name based on the host it is run on. See `get_info` for more information.
get_info	usage: get_info [-h] [--run] [--exp] [--live] [--ended] [--hutch HUTCH] [--station STATION] [--getHutch] [--gethutch] [--getstation] [--getbase] [--getinstrument] [--getcnf] [--files_for_run FILES_FOR_RUN] [--nfiles_for_run NFILES_FOR_RUN] [--setExp SETEXP] optional arguments: -h, --help show this help message and exit --run get last run --exp get experiment name --live ongoing? --ended ended --hutch HUTCH get experiment for hutch xxx --station STATION optional station for hutch with two daqs, e.g. cxi and mfx --getHutch get hutch (uppercase) --gethutch get hutch (lowercase) --getstation get hutch station (for multiple daqs) --getbase get base daq name (hutch_station if multiple daqs, otherwise hutch) --getinstrument get instrument (HUTCH_station if multiple daqs, otherwise hutch) --getcnf get cnf file name --files_for_run FILES_FOR_RUN get xtc files for run --nfiles_for_run NFILES_FOR_RUN get xtc files for run --setExp SETEXP set experiment name
get_lastRun	usage: get_lastRun options OPTIONS: -l add live status -i/H information for hutch (override autodetection)
grep_ioc	usage: grep_ioc KEYWORD [hutch] hutch can be any of: xpp, xcs, cxi, mfx, mec, xrt, aux, det, fee, hpl, icl, las, lfe, tst, thz, all If no hutch is specified, all hutches will be searched
grep_pv	GREP SEARCHES THROUGH ALL IOCs IN /reg/d/iocData/ FOR PVs THAT MATCH GIVEN KEYWORD/HANDLE. Ideally, find_pv should be used as it gives a lot more information (but can be slow sometimes)
hdf5_to_gif.py	Converts images in hdf5 files created with h5_img_collect to gifs. Specify the path with -p and and how long each frame should last (ms) with -t. Will save to cwd or a specified directory with -d as {original_filename}.gif
image_saver	Uses h5_img_collect to save images from a camera in an hdf5 format. Command line arguments -c, -n, -p, and -f to specify camera name, number of images, path to save hdf5 file to, and name to save hdf5 file as. Also can use -g switch to open a GUI with a button that when pressed takes images with specified parameters - can be pressed multiple times. The number of images and the label on each button can be changed within the gui. Images from the gui will be converted into gifs and saved into a (new) ~/gifs directory.
iocmanager	iocmanager [hutch] Control status of all IOCs running in a particular hutch in an interactive GUI. Current hutch is used if not provided.
ioctool	usage: ioctool <ioc>|<pv> [option] Script that returns information about an ioc given its name or a PV it hosts default option is 'name', list of options: status : print power status of machine, try to ping interfaces name : returns the name of the ioc dir : returns the path to the directory the ioc is running from cddir :opens the directory the ioc is running from (start with "source" before calling script with this option) cfg : returns the the file name of the ioc .cfg (or st.cmd) less: opens the ioc .cfg (or st.cmd) in less cat : opens the ioc .cfg (or st.cmd) in cat data : returns the path of the appropriate iocData directory if it exists autosave : opens the most recent autosave file in less archive : opens the most recent archive file in less log : opens the most recent log file in less telnet : starts a telnet session with the ioc
ipmConfigEpics	usage: ipmConfigEpics [-b boxname] [-H hutch] [-d] [-h] [-l] -b: specify boxname to view -H: specify a hutch to use, overriding the automated selection -d: fix issues with Bld Damage (likely camera IOC w/plugins on same machine) -h: display this help text -l: list available boxnames
kinit_helper	usage: kinit_helper Defines two functions - kauth and afsauth. kauth will create a new kerberos token, renew an existing one, or do nothing if a valid token exists. afsauth will check that the user and host are able to access afs; if so, and an afs token doesn't already exist, kauth will be called and a new afs token will be made. By default, calls afsauth.
makepeds	usage: makepeds options Make a pedestal file for offline use OPTIONS: -u user (needs to be able to log into the psananeh/feh) -r runnumber for pedestal -e EXPNAME in case you do not want pedestals for the ongoing experiment -J make pedestals for Jungfrau (default only cspad/EPIX detectors) -j make pedestals for Jungfrau - 3 run version(default only cspad/EPIX detectors) -O make pedestals for Opals (default only cspad/EPIX detectors) -Z make pedestals for Zyla (default only cspad/EPIX detectors) -p TEXT: add to elog post -c EVTCODE X use events with eventcode X set -n # : if you have created a noise file, then write pixel mask file for pixels with noise above #sigma -N # : use this number of events (default 1000) -D : dark run for XTCAV -L : lasing off run for XTCAV -v STR: validity range (if not run#-end, e.g. 123-567 or 123-end) -l: do NOT send to batch queue -F : use the FFB (default if no experiment is passed) -g : run on an FFB batch node -f : full epix10k charge injection run -C # : if noise filecreated, write pixel mask file for pixels with noise below xxx (currently integer only...) -m # : write pixel mask file for pixels with pedestal below xxx (currently integer only...) -x # : write pixel mask file for pixels with pedestal above xxx (currently integer only...) -i start calibman. -r 0: show all darks, -r n: show runs (n-25) - 25 -d give path for alternative calibdir
makepeds_psana	usage: makepeds_psana options Make a pedestal file OPTIONS: -r runnumber for pedestal -e EXPNAME in case you do not want pedestals for the ongoing experiment -H HUTCH in case you do not pass an experiment name -Z pedestal for zyla -O make pedestals for Opals -J pedestal for jungfrau (needs first of set of 3 runs!) -D : dark run for XTCAV -L : lasing off run for XTCAV (-b specifies the number of assumed bunches, def 1) -l : donot send to batch queue -F : use FFB -f : full epix10k charge injection run -v STR: validity range (if not run#-end, e.g. 123-567 or 123-end) -N # : use this number of events (default 1000) -n # : if noise filecreated, write pixel mask file for pixels with noise above xxx (currently integer only...) -C # : if noise filecreated, write pixel mask file for pixels with noise below xxx (currently integer only...) -m # : write pixel mask file for pixels with pedestal below xxx (currently integer only...) -x # : write pixel mask file for pixels with pedestal above xxx (currently integer only...) -c EVTCODE X use events with eventcode X set -i start calibman. -r 0: show all darks, -r n: show runs (n-25) - 25 -d give path for alternative calibdir -t : test, do not deploy. -y : when specify cuts for status mask, apply those for epix100.
motor-expert-screen	usage: motor-expert-screen options MOTOR_PV_BASENAME Start an EDM for the specified motor. Attempts to choose the correct type. OPTIONS: -h shows the usage information
motor-typhos	usage: motor-typhos options MOTOR_PV_BASENAME Start a typhos screen for the specified motor. Attempts to choose the correct type. OPTIONS: -h shows the usage information
motorInfo	usage: motorInfo MOTOR_PV (motor_pv_2/autosave/archive/pmgr_diff/pmgr_save) OPT If given two motors, compare their settings If given autosave as second argument, compare the current settings to the autosaved values: differences will be printed If given archive, the archive values will be printed for the last week. If only the base PV is given, extra arguments will be needed OPTIONS: -h shows the usage information -f fields to use as a comma separated list (default: use all autosave values) -s start time for archiver info (YYYY/MM/DD HH:MM:SS) -e end time for archiver info (YYYY/MM/DD HH:MM:SS)
pcds_conda	Source this to activate a pcds conda environment. By default, this activates the latest environment. Use export PCDS_CONDA_VER=VERSION before running to pick a different env. Pick up EPICS environment variable settings just in case user did not
pkg_release	Checks out a package from the pcdshub github at a particular tag. Does not update "latest" softlinks, these are inconsistent between packages. Make sure your tag exists before running.
pmgr	pmgr [hutch] [--debug] [--applyenable] --debug : Displays the debug button, which prints out any edits made --applyenable : Displays the apply all button, which applies settings to all motors
pydev_env	Source this file to activate a development environment based on the latest shared environment and on past calls to pydev_register
pydev_register	Use this script to register development packages so that they will be available when you source pydev_env
pyps-deploy	usage: pyps-deploy [-h] -r RELEASE -c CONDA [--repo REPO] [--app-bin APP_BIN] app Sets up a pyps/apps deployment for a particular github python package. This will create an executable under .../pyps/apps/APP-NAME/RELEASE/APP-NAME and repoint the symbolic link at .../pyps/apps/APP-NAME/latest to the new release folder. positional arguments: app Name of the app to deploy optional arguments: -h, --help show this help message and exit -r RELEASE, --release RELEASE App version -c CONDA, --conda CONDA Conda environment name --repo REPO Clone this repo and mask the environment package. Use this when you have only a small change that does not need a full environment release. --app-bin APP_BIN Use in conjunction with --repo arg when the launcher is not in the bin directory
questionnaire_tools	usage: questionnaire_tools [-h] [-f FROMEXP] [-t TOEXP] [-r READEXP] [-c] [-d ADD_DEVICE] [-l] [-p PRINT_DEVICE] [--dev] [--experimentList] [--propList] optional arguments: -h, --help show this help message and exit -f FROMEXP, --fromExp FROMEXP experiment to copy from -t TOEXP, --toExp TOEXP experiment to copy to -r READEXP, --readExp READEXP experiment to read CDS tag from -c, --copy_CDS copy data from CDS tab -d ADD_DEVICE, --add_device ADD_DEVICE name of device to be added -l, --list_devices list device to be added -p PRINT_DEVICE, --print_device PRINT_DEVICE print data for device --dev connect to dev database --experimentList list of experiments --propList list of proposals
restartdaq	usage: restartdaq options OPTIONS: -w sort windows after start -p select partition (same as used last) -s silent (do not email jana)
serverStat	usage: serverStat servername options Script to check status of servers & reboot/power cycle them SIGNATURE: serverStat SERVERNAME [command] default command is 'status', list of commands: status : print power status of machine, try to ping interfaces on : power machine on off : power machine off cycle : power cycle machine, wait a few second in off state reset : reset machine (ideally try that before power cycling) console: open the ipmi console where possible expert : display info and run checks on server
startami	usage: startami options we are starting another ami session here OPTIONS: -s: stop the ami client current running on this machine -c: config file you'd like to use (i.e. cxi_test.cnf)
stopami	Kill an AMI process running in the current hutch.
stopdaq	Stop the daq in the current hutch.
takepeds	usage: takepeds Takes a run with dark images for use in pedestals, and posts to the elog.
verify-hutch	usage: verify-hutch hutch Verifies that the passed argument is a known hutch, exit 0 for success and exit 1 for failure.
wheredaq	Discover what host is running the daq in the current hutch, if any.
wherepsana	Usage: where_psana [-h] [-c CONFIG] [-d DETAIL] Checks where we have shared memory servers for psana running and could run psana jobs. Optional arguments: -h Show usage -c Pick a specific DAQ config file rather than automatically selecting current hutch's file -d Also show information about dss node mapping
set_gem_timing	Usage: set_gem_timing [SC or NC]

engineering_tools's Issues

Figure out good way to position screens well

This gave me a laugh. Maybe eventually we should have some uniform way of doing this across the various scripts and hutches.

Originally posted by @ZLLentz in #101 (comment)

No CI in repo

Expected Behavior

It'd be nice to e.g. run flake8 and shellcheck here for some quick automated checking

Current Behavior

No CI, no motivation to manually point out flake8 violations

Possible Solution

Partial adoption of pcds-ci-helpers, maybe just the pre-commit job + add a pre-commit to this repo

Context

I find myself saying flake8 and shellcheck things in most PR reviews here

pedestal/calibration scripts on epix10k style detectors (makepeds)

We might have epix10k detectors in the partition that do NOT have the necessary calibration cycles. We should only run the pedestals code on detectors that change.

We should also have a loop over all epix10k detectors with calibcycles (look at the solution for the Jungfrau detectors).

A script to check the configuration written by Chris is below:
python ~cpo/ipsana/epix_gains.py exp=meclv0818:run=204 Epix10kaQuad0

Expected Behavior

Don't run on data when it's not worth it. Run on all detectors that need calibrating.

Current Behavior

We will only run over the first detector (I think?)

Possible Solution

See above: use Chris' script before running code. Add printout to make clear that work is going on. Use jungfrau example for loop.

camViewer script with option to enable and disable camera IOCs.

Add an option to toggle if a given IOC is enabled or not.
The use case if that it is hard to keep track wether machines are overloaded or not when we have 20-30 defined cameras of which maybe 8 are running. This procedure needs to be simple for the Beamline scientist. We also need to make sure that the operator needs to be permitted to do these functions.

Expected Behavior

create:
camViewer -c --enable
camViewer -c --disable

Current Behavior

Don't have this.

Possible Solution

Use iocmanager command line tools.

ENH: wheredaq speed improvement

Expected Behavior

wheredaq should be quick to tell you if the DAQ runs or not

Current Behavior

Currently it can be pretty slow, in particular when the DAQ is not running.

Possible Solution

Chris Ford suggested to only ask for the starts of the control_gui task, rather than the whole cnf file.

Your Environment

terminal as opr.

check if statsPlugin is off in ipmConfigEpics

Expected Behavior

ipmConfigEpics should check if it successfully turned off the statsPlugin and report the results to the user.

Current Behavior

The helpBld function in ipmConfigEpics currently caputs two values to turn off the statsPlugin for two different ipm boxes but there is no error handling in place.

Possible Solution

trap?

Context

If the script is run using the helpBld function now, it gives channel connect errors for both PVs.
xcs-control:scripts pennebak$ ./ipmConfigEpics -d -b ipm3 Channel connect timed out: 'XCS:DG3:CVV:02:Stat2:EnableCallbacks' not found. Old : New : Channel connect timed out: 'HFX:DG3:CVV:01:Stat2:EnableCallbacks' not found. Old : New : Turned off statsPlugin for XCS ipm3 and dio3

Your Environment

The above was attained by running the script on xcs-control with changes as of #24.

Check PVs in ipmConfigEpics

Expected Behavior

Check that the ipimb/wave8 PVs are there before trying to open the GUI and print a message to the screen if they are not. ipmConfigEpics could also print what it is doing before the GUI starts so that one knows when we are just waiting for the GUI to appear and when we have hit a snag in the script.

Current Behavior

The script does not currently check PV's and runs into errors when trying to open the GUI for nonexistent PV's. There is also very little reporting to the user so is often hard to tell between snags and waits.

Possible Solution

"I'd start by the :SUM PV which exists for both ipm & wave8 and use 'caget -t | grep ....' to determine if everything is up as expected." -Silke

makepeds: make pedestal (and other files) for Rayonix

Expected Behavior

have a standard procedure to create pedestals for the Rayonix

Current Behavior

Have to copy pedestal/mask files or run very old analysis release to get old GIU

Possible Solution

add this as option too makepeds like the pedestals for the OPAL.
Also need geometry option (though we need a geometry first to copy)

Context

Chuck's psocake would like all detectors to behave the same and thus access the data via det.calib(). Unfortunately ones needs a pedestal created the LCLS-tools for this which is not exactly how this detector should be corrected.

Remove "AMO" & "SXR" options from `grep_ioc` script. Add "ICL" & "TST"

[LCLSECSD-346] pcds-envs (pcds_conda) can interfere with engineering_tools settings (libca / caget / etc)

Expected Behavior

• engineering tools scripts should not be dependent on the shell environment
  • i.e., the scripts should be standalone and setup environment settings as needed

Current Behavior

• Some scripts do not work when pcds_conda is activated

For example, using env -i ... to ignore my bashrc/bash_profile entirely, the following works without issue:

$ env -i bash --noprofile --norc
bash-4.2$ export PATH=/reg/g/pcds/engineering_tools/latest/scripts:$PATH
bash-4.2$ motor-expert-screen MY:MOTOR
bash-4.2$ exit

$ env -i bash --noprofile --norc
bash-4.2$ source /cds/group/pcds/pyps/conda/pcds_conda
(pcds-5.1.1) bash-4.2$ export PATH=/reg/g/pcds/engineering_tools/latest/scripts:$PATH
(pcds-5.1.1) bash-4.2$ motor-expert-screen MY:MOTOR
caget: error while loading shared libraries: libreadline.so.5: cannot open shared object file: No such file or directory

Possible Solution

Steps to Reproduce (for bugs)

See above.

Context

Failing to use engineering tools scripts with pcds_envs activated

Your Environment

Can be a clean environment - see examples above.

Helper Scripts for Maintaining Github Forks

Expected Behavior

We should have helper scripts for the Github workflow:

• Cloning a fork and setting upstream appropriately
• Synchronizing a fork with the upstream

Current Behavior

No such scripts currently exist

Possible Solution

Port over @ZLLentz's tools and clean them up

git_setup_fork ()
{
    usage="git_setup_fork <repo>";
    if [ -z "${1}" ]; then
        echo "${usage}";
        return;
    fi;
    REPO=`basename "${1}"`;
    UPSTREAM=`dirname "${1}"`;
    GITHUB_USERNAME="ZLLentz";
    git clone "[email protected]:${GITHUB_USERNAME}/${REPO}.git";
    pushd "${REPO}";
    git remote add upstream "[email protected]:${UPSTREAM}/${REPO}.git";
    popd
}

git_sync_upstream ()
{
    git checkout master;
    git fetch upstream;
    git pull upstream master;
    git push origin master
}

Context

In the engineering general, there is some consensus that maintaining a fork is a pain point.

Create verify_hutch or similar function/script

We should have a script that verifies valid hutch names. Probably lowercase default and uppercase by option? This can then be used in other engineering_tools scripts instead of the assortment of checks we currently have built in. Using this would also be an improvement over the "unknown_hutch" checks in imgr and other scripts.

Originally posted by @ZLLentz in #101 (comment)

Add calibcycle option to makepeds_psana

Expected Behavior

for the epix10ka detectors, we can take runs with either the full charge injection (takes hours) or a run with only the first few calibcycles to redo the pedestals. The first 3 cycles are the "plain" gains, the next 2x2 are the pedestals for the auto ranging options, then follow the actual charge injection cycles.

Current Behavior

Currently, we hardcode to only use the first 3 calibcycles as this corresponds to how we used the detector in XCS in December 2018.

Possible Solution

add one more option to makepeds_psana (number of calibcycles in epix10ka calibration data to use which should default to 3).

Add a kinit helper to check when kinit needs to be run

Expected Behavior

There should be a script you can run either standalone or inside other scripts to check whether your kerberos tickets are set up properly. This may also run kinit/aklog for you.

Current Behavior

No such script exists

Possible Solution

• Parse the output of klist
• Run klist -s and check for nonzero exit status

Context

#146 (comment)

kerberos tickets are how we:

• manage write access to afs
• manage password-free logins between servers

ENH: Future improvements for BLmotors

1. Remove unnecessary argument handling Originally posted by @ZLLentz in #91 (comment)
2. Remove exec call Originally posted by @ZLLentz in #91 (comment)
3. Remove unnecessary tuple creation for res Originally posted by @ZLLentz in #91 (comment)
4. Clean up file handling Originally posted by @ZLLentz in #91 (comment)
5. Simplify device sorting logic Originally posted by @ZLLentz in #91 (comment)
6. Fix other things left over from Mike's code...
7. Parallelize EPICS calls Originally posted by @ZLLentz in #91 (comment)
8. Maybe rename?

Rewrite isCam from serverStat!

Expected Behavior

serverStat is supposed to be used to cycle servers as a wrapper on psipmi.
It is NOT supposed to fix cameras or tell people to use the iocmanager to reboot servers, if anything, the opposite!

Current Behavior

XPP report this to happen for serverStat yag2 (DAQ camera!):
when I try to serverStat is it says “go to the ioc manager to reboot/check status” in the ioc manager I find SB3 - YAG 2 and “noconnect”

Possible Solution

remove the whole isCam section and at the very least, don't have it find things that also actively exist in the DAQ cnf file

get_lastRun does not work w/o a run anymore.

Expected Behavior

cxi-daq:~> get_lastRun
returns:
no runs yet

Current Behavior

cxi-daq:~> get_lastRun
ERROR:main:Invalid response from server
ERROR:main:No runs?
Traceback (most recent call last):
File "/reg/g/pcds/engineering_tools/cxi/scripts/get_info", line 172, in
print(int(rundoc['num']))
TypeError: 'NoneType' object is not subscriptable
no runs yet

Possible Solution

Figure out what broke.

Steps to Reproduce (for bugs)

call as above. Seems to be broken in many releases so that 'm not sure why we did not see the earlier.

pydev_register path verification

pydev_register should verify that the provided path is valid

Originally posted by @ZryletTC in #60 (comment)

calling `serverStat` args issues

Issue noted on hutch and dev subnets. Calling serverStat without args will or an invalid arg will hang indefinitely. Calling serverStat --help will yields the output, this is a non-gige camera, go to the IOC manager to reboot/check status.

No arg to get usage to print?

Add imgr

Add a wrapper or link to imgr.py in this repo. It is currently only accessible at /reg/g/pcds/pyps/apps/iocmanager/

`ipmitool` used directly in `daq_control` and in `check_host`

Not to throw a last-minute wrench here, but I noticed that we also use ipmitool directly in daq_control and in check_host. I do not know how much we use these scripts since I'm detached from beamline ops but I thought it should be noted here.

Originally posted by @ZLLentz in #57 (comment)

Error handling and output in some scripts could use tweaking: get_lastRun, get_curr_exp, ...

Expected Behavior

• If a hutch can't be determined or an experiment can't be found (for example), return a fixed value without error messages to the user
• Wrapping these scripts with Python code is common and really should never fail, and should be usable on machines more typically used for testing

Current Behavior

As we were trying to make filenames for gige cameras to save to by way of hutch + experiment name + run number, we ran into the following:

[klauer@lfe-console  ~]$ /reg/g/pcds/engineering_tools/lfe/scripts/get_lastRun
ERROR:__main__:No runs?
Traceback (most recent call last):
  File "/reg/g/pcds/engineering_tools/R2.0.1/scripts/get_info", line 164, in <module>
    exp = resp.json().get("value", {}).get("name")
  File "/reg/g/pcds/pyps/conda/py36/envs/pcds-3.4.0/lib/python3.6/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/reg/g/pcds/pyps/conda/py36/envs/pcds-3.4.0/lib/python3.6/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/reg/g/pcds/pyps/conda/py36/envs/pcds-3.4.0/lib/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/reg/g/pcds/pyps/conda/py36/envs/pcds-3.4.0/lib/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
No runs taken yet

[klauer@lfe-console  ~]$ /reg/g/pcds/engineering_tools/lfe/scripts/get_hutch_name
lfe

[klauer@lfe-console  ~]$ /reg/g/pcds/engineering_tools/lfe/scripts/get_curr_exp
Traceback (most recent call last):
  File "/reg/g/pcds/engineering_tools/R2.0.1/scripts/get_info", line 158, in <module>
    exp = resp.json().get("value", {}).get("name")
  File "/reg/g/pcds/pyps/conda/py36/envs/pcds-3.4.0/lib/python3.6/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/reg/g/pcds/pyps/conda/py36/envs/pcds-3.4.0/lib/python3.6/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/reg/g/pcds/pyps/conda/py36/envs/pcds-3.4.0/lib/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/reg/g/pcds/pyps/conda/py36/envs/pcds-3.4.0/lib/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
/reg/g/pcds/engineering_tools/lfe/scripts/get_curr_exp: line 43: [: ==: unary operator expected

get_hutch for machines in the XRT might need special treatment

get_hutch for machine in the XRT returns "fee" as it uses the subnet.
This is fine, but when you try to start the iocmanager without argument on a xrt machine, you will get the FEE iocmanager.

Expected Behavior

I would like the XRT iocmanager

Current Behavior

I get the FEE iocmanager

Possible Solution

Treat as special case? Accept that this is how it works as we typically open the XRT iocmanager from a hutch machine with an extra argument anyways?

takepeds should stop if pedestal failed

If pedestal failed for whatever reason, the script should stop. Currently, it looks like it still proceeds to post in the elog and tell users to call makepeds.

Should stop after this line:

engineering_tools/scripts/takepeds

Line 6 in a608fb6

$DAQ_RELEASE/tools/scanning/take_pedestals -p 0 -r

set -e might work (hoping that take_pedestals gives non-zero status on error)?

ENH: Future improvements for camViewer

1. Reference imgr.py from within engineering_tools
2. Add full worded options (--enable/--disable) in addition to the the single letter option (-e/-d)
3. Troubleshoot problem with accessing cameras outside of hutch (lfe/fee)

MMC100 support in motor-expert-screen

Expected Behavior

motor-expert-screen shows the appropriate screen for all motor types.

Current Behavior

Currently motor expert screen doesn't support MMC 100's. It instead shows the screen for IMS motors.

Possible Solution

If RTYP field is mmca in the PV, use mmc_main.edl from /reg/g/pcds/epics/ioc/tmo/mmc/R1.0.0/mmcScreens

Context

Working with MMC100s in CXI and TMO

Your Environment

engineering_tools R3.1.0

get_info: Fix hutch subnets, remove AMO & SXR, add TMO, H2.2 (if we know the subnet yet)

Expected Behavior

I expect that the subnets are correct. At this time, also only deal with subnets we still care about.
Also fix serverStat cds to fez mapping.

Current Behavior

Subnets are from pre-network change and were not even correct then.

ioctool - Option to view <ioc>.pvlist

Context

An operator wanted to be able to look at the PVs that are hosted by an IOC. This seems like a potentially useful feature.

ENH: Future improvements for ioctool

Expected Behavior

Next passes could include:

1. Perform a better search for st.cmd and cfg files (including being able to find top level st.cmd files and those in 'children' directories)
2. Add telnet error handling that checks to see if the ioc is on a subnet that is able to connect.
3. Bring up current autosave file and/or other things in iocData
4. Error handling for when multiple IOCs are returned
5. Option to cat similar to less
6. Fix the ioctool section in the README.md file, currently has an extra status line
7. Include a way to enter a camera name to find ioc

Current Behavior

Current script does not include those possible additions

Possible Solution

Changes can be added right to the script by using existing options and functions

Steps to Reproduce (for bugs)

Context

Your Environment

makepeds: use special/local calib-dir

The detector group requests that ability to write the calibration files to a requested directory rather than the current experiment directory.

Expected Behavior

We should have an option to makepeds that take a directory to use instead of the experiment's calib-dir.

Current Behavior

We don't have that.

Possible Solution

look for each supported command what the option is to pass a directory. Make sure that also the pedestal_workdir needs to be changed.

makepeds : Jungfrau constants

makepeds should have an option to deploy non-geometry calibration files like the gain & offset files.

Expected Behavior

makepeds should add an option or expand -g (geometry) potion to also copy the gain&offset fl;ies/

Current Behavior

We copy files by hand which is not the preferred method and gets forgotten,

Possible Solution

use 'jungfrau_gain_constants'
implementation should be similar to use of 'calibfile' for the geometry

motor-expert-screen error messages with NoMachine

Expected Behavior

I suspect that this does not work when not running locally (or at least not through NoMachine). There should be some test to only move the motor when this is a 'direct' session if possible. I have not tried to use 'plain' ssh & x forwarding.

Current Behavior

When running motor-expert-screen via NoMachine, I get an error message regarding some failed test in line 141. This is where xdotool is used to get the screen position and we try to move the screen to where the mouse is.

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

pcds_conda drops you into the new env even before it's ready

Expected Behavior

source pcds_conda without the env variable set should drop you into the latest environment that exists and is fully-formed.

Current Behavior

If I've pressed the "deploy env" button but the environment isn't complete, it still drops you into the new environment.

Possible Solution

Maybe there's an automatic way to know if the env is ready?
Or maybe it's better and safer to hard-code the "latest" env name and update it manually once ready.

Ioctool does not work for the pytmc-generated twincat IOC autosave files because the file names are different

          This branch does not work for the pytmc-generated twincat IOC autosave files because the file names are different:

$ ls /cds/data/iocData/ioc-lfe-motion/autosave/
info_positions.req                info_positions.sav_210325-173832  info_positions.sav_230119-152437  info_settings.sav_201118-210022  info_settings.sav_210823-132817
info_positions.sav                info_positions.sav_210325-175539  info_positions.sav_230121-060135  info_settings.sav_210106-165713  info_settings.sav_210918-133200
info_positions.sav0               info_positions.sav_210326-165516  info_positions.sav_230121-123032  info_settings.sav_210115-133120  info_settings.sav_211005-100349
info_positions.sav1               info_positions.sav_210402-105142  info_positions.sav_230204-164422  info_settings.sav_210119-153527  info_settings.sav_211012-101144
info_positions.sav2               info_positions.sav_210423-095513  info_positions.sav_230220-064720  info_settings.sav_210211-110618  info_settings.sav_211021-102332

Originally posted by @ZLLentz in #95 (comment)

Pythonify and clean up evrStatus script

The next go at this should make it a Python script, because it's the simplest scripting language with strong support for data structures. Without data structures (even just simple things like dicts), making scripts like this becomes more difficult than it needs to be, and then becomes a maintenance hazard when something changes and it needs to be modified.

As a Python script we could even start pulling in useful bits into our modules and re-use them elsewhere in a more fluid way.

Originally posted by @ZLLentz in #67 (comment)

ipmConfigEpics hutch override

Expected Behavior

Have a hutch override to change which hutch's ipms can be accessed. This would allow all options to be tested from one hutch without needing to ssh to another node to check if a given ipm works.

Current Behavior

No such override exists. The script currently uses get_info --gethutch to determine which hutch to use.

Possible Solution

Add an option like -H <hutch-name> to override. Change get_hutch handling elsewhere in the script.

Context

For example, this would be helpful if we want to check an XCS box from XPP because that is where we currently are.