apeck12 / sfx_utils

Miscellaneous functions for aiding calibration during SFX experiments at LCLS.

sfx

Miscellaneous functions for aiding calibration during SFX experiments at LCLS.

Contributing

This repository is the centralized location to discuss and share calibration files, documentation, scripts and code for SFX analysis.

See Discussions above for a forum experience. Contribute or check Wiki above for tutorials and tips, shared experience, etc.

For contributing code and script, please don't push to the main branch directly. Rather, work in a specific branch for the task at hand, and make a pull request (PR) so it can be safely merged into the main branch after the person you assign as a reviewer does the code review (if necessary).

More importantly, have fun! If you have any questions, feel free to post a new issue or just reach out directly to someone else on the team.

Etiquette

Before you start working on contribution, please make sure your local main branch is up-to-date. Create a new branch from it, add or edit, then push a pull request.

To create your local repository (from this GitHub hosted remote repository):

$ git clone [email protected]:lcls-users/sfx.git

To update your local main branch:

$ git checkout main
$ git pull

Now you can create a new branch that will be up to date with the current (remote) main branch:

$ git checkout -b my-new-branch
$ ... do stuff ..
$ git add <the files that you have been working on>
$ git commit -m "Hey everyone, I have been working on those files to do this and that"
$ git push origin my-new-branch

At this point, checkout the repo on GitHub and create a Pull Request (like this one). Once merged, the main branch will be updated with your work!

sfx_utils's Issues

Script logic for elog automation

Timetool methods and documentation

Write stream files with a selection of the reflections from another stream file

(I know you already are working on it Ariana, just thought I'd put this here for my own records)

We'd like to be able to:

• read a given stream file
• do some analysis on the reflections
• select reflections based on this analysis
• write a stream file with only the reflections selected

MaskInterface visual to choose threshold values

First mentioned in PR #27. Provide a histogram of the image values so that the user has more information for choosing the threshold values used during masking.

MaskInterface: loading a cctbx-style mask for the Jungfrau4M

To enable this, we need to write a function that performs the reverse of correct_panel here:
https://github.com/cctbx/cctbx_project/blob/master/xfel/util/jungfrau.py
so that the reshaped mask matches the unassembled psana shape.

Description of how to get started

Add information on how to get the ssh key, get started with the jupyter hub and use the right kernel

Diagnostic statistics per-run

We'd like to pull out the following statistics per event and plot their trajectory over the course of a given run:

• total intensity
• minimum intensity
• maximum intensity
• median intensity
  By default, we assume that we're examining the calibrated images.

MPI parallelization of processing of runs

Best way to share geometry and masks files.

Let's discuss here what would be the most convenient way to do this?

Mask interface

Description of computer clusters and nodes available

Add a page in the wiki on what nodes are accessible from outside SLAC (pslogin, psexport), what nodes have access to the data, etc.

Make timetool_analysis class

Unit of detector pixel values?

I believe they are in keV in the assembled image, would be nice to check what they are otherwise - and document it somewhere.

If indeed in keV, it would be nice to document how to convert into "photons" and vice-versa.

Geometry conversion between psana, cctbx, crystfel for all LCLS detectors

Powder migration outside of GeomOpt

MemoryError when making a powder over too many images

sfx_utils/sfx_utils/geom_opt.py

Line 33 in 0f45dc6

self.powder = np.amax(self.psi.get_images(n_images), axis=0)

geom_opt.opt_distance(plot=True, n_images=40000)
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
/tmp/ipykernel_25483/1338048410.py in <module>
----> 1 geom_opt.opt_distance(plot=True, n_images=40000)

~/sfx_utils/sfx_utils/geom_opt.py in opt_distance(self, sample, n_images, center, plot)
     61         """
     62         if self.powder is None:
---> 63             self.powder = self.compute_powder(n_images)
     64 
     65         if sample == 'AgBehenate':

~/sfx_utils/sfx_utils/geom_opt.py in compute_powder(self, n_images, ptype)
     31         """
     32         if ptype == 'max':
---> 33             self.powder = np.amax(self.psi.get_images(n_images), axis=0)
     34         elif ptype == 'mean':
     35             self.powder = np.mean(self.psi.get_images(n_images), axis=0)

~/sfx_utils/sfx_utils/psana_interface.py in get_images(self, num_images, assemble)
     65             images = np.zeros((num_images, 
     66                                self.det.image_xaxis(self.run).shape[0],
---> 67                                self.det.image_yaxis(self.run).shape[0]))
     68         else:
     69             images = np.zeros((num_images,) + self.det.shape())

MemoryError: Unable to allocate 826. GiB for an array with shape (40000, 1666, 1664) and data type float64

Adding the option to compute a powder over the whole run

It might be useful to add an option to set n_images here so that it goes over all images in the run: https://github.com/apeck12/sfx_utils/blob/main/sfx_utils/geom_opt.py#L41

Maybe n_images=-1?

Intensity distribution trajectory

In addition to information about intensity statistics during a run, it would be helpful to get a visual display of how intensity is distributed in range and in time. This would allow to quickly identify calibration issues, and to identify regime changes during collection that would help for example pinpoint events where the detector could have been at risk of being damaged.

An idea could be to populate an array for each run intensity_distribution(n_events, n_intensity_bins) that could be displayed in the form of a colored table as sketched below.

Considerations on binning:

• The data array could have a finer binning with coarser binning being possible upon plotting (e.g. using plt.hexbin()).
• Several binning strategies could also be explored: linear or symlog.

Random extra thought: a similar plot for the whole experiment (with runs on the X-axis) might also be very useful for experiment summaries.