comcifs / instrument-geometry-info Goto Github PK

This is an excerpt of #42
There are some differences between the current output of the imgCIF_creator and the published imgCIFs, which are mainly related to different naming in the hdf5 file which were manually renamed before:

loop_
  _axis.id
  _axis.type
  _axis.equipment
  _axis.depends_on
  _axis.vector[1]
  _axis.vector[2]
  _axis.vector[3]
  _axis.offset[1]
  _axis.offset[2]
  _axis.offset[3]
         chi       rotation  goniometer          detx      -0.0046   0.0372    -0.9993  0        0        0        
         omega     rotation  goniometer          .         1         0         0        0        0        0        
         two_theta           rotation  goniometer          .         1         0        0        0        0        0        
         phi       rotation  goniometer          chi       1         -0.0037   0.002    0        0        0        
         detx      translation         detector  dety      -1        0         0        -166.87357         172.49715          0        
         dety      translation         detector  trans     0         1         0        0        0        0        
         trans     translation         detector  omega     0         0         -1       0        0        -287.22243

==> chi should depend on omega
==> in the publication (he4557) _axis.equipment for two_theta was detector, not goniometer
==> trans should depend on two_theta
==> detx should depend on trans
==> dety depends on detx
==> detx vector should be [1 0 0]
==> dety vector should be [0 -1 0]

1. chi should depend on omega: in the h5 master it was detx, and in the imgcif_mapper I looped through the dependency chain until the base (omega) was reached. I did not do this in the creator yet, but I can implement it again.

2. _axis.equipment for two_theta was detector, not goniometer: I've just duplicated the omega axis and renamed it because it's the same except for the equipment type and was not present in the h5 master. I was trying to be more generic instead of hardcoding things. Is it ok to duplicate the omega axis, rename it to two_theta and replace the equipment with detector?

3. In general is it correct that two_theta and detector_2theta are different axes, so the first for the goniometer and the second for the detector? Are they always present?

4. trans should depend on two_theta: I inferred the depends_on field from the /entry/sample/transformations/sam_z entry which depends on omega, previously this was manually set to two_theta. It always feels a bit arbitrary to replace things manually... I could replace it again.

5. detx should depend on trans: this was inferred from /entry/sample/transformations/sam_x which depends on sam_y so this also requires manual renaming

6. dety depends on detx: this was inferred from /entry/sample/transformations/sam_y which depends on sam_z so this also requires manual renaming

7. detx vector should be [1 0 0] and dety vector should be [0 -1 0]: this is the same issue as mentioned in the last mail and I did not change that yet:

You're also right with the axis vectors, that is something I wanted to discuss, but I see that I did not mention this yet... In the hdf5 file there is the /entry/instrument/detector/module/fast_pixel_direction with vector [-1,0,0] (used previously) and in /entry/sample/transformations/sam_x the vector is [1,0,0]. I was using the second one for convenience, because I thought it was the same and realized later that it's different and wanted to discuss if the first one is the correct one.

Name of manufacturer

Stoe

Model

STOE Stadivari

Location

Benemerita Uiversidad Autonoma de Puebla

Principal axis orientation

270

Goniometer axes

Phi, a, Chi, a, Omega, a

kappa

No response

chi

180

Image orientation

top left

Fast direction

horizontal

Pixel size

0.172

Number of pixels

195, 487

Two theta axis

a

Other detector axes

No response

xds.inp

No response

Data DOI

No response

Comments

second try

Name of manufacturer

Bruker

Model

Kappa APEXII

Location

Utrecht University

Principal axis orientation

270

Goniometer axes

Phi, c, Kappa, c, Omega, c

Rotation axis name

Phi

kappa

Kappa 50

Image orientation

top left

Fast direction

horizontal

Pixel size

0.075

Two theta axis

clockwise

Other detector axes

No response

xds.inp

No response

Data DOI

doi.org/10.5281/zenodo.6365376

Comments

No response

_axis.equipment of gravity gives the downward direction. Useful for orienting the detector image.

I used creator to generate an imgCIF from b4_1_master.h5. In addition to my mail of 03-02-2023 I have some comments:
loop
_axis.id
_axis.type
_axis.equipment
_axis.depends_on
_axis.vector[1]
_axis.vector[2]
_axis.vector[3]
_axis.offset[1]
_axis.offset[2]
_axis.offset[3]
chi rotation goniometer detx -0.0046 0.0372 -0.9993 0 0 0
omega rotation goniometer . 1 0 0 0 0 0
two_theta rotation goniometer . 1 0 0 0 0 0
phi rotation goniometer chi 1 -0.0037 0.002 0 0 0
detx translation detector dety -1 0 0 -166.87357 172.49715 0
dety translation detector trans 0 1 0 0 0 0
trans translation detector omega 0 0 -1 0 0 -287.22243

==> chi should depend on omega
==> in the publication (he4557) _axis.equipment for two_theta was detector, not goniometer
==> trans should depend on two_theta
==> detx should depend on trans
==> dety depends on detx
==> detx vector should be [1 0 0]
==> dety vector should be [0 -1 0]

loop_
_array_structure_list_axis.axis_id
_array_structure_list_axis.axis_set_id
_array_structure_list_axis.displacement
_array_structure_list_axis.displacement_increment
array_x 1 0.0375 0.075
array_y 2 0.0375 0.075

==> _array_structure_list_axis.axis.id is a pointer to _axis.id above. So I think that array_x should be detx, and likewise for array_y

loop_
_diffrn_detector_axis.axis_id
_diffrn_detector_axis.detector_id
det_z det1

==> again _diffrn_detector_axis.axis.id is a pointer to axis.id, and thus should be trans
loop
_diffrn_scan_axis.scan_id
_diffrn_scan_axis.axis_id
_diffrn_scan_axis.displacement_start
_diffrn_scan_axis.displacement_increment
_diffrn_scan_axis.displacement_range
_diffrn_scan_axis.angle_start
_diffrn_scan_axis.angle_increment
_diffrn_scan_axis.angle_range
SCAN1 chi . . . 0.0 0 0
SCAN1 omega . . . 0.0 0.1 360.0
SCAN1 phi . . . 0.0 0 0
SCAN1 detx . . . 0.0 0 0
SCAN1 dety . . . 0.0 0 0
SCAN1 trans 287.2224260231453 0 0 . . .

==> only omega is scanned. Leave the other _axis.id out?? I don't mind if they stay in, but normally we use only one scan-axis

The construction with _diffrn_scan_axis and diifrn_scan_frame is different from the one we published (he5447) but seems OK to me.
loop
_diffrn_data_frame.id
_diffrn_data_frame.detector_element_id
_diffrn_data_frame.array_id
_diffrn_data_frame.binary_id
frm1 ELEMENT01 IMAGE01 1
frm2 ELEMENT01 IMAGE01 2
==> I have a question to @jamesrhester : _diffraction_data_frame.id should be a pointer to _array_structure.id but that data name is never given. Is that a problem?

==> Then, _diffrn_data_frame.array_id is also a pointer to _array_structure.id, so should be the same as _diffraction_data_frame.id ?
Now they are IMAGE01 and frm1 respectively.
==> I cannot find _array_data.id in the dictionary?

@jamesrhester can you please look if the construction with pointers to data is complete, not redundant and does not contain unnecessary data items

Name of manufacturer

Stoe

Model

Stadivari

Location

Benemerita Uiversidad Autonoma de Puebla

Principal axis orientation

270

Goniometer axes

Phi, a, Chi, a, Omega, a

kappa

No response

chi

chi, 0

Image orientation

top left

Fast direction

horizontal

Pixel size

0.172

Two theta axis

a

Other detector axes

No response

xds.inp

No response

Data DOI

10.5281/zenodo.7155191

Comments

No response

Facility name

ESRF

Beamline name

ID23

Is updated?

• Is an update

Start date

No response

Finish date

2018-04-18

Principal axis orientation

0

Goniometer axes

omega, c

Rotation axis name

omega

kappa

No response

Image orientation

top left

Fast direction

horizontal

Pixel size

0.172, 0.172

Number of pixels

2463, 2527

Detector axes

No response

xds.inp

BACKGROUND_RANGE= 1 10


!masking non sensitive area of Pilatus
UNTRUSTED_RECTANGLE= 487  495     0 2528
UNTRUSTED_RECTANGLE= 981  989     0 2528
UNTRUSTED_RECTANGLE=1475 1483     0 2528
UNTRUSTED_RECTANGLE=1969 1977     0 2528
UNTRUSTED_RECTANGLE=   0 2464   195  213
UNTRUSTED_RECTANGLE=   0 2464   407  425
UNTRUSTED_RECTANGLE=   0 2464   619  637
UNTRUSTED_RECTANGLE=   0 2464   831  849
UNTRUSTED_RECTANGLE=   0 2464  1043 1061
UNTRUSTED_RECTANGLE=   0 2464  1255 1273
UNTRUSTED_RECTANGLE=   0 2464  1467 1485
UNTRUSTED_RECTANGLE=   0 2464  1679 1697
UNTRUSTED_RECTANGLE=   0 2464  1891 1909
UNTRUSTED_RECTANGLE=   0 2464  2103 2121
UNTRUSTED_RECTANGLE=   0 2464  2315 2333
TRUSTED_REGION=0.0 1.41 !Relative radii limiting trusted detector region

!correction tables to compensate the misorientations of the modules

X-GEO_CORR=./x_geo_corr.cbf
Y-GEO_CORR=./y_geo_corr.cbf


MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT= 3

!STRONG_PIXEL= 3.0

OSCILLATION_RANGE= 0.1500
STARTING_ANGLE= 101.0
STARTING_FRAME= 1
X-RAY_WAVELENGTH=  0.97200
NAME_TEMPLATE_OF_DATA_FRAMES= ../data/270218_C124_SQ109_B4_CRYO_14_w1_1_????.cbf

!STARTING_ANGLES_OF_SPINDLE_ROTATION= 0 180 10
!TOTAL_SPINDLE_ROTATION_RANGES= 60 180 10

DETECTOR_DISTANCE= 155.00
DETECTOR= PILATUS MINIMUM_VALID_PIXEL_VALUE= 0.0 OVERLOAD= 1048500


SENSOR_THICKNESS=0.32
ORGX= 1217.51 ORGY= 1266.16
NX= 2463 NY= 2527
QX= 0.1720  QY= 0.1720
VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS= 7000 30000

DIRECTION_OF_DETECTOR_X-AXIS= 1.0 0.0 0.0
DIRECTION_OF_DETECTOR_Y-AXIS= 0.0 1.0 0.0
ROTATION_AXIS= 1.0 0.0 0.0
INCIDENT_BEAM_DIRECTION= 0.0 0.0 1.0
FRACTION_OF_POLARIZATION= 0.99
POLARIZATION_PLANE_NORMAL= 0.0 1.0 0.0

SPACE_GROUP_NUMBER=4 
UNIT_CELL_CONSTANTS= 51.38 75.12 56.46 90.0 107.03 90.0 

INCLUDE_RESOLUTION_RANGE= 30 1.25
!RESOLUTION_SHELLS= 15.0 8.0 4.0 2.8 2.4
!FRIEDEL'S_LAW= FALSE !default is TRUE
!STRICT_ABSORPTION_CORRECTION=TRUE

REFINE(INTEGRATE)= BEAM ORIENTATION CELL
!== Default value recommended
!DELPHI= %.3f
BEAM_DIVERGENCE_E.S.D.=     0.042
REFLECTING_RANGE_E.S.D.=     0.123
MAXIMUM_NUMBER_OF_PROCESSORS= 3  
MAXIMUM_NUMBER_OF_JOBS=       2  
SPOT_RANGE=591 601

SPOT_RANGE=291 301

FRIEDEL'S_LAW=TRUE
SPOT_RANGE=591 601

SPOT_RANGE=291 301

Data DOI

https://zenodo.org/record/3816663/files/CYP124_SQ109.tar.bz2

Comments

Rotation axis based on XDS.INP having rotation axis parallel to detx and in same direction.

The rsync: protocol used by SBGrid acts the same as the file: protocol in that the per-frame URIs are formed simply by appending the frame file name to the URI.

miniCBF headers like giving values of -9999 for axes that are not present. So, for example, SBGRID 952 has the following miniCBF header:

# Detector: PILATUS3 2M, S/N 24-0124
# 2020/Mar/19 17:43:25
# Pixel_size 172e-6 m x 172e-6 m
# Silicon sensor, thickness 0.001 m
# Oscillation_axis omega
# Excluded_pixels:  badpix_mask.tif
# Chi 0.0000 deg.
# Angle_increment 0.1000 deg.
# Polarization None
# file_comments 
# N_oscillations 1800
# Beam_xy (747.00, 828.00) pixels
# Exposure_time 0.150000 s
# Phi -9999.0000 deg.
# Energy_range (0, 0) eV
# Start_angle -118.8000 deg.
# Detector_distance 0.324127 m
# Detector_Voffset 0.0000 m
# Alpha 0.0000 deg.
# Flat_field: (nil)
# Threshold_setting 6750 eV
# Exposure_period 0.153000 s
# N_excluded_pixels: = 321
# Kappa -9999.0000 deg.
# Tau = 0 s
# Transmission 100.0
# Detector_2theta 0.0000 deg.
# Flux 1e+11
# Count_cutoff 1048500
# Trim_directory: (nil)
# Wavelength 0.918400 A

from which it is clear that there are no phi or kappa axes, but that there is a chi axis. However, creator includes phi and kappa axis definitions.

The detector axes are given in pixels instead of mm.

Facility name

Other (please write name in the comments at end of form)

Beamline name

Test beamline

Configuration

Test

Is updated?

• Is an update

Start date

2021-05-05

Finish date

No response

Phi axis orientation

180

Phi axis rotation

clockwise

Goniometer axes

Chi, clockwise, omega, anticlockwise

Image orientation

top left

Fast direction

horizontal

Detector axes

two_theta/ rotation/ centred on goniometer
detector_chi/rotation/ axis through centre of detector

xds.inp

No response

Data DOI

No response

Comments

This issue is a test issue for working on automatic processing of issues and ironing out other bugs!

This arose from discussion of #36 . An option --preserve-axis-names can be provided in the command line to allow the axis names found in the minicbf to be used for the imgCIF instead of a standard list of names. I think this is a low priority.

This is an excerpt of #42
There are some differences between the current output of the imgCIF_creator and the published imgCIFs:

loop_
_array_structure_list_axis.axis_id
_array_structure_list_axis.axis_set_id
_array_structure_list_axis.displacement
_array_structure_list_axis.displacement_increment
array_x 1 0.0375 0.075
array_y 2 0.0375 0.075

==> _array_structure_list_axis.axis.id is a pointer to _axis.id above. So I think that array_x should be detx, and likewise for array_y

loop_
_diffrn_detector_axis.axis_id
_diffrn_detector_axis.detector_id
det_z det1

==> again _diffrn_detector_axis.axis.id is a pointer to axis.id, and thus should be trans
loop
_diffrn_scan_axis.scan_id
_diffrn_scan_axis.axis_id
_diffrn_scan_axis.displacement_start
_diffrn_scan_axis.displacement_increment
_diffrn_scan_axis.displacement_range
_diffrn_scan_axis.angle_start
_diffrn_scan_axis.angle_increment
_diffrn_scan_axis.angle_range
SCAN1 chi . . . 0.0 0 0
SCAN1 omega . . . 0.0 0.1 360.0
SCAN1 phi . . . 0.0 0 0
SCAN1 detx . . . 0.0 0 0
SCAN1 dety . . . 0.0 0 0
SCAN1 trans 287.2224260231453 0 0 . . .

==> only omega is scanned. Leave the other _axis.id out?? I don't mind if they stay in, but normally we use only one scan-axis

Name of manufacturer

Stoe

Model

Stoe Stadivari with Chi axis

Location

Benemerita Uiversidad Autonoma de Puebla

Principal axis orientation

270

Goniometer axes

Phi, a, Chi, c, Omega, a

kappa

No response

chi

180

Image orientation

top left

Fast direction

horizontal

Pixel size

0.172

Number of pixels

195, 487

Two theta axis

a

Other detector axes

No response

xds.inp

No response

Data DOI

10.5281/zenodo.7155191

Comments

try to get chi right

Facility name

No response

Beamline name

I04

Configuration

Eiger 16M

Is updated?

• Is an update

Start date

No response

Finish date

No response

Phi axis orientation

0 degrees

Phi axis rotation

anticlockwise

Goniometer axes

omega, cantilockwise, chi, anticlockwise, phi, anticlockwise

Image orientation

top left

Fast direction

horizontal

Detector axes

none

xds.inp

No response

Data DOI

No response

Comments

No response

The zenodo.2611942 dataset consists of 4 .tar.xz files which all extract into a single directory. In order to generate a correct remote location for the frames, the correct tar.xz file associated with each frame needs to be determined.

According to @kroon-lab doc for necessary items and a couple I've added, the following are currently missing from cbf_metadata.cif:

_diffrn_scan_axis.angle_range is missing
_diffrn_scan_axis.displacement_range is missing
_diffrn_radiation.type is missing
_array_data.array_id is missing
_array_data.binary_id is missing

To fix the last two, a single column _array_data.array_id should be added with value 1 and the current _array_data.id should be _array_data.binary_id

Also the three _array_structure items at the top should have a period (.) after _array_structure, not an underscore.

And _diffraction_radiation.type should be _diffrn_radiation.type

I just create an issue from your mail @kroon-lab, to keep track

If I give the URLs like they are on Zenodo, I can only give one name, whereas there are four different data files (_b4_1_000001.h5, _b4_1_000002._b4_1_000002.h5,h5,_b4_1_000002.h5). So giving one URL is not enough.

Command-line options could be provided that would allow a user to define one or more arbitrary axes, instead of relying on our internal heuristics and pre-programmed list. I think this is a lower priority task. Arose in discussion of #36

Name of manufacturer

Stoe

Model

Stadivari

Location

Benemerita Uiversidad Autonoma de Puebla

Principal axis orientation

270

Goniometer axes

Phi, a, Chi, c, Omega, a

kappa

No response

chi

180

Image orientation

top left

Fast direction

horizontal

Pixel size

0.172

Number of pixels

195, 487

Two theta axis

a

Other detector axes

No response

xds.inp

No response

Data DOI

10.5281/zenodo.7155191

Comments

No response

Facility name

Alba

Beamline name

XALOC

Is updated?

• Is an update

Start date

2015-04-19

Finish date

No response

Principal axis orientation

0

Goniometer axes

angle, a

Rotation axis name

angle

kappa

No response

Image orientation

top left

Fast direction

horizontal

Pixel size

0.172, 0.172

Number of pixels

2463, 2527

Detector axes

No response

xds.inp

No response

Data DOI

https://zenodo.org/record/820648/files/PDB_5J1G_NATIVE_diffraction_images.tgz

Comments

The only unknown for the beamline geometry is the direction of the rotation axis. The present submission is a guess in order to generate a geometry for testing. Also, the miniCBF files at the above location do not indicate the oscillation axis beyond the cryptic "X, CW" notation. The PHI axis is fixed at zero for all scans, so presumably something like omega is scanned.

Name of manufacturer

Stoe

Model

Stadivari

Location

Benemerita Universidad Autonoma Puebla

Principal axis orientation

270 degrees

Goniometer axes

Phi, goto 1

Rotation axis name

No response

kappa

No response

Image orientation

No response

Fast direction

No response

Pixel size

No response

Two theta axis

No response

Other detector axes

No response

xds.inp

No response

Data DOI

No response

Comments

No response

Facility name

Diamond

Beamline name

I04

Is updated?

• Is an update

Start date

No response

Finish date

No response

Principal axis orientation

0 degrees

Goniometer axes

phi, c, chi,c, omega, c

Rotation axis name

omega

Image orientation

top left

Fast direction

horizontal

Detector axes

none

xds.inp

No response

Data DOI

No response

Comments

No response

The original issue_to_imgcif tool did not access the raw data. A tool with access to the raw data can:

1. Suggest default axis names and stacking based on typical items found in known headers
2. Ignore axes that do not move during any of the scans
3. Determine the array dimensions from the CBF Mime header (guaranteed to be available)

A publically-available dataset is provided at https://zenodo.org/record/2611942 for testing so this should be easy to do, especially as each frame contains a full header.

When trying to enter the value "-173" as a Celsius temperature the input was rejected.

Currently the offset for the trans/distance axis in the axis description is taken from the first scan and I was wondering if this is ok if there are multiple scans with different trans/distance settings. Should the offset be 0 in the axis description?

Name of manufacturer

Stoe

Model

Stadivari

Location

Benemerita Uiversidad Autonoma de Puebla

Principal axis orientation

270

Goniometer axes

Phi, a, Chi, a, Omega, a

Rotation axis name

Omega

kappa

No response

Image orientation

top left

Fast direction

horizontal

Pixel size

0.172

Two theta axis

a

Other detector axes

No response

xds.inp

No response

Data DOI

No response

Comments

In the mini-cbf header all rotation axes are defined by looking along the axis away from the goniometer, and a positive rotation is closewise. If we invert the axes definitions, the signs of the rotation angles should be reversed.

Name of manufacturer

Stoe

Model

STOE Stadivari

Location

Mexico

Principal axis orientation

270

Goniometer axes

Phi, a, Chi, a, Omega, a

kappa

No response

chi

Chi, 180

Image orientation

top left

Fast direction

horizontal

Pixel size

0.172

Number of pixels

195, 487

Two theta axis

a

Other detector axes

No response

xds.inp

No response

Data DOI

No response

Comments

No response

The Zenodo record 261942 contains cbf files that are "full" cbf. Every frame contains a full imgCIF description of the scan that it is part of together with the axis descriptions. If such frames are available a complete imgCIF file can be generated with the only user input being to provide the remote location of the data.

Name of manufacturer

Bruker

Model

KAPPA APEX II

Location

Utrecht University

Principal axis orientation

270 degrees

Goniometer axes

Phi, c, Kappa, c, Omega, c

Rotation axis name

Phi

kappa

Kappa, 50

Image orientation

top left

Fast direction

vertical

Detector axes

-1 0 0, rotation

xds.inp

No response

Data DOI

No response

Comments

No response