ess-dmsc / nexus-constructor Goto Github PK

Construct NeXus templates and files with instrument geometry information using a GUI

License: BSD 2-Clause "Simplified" License

Python 20.24% Makefile 0.29% XSLT 0.53% Batchfile 0.09% Shell 0.06% CSS 0.46% HTML 77.33% C 0.16% Java 0.02% MATLAB 0.19% Fortran 0.06% JavaScript 0.55%

pyside2 nexus-files qtforpython ess nexus-constructor gui

nexus-constructor's Introduction

NeXus Constructor

The NeXus Constructor facilitates constructing NeXus files in which to record data from experiments at neutron science facilities. This includes all supporting metadata typically required to perform analysis of such experiments, including instrument geometry information.

The application can directly output a NeXus file, or create a NeXus structure, JSON template of the file to send to the File Writer. The JSON template can include data stream details, such that the File Writer populates the NeXus file it creates with data acquired from the detector and other beamline apparatus during an experiment.

There is a "Getting Started" guide to help new users.

Although the application may be useful to other institutions using NeXus, the NeXus Constructor is being developed as part of the software suite for the European Spallation Source (ESS). Please see this (open access) paper for more information on the ESS software.

The NeXus Constructor should work on MacOS, Linux and Windows; however, none of the current developers use Windows, so occasionally there are issues on Windows - if you find one then please let us know!

Installing dependencies

Binary packages for release versions can be downloaded on the releases page, or to run the latest development version please follow the instructions below.

Python 3.8 or greater is required. https://www.python.org/downloads/

It is HIGHLY recommended that a virtual environment is created for running the nexus-constructor.

Runtime Python dependencies are listed in requirements.txt at the root of the repository. They can be installed from a terminal by running

pip install -r requirements.txt

For MacOS Ventura, it is currently necessary to install libomp manually, e.g. via brew, MacPorts, etc.

Usage

Run the Python script nexus-constructor.py located in the root of the repository.

A guide for getting started with the nexus-constructor can be found here

Developer information

Development dependencies

Development dependencies (including all runtime dependencies) can be installed by using the following command

pip install -r requirements-dev.txt

Install the pre-commit hooks by running the following command

pre-commit install

Now when a commit is made, the hooks will be run automatically.

For more information on pre-commit see the official documentation

Developer Documentation

See the Wiki for developer documentation.

Contributing

The project

The project is managed on the ESS's JIRA system which is here.

If you do not have an ESS account but want to raise an issue then please open an issue on the repository. Someone will triage it and link it from the JIRA system.

Pull requests

Pull requests are very welcome and should be done through GitHub in the standard way.

Code

If you are submitting code then please ensure it passes the pre-commit checks (see above).

nexus-constructor's People

Contributors

Stargazers

Watchers

Forkers

trnielsen

nexus-constructor's Issues

Try using pytest-qt to create a unit test

See if https://github.com/pytest-dev/pytest-qt is useful. Does it work when using QML or only with QWidgets?

Sketch out full interface design

From this we can identify components to make and prioritise implementation tickets.

Some example ideas for details:

What information and controls are needed for each instrument component type?
How can relative position + orientation information for each component be entered and displayed?
How can "shapeless" components be represented in the 3D view?
Display axes and/or grid lines in the 3D view? (https://forum.qt.io/topic/66808/qt3d-draw-grid-axis-lines).
Colour match elements in the instrument components list with models in the 3D view?
Component type selector, combo box in listview or a modal window when "add component" button is pressed?
Do we need icons (for different component types for example?). There are some resources like https://thenounproject.com/ or we may need to make some.
Switch between wireframe and solid in 3D view?
File writing controls visible in main interface or accessible through MenuBar?
Pixel mapping interface (spreadsheet/tableview style interface? label pixels in 3D view?)

Add documentation to README

In the readme briefly explain how to:

install dependencies with pip using the requirements file.
run the unit tests.

Prepopulate ListView with sample component

sample is the origin of the coordinate system, there must be one, and there can only be one. It therefore makes sense to prepopulate the ListView with this component so that it is always an available option when selecting relative transformations.

Create GUI testing document

Due to problems with, and difficulties using Pyside2, QML, and pytest-qt together, automated GUI tests were removed in 2cf70fb.
To ensure the GUI functions prior to making releases, we will need a document describing manual tests for the interface and their expected outcomes.

Add more functionality to the interface

For example try adding buttons to add or remove lines from the list model.
You could try some of the other Qt Quick Controls too.
Controls can also have tool tips: https://doc.qt.io/qt-5/qml-qtquick-controls2-tooltip.html

I don't think there is any reason not to use the latest version of QtQuick: import QtQuick 2.11

Add component type selector dialog

Add component type selector dialog to "add component" button. I suggest "source" (see NXsource documentation) and "detector" options would be good to start with.

Add OFF loading and rendering functionality

Talk with @matthew-d-jones about this, he has some existing Python code which can be adapted.

Multiple transformations per component

We need to support multiple transformations per component. Therefore, with the except of the sample (which define the origin), transformations should be relative to another transformation rather than to a component.

Perhaps they could be added as child elements to the components in the listview?

Add a qtquickcontrols2.conf file to configure appearance

https://doc.qt.io/qt-5/qtquickcontrols2-configuration.html

Create Window with main controls for real implementation

Start "real" implementation and rename existing directory to indicate it is the prototype.
Create an application window with the main controls: Scene3D, component ListView, add component button and MenuBar.

Add controls for loading shape from OFF file

Add controls for loading shape from OFF file for all component types.

Use Black for formatting

https://black.readthedocs.io/en/stable/

Black is becoming quite popular in the Python world as a no holds barred formatter.
I propose we use that as it saves time formatting code and we don't need to discuss PEP8 conformance.

Perhaps it can be added as a git pre-commit hook?

Add load from JSON

The JSON format to support loading from is that which can be read by the File Writer: https://github.com/ess-dmsc/kafka-to-nexus/

An example can be generated using this script: https://github.com/ess-dmsc/nexus-json/blob/master/SANS2D_example.py

Create a Window with PySide2 + QML

Try creating a Window containing a Listview using PySide2 and Qt Quick 2 (QML).

Dummy object in 3D view for component with no shape

A "None" option should be available for shape of any component type. Selecting "None" would indicate that a dummy object should be placed in the 3D view at the position of the component, but that there would be no geometry for the component in the output file.

Add code coverage

Using pytest and codecov.io

Use Pane QML type rather than Rectangle

Using Pane in place of Rectangle will use the correct styling for background colour.

Write to NeXus file

To be non-lossy, for example not loose colour information for models, we may need to add some custom fields.

Single component visualisation in Edit Component Window

Being able to view the current component would be useful for knowing details have been entered correctly, and I feel essential for a useful face number to pixel id mapping interface.
However, due to a bug/bugs in Qt3D, it's not currently possible to have Scene3D instances in child windows.

Examples that seem to match my experiences trying to add a Scene3D to the editor window can be found in the Qt bug tracker at:

Some of these have linked code, one currently due to release with Qt 5.11.3, and Qt 5.12.0, so we should wait for the next release before tackling this issue

Output File Writer json commands

Related to #53 (Load File Writer json)

Links to the format can be found in that issue.

Disable face culling in the 3D renderer

When setting up the custom mesh graphics buffers, each triangle is repeated with a reverse winding to ensure it renders regardless of which side is facing the camera.

However, it appears that face culling can be disabled with the right configuration. If this can be achieved, the duplicates in the graphics buffer won't be needed, and the memory they consume will be cut in half.

Add a JSON visualiser to complement the 3D display

Status indicator for long running tasks

File loading, mesh generation, json updating can all take a while with large files. Some indicator of what's going on would be useful.

Avoid javascript business logic

As discussed following #14 we want as close to all of the business logic of the application to be done in python instead of QML javascript so we're not introducing additional languages to the teams projects.

Consider supporting reading shape from STL or STEP format

The Design Division at ISIS uses "Solid Edge" CAD software which does not output to OFF or OBJ without an expensive plugin. It does output to STL or STEP, so if you estimate a day or less to implement support for one of these formats then it will be worthwhile, otherwise we'll manage with converting via an intermediate application for now.

The relevant (shape, not texture etc) information in such files is pretty much the same as in OFF: vertex coords and face definitions using vertex indices. My hope is therefore that it would be low effort to introduce support.

Looks like there is at least one suitable python library for reading STL files: https://pypi.org/project/numpy-stl/

Remove prototype code

The prototype folder hasn't been touched in months, and there's nothing in it that hasn't been also implemented in the main program.
The prototype build has been tagged, so should be easy to find.

I think it's time to remove it's code from master.

Add a pytest test

Just an example test to get us started and give something for our CI system (Jenkins) to run.
The test script can go in a /tests directory.

Load from json description

This is likely to raise other, blocking issues. For example, what if the json description has an NXtransformation which is not in an instrument component?

Leave validation to #472

Display axes in 3D view

For example, as seen in the bottom-left corner of this screenshot from Blender

Add common component controls to ListView elements

Position, orientation, what component these are with respect to, delete button and update button.

Add linting

Jenkins should run flake8

Look into nexusformat library

Try out https://github.com/nexpy/nexusformat and see whether it will be useful for this project. The NeXus Geometry classes themselves are very new additions to NeXus so if we decide to use this library we may have to update it to support these classes.

Try creating expandable list view

One idea would be to have instruments components (detector panel, sample, source, etc) as elements in a listview, with more details visible when the element is selected. This is a, somewhat old, example: http://doc.qt.io/archives/qt-4.8/qt-declarative-modelviews-listview-expandingdelegates-example.html

Information common to every component would be shape and position+orientation with respect to another component.

Windows executable

Having a single file executable or installer early in development would make life easier for potential testers of the application.
pyinstaller and pyupdater are probably the standard tools for this at the moment, worth checking though.

Correct the geometry nexus group names

For NXdetector groups:
detector_shape if NXoff_geometry containing detector_faces, or NXcylindrical_geometry containing detector_number
pixel_shape otherwise

and shape in any other group.

Try embeding a Scene3D in the interface

Hopefully this is as simple as adding a few lines to the QML...

Remove non-standard json description

We can currently output to our NeXus-json description compatible with the file writer, and another json description. The latter of these should be removed. It may be in use because the NeXus-json description is lossy, in which case may need to wait until we have a non-lossy save format (NeXus file?) before removing it.

See "JSON Formats" section of README, and make sure README is updated as part of this ticket.

Move developer notes to a separate markdown document

Move developer notes to a separate markdown document and link to it from the main README

Look into using an undo/redo stack to revert changes in the model

Should allow live updates to the model, which can be reflected in a visualisation, while still allowing changes to be undone.

Look into a windows jenkins build

Doesn't look there currently is one. It shouldn't need to "build" as it's python, but would be nice to run tests on it.

Update Qt to 5.12

This should give us TableView in Qt Quick Controls 2, which may be useful. Hopefully also solve blocker for #56

Input validation and error display

Ensure that number fields are numeric, and that translate parent's don't form circular dependencies.

Add load from Kafka

Blocked by #53

Get the last "write start" message in the given topic and use the JSON parser implemented for #53 to load geometry information from it.
This would provide a useful tool to visually check geometry information which was sent to the File Writer.

Improve memory usage in repeated mesh detector graphics buffers

Currently, the QtOFFGeometry class repeats the mesh for each instance in a grid.

This means the graphics buffers use much more memory than needed, directly proportional to the number of instances in the grid. Instead, it should just create the mesh for a single instance, and a model should create unique transform matrices for each pixel instance that can be used in the QML.