unibas-gravis / scalismo-ui

Visualization for Statistical Shape Models and Medical Images based on Scalismo.

Scalismo-ui

Scalismo-ui is a library for visualizing statistical shape models and related 3D objects, such as 3D meshes or 3D medical images. It is designed to complement the Scalismo framework and to make it easy to build graphical shape modelling applications and to allow for "visual debugging" of complex image analysis algorithms. Further, it can be used as a standalone application for visualizing shape models, 3D meshes and 3D medical images.

Scalismo-ui was developed and is currently maintained by the Graphics and Vision Research Group of the University of Basel.

Why

We believe that visualization is a key factor in the development of complex image analysis algorithm and that visual inspection is the most powerful debugging tool. But it is only used if visualization is simple to use. In Scalismo-ui, every object can be visualized by adding only one line of code:

val objectView = ui.show(group, object, "name of object")

It is equally simple to change the representation of the object, such as in this example its color

objectView.color = Color.RED

Usage

Scalismo-ui as a library

Add the following dependency to your build.sbt:

resolvers += Resolver.bintrayRepo("unibas-gravis", "maven")

libraryDependencies += "ch.unibas.cs.gravis" %% "scalismo-ui" % "0.92.0"

The graphical user interface can be called from an application using the command

val ui = ScalismoUI()

Standalone application

Scalismo-ui can also be used as a standalone application. Simply call

sbt run

to start the application. For more frequent use of scalismo-ui, it is convenient to have the application available as a jar file. This can be built by calling

sbt assembly

The resulting jar file is target/scala-3.1.0/scalismo-ui.jar

Documentation

Checkout the Scalismo-website for examples on how to use Scalismo-ui in your programs.

Undocumented features

Scalismo-ui features a simple plugin api. We hope to be able to add documentation soon. For the moment, check out the source file SimplePluginAPI.scala

For questions regarding scalismo-ui or suggestions for improvements, please post to the mailing list scalismo google group.

Maintainers

Scalismo-UI is maintained by researchers from the Graphics and vision research group at the University of Basel. See the following links for more information:

• Organisation unibas-gravis on Github
• Website at University of Basel.

Contributions

Contributions to scalismo-ui are very welcome. Please send us a pull request from your fork of this repository! We do have to ask you to sign the contributors agreement before we can merge any of your work into our code base.

Copyright and Licenses

All code is released under the GNU public license, available here.

Copyright, University of Basel, 2023.

scalismo-ui's Issues

Null-pointer exception when initializing the SimplePosteriorInteractor

Under some circumstances there are issues with the initialization order in the SimplePosteriorInteractor, leading to a null pointer exception. The null-pointer exception happens because the variable node-visibility is not yet initialized.

Slicing through small images does not have the right resolution

Problem

When a small image (i.e. one whose extend is only a few mm) is shown with ScalismoUI, slicing does not work properly. With the position slider only few slices can be shown.
My guess is that this is because the position slider in the ViewportPanel2D takes only integer values, which results in that only slices which are a mm apart are shown. For small images, this is not the right behavior.

Expected behaviour:

The number of slices that can be shown should always be large enough to explore the image, irrespective of the size of the image. One possibility would be to always show at least 100 slices (and more if the extend of the sliced scene is larger than 100 mm).

Workaround

Using shift-leftclick on the orthogonal planes correctly sets the slicing position. Holding shift clicked and moving the mouse can therefore be used to move navigate through the slices.

Group nodes visibility - Changing visibility of all render-able children

In order to hide all children of a group node, the user needs first to select them all to toggle the visibility. It would be convenient if the group node would allow setting the visibility for all children.

Simple API lacks a close function

Currently there is no way to close ScalismoUI programmatically using the SimpleAPI. As the SimpleAPI is supposed to be the only way a user interacts with ScalismoUI, such a functionality needs to be added.

One important use case where we desperately need this functionality is for the "runnable" tutorials using tut.

UI does not restore saved width/height

The ScalismoFrame class has internal logic to save its position and size when it is closed, and to restore to these values when it is reopened, to take into account the user's preferences.

However, when instantiated via scalismo.ui.api.ScalismoUI (as is the default when using the high-level API or the ScalismoViewer app), this functionality is lost, because after initialization, the frame gets re-packed to whatever Java thinks is a good layout (not what I thought was a good layout when I last ran it ; -) ).

This only affects the size and position, not perspective, display scaling etc. (which do get restored as expected).

To fix this, the call to frame.pack() should be removed here: https://github.com/unibas-gravis/scalismo-ui/blob/master/src/main/scala/scalismo/ui/api/ScalismoUI.scala#L33

(sorry for not posting a full PR, but this is a trivial change)

Change landmark size

The quickstart documentation shows how the colour of both meshes and landmarks in the Scalismo UI can be changed quite easily. Is there a similar functionality available for changing the size of landmarks? I found a previous reference to this issue on a google group, but no solution was available at the time.

Some of the meshes I am working with fit within a unit sphere - importing meshes landmarks results in beach ball sized landmark points that completely obscure the mesh and make it impossible to see if these landmarks actually represent the points that they should. Can the size of the landmarks be decreased in code? It does not seem to be possible fro the Scalismo UI.

Storing scene view settings

Implementation of the possibility to store scene view settings.
Mainly:

• Camera settings
• Slicing positions
• Window size
  This would make it easier to recreate exact screenshots.

Other possibilities would also be to store:

• Level, window and opacity of 3D images
• Opacity and color of meshes and landmarks

The settings can be saved to and loaded from a config file which also enables sharing of scene view settings.

ScalarMeshField 3D-visualisation

An issue with visualisation of scalar mesh fields in Scalismo-UI.
Setting both scalar range sliders to their minimum should give a complete red mesh. This works in the 2D slices, but in 3D, the vertices with the minimum value in the scalar range will still be coloured blue.

Taking screenshots does not work on Scalismo 0.91

Taking a screenshot either from the GUI or from the program, results in an image in which the object shown in the window is moved to the upper right corner.

Value probe returns unexpected results

Setting: Two groups with one or multiple nodes of type DiscreteScalarImage3D
Observation: Value probe (CTRL + mouse pointer) does not return the values of the currently visible image but of the one that was first 'made visible' in the scene, even if hidden by another image (from the same or a different group node).
Expected behaviour: Always return results for the currently visible image.
Problems: How to deal with overlay of images with opacity other than 0 or 100% ? Should the probe be deactivated in that case?

Saving of volume images from within the UI

Currently, there exists no option to save a volume image that is currently displayed in the UI. While meshes and other data structures can be stored when interacting with the scene tree with a right mouse button click.

I do not see any problem in allowing the images also to be saved through a RMB click.

After typing either key 'P' or 'F' the UI is frozen under Windows (not under Linux)

Under Windows 10 the UI freezes after typing one of the keys 'F' or 'P'. Typing the key once is enough. The same thing does not happen when using the exact same .jar under Linux. This bug was verified on two different window machines.

Steps to reproduce the bug:

1. Load the UI
2. Load some data (e.g. a .stl-mesh)
3. Verify that you can turn the shape using the mouse (left-draging with the mouse)
4. Hit the key 'F' or 'P' once
5. Try to rotate the shape again -> will not work as it is frozen.

Not possible to load ScalarMeshFields from the UI

ScalismoUI supports the visualization of ScalarMeshFields. However, it is not possible to load them from the GUI. In the current version, loading a ScalarMeshFields simply loads them as a mesh.

A new option should be added (preferably in the context menu of the group), which makes it possible to load ScalarMeshFields.

Impossible to change color of TetrahedralMesh3D

Visualization is not updated when picking a different color for a TetrahedralMesh3D.

Sidebar color pallet error when setting mesh to black

It is currently not possible to set the color value of a mesh/landmark/other to black within scalismo-ui. When doing so in code, this removes the option of further on changing the color value as shown in the attachment:

What would be the best way to solve this problem?
An additional color intensity bar?
Or, setting the color pallet to the standard colors even if black is chosen?

Missing change of background color

It would be a handy feature to change the background colour of the panels. Especially white background for use when making screenshots for a paper or presentation.
The implementation could be done in different ways:

• A colour palette window from the menu panel (in a pop-up window like the scaling window)
• A colour palette in the Scene view
• Button in the scene view to change the background colour between white and black.

Non-visualizing implementation of ScalismoUI for debugging

A common use case of Scalismo ui is visualization of an algorithm. Such a visualization is extremely useful for debugging, but is disturbing when the algorithm is run as part of a larger pipeline or in batch mode. The current solutions to this problem are 1) to comment out all the visualization code, or 2) to write a custom wrapper around Scalismo-ui, which allows to disable the visualization. Both options are cumbersome and may discourage the user from adding visualizations.

A possible solution for this problem could be to provide two implementations of the SimpleAPI, one that does the actual visualization, and one that does not show anything. This would make it possible that the user simple changes the underlying implementation to switch the visualization on or off.

Bundle example snippets to create visualizable objects

Contrary to the "core" scalismo project, this project currently does not have any unit tests. The reason is that it's notoriously hard to write such tests for User Interfaces, as they almost entirely depend on user interaction.

However, in order to properly test how the UI presents scalismo objects, and their visualization properties, an easy way to set them up in the first place is needed.

I therefore propose that this project should contain "somewhere" code which produces usable base objects, for each of the supported renderable objects. That can really be absolutely minimal "mock" objects -- for instance a TriangleMesh that is a pyramid, or landmarks/pointclouds consisting of just 3 points, etc. The idea is simply that there would be some code somewhere which
a) can be used quickly and easily when needed, and
b) is concise and simple enough to facilitate understanding for newcomers.

As to where "somewhere" should be, I'm not entirely sure. The easiest way would be to include it in src/main, but that feels kind of wrong, because the stuff would end up in the distribution. src/test seems to make sense, but I fear that most people associate it with build-time tests only. Maybe src/example? ... But anyway, that's a detail for further discussion.

Here's a very basic sketch, to give a more concrete idea of what I mean:

object Examples {

  // example of points which form a "distorted" pyramid (i.e., a tetrahedron)
  lazy val PointCloudExample: IndexedSeq[Point3D] = {
    List((0,0,0), (80,10,20), (30,50,25), (40,20,70)).map{
      case (x,y,z) => Point3D(x,y,z)
    }.toIndexedSeq
  }

  // triangle mesh formed from the point cloud example
  lazy val TriangleMeshExample: TriangleMesh3D = {
    val triangleCells: List[TriangleCell] = List((0,1,2),(0,1,3),(0,2,3),(1,2,3))
      .map{p => TriangleCell(PointId(p._1), PointId(p._2), PointId(p._3))}
    TriangleMesh3D(PointCloudExample, TriangleList(triangleCells.toIndexedSeq))
  }

  // trivial scalar mesh field formed from the triangle mesh example
  lazy val ScalarMeshFieldExample: ScalarMeshField[Int] = {
    // scalar values of points assigned from the index of the point in the mesh,
    // scaled by an arbitrary factor.
    val scalars = TriangleMeshExample.pointSet.pointSequence.zipWithIndex.map(_._2 * 42)
    ScalarMeshField(TriangleMeshExample, scalars)
  }

  // example for creating a UI and adding example objects, using the API.
  def main(args: Array[String]): Unit = {
    val ui = ScalismoUI()
    val group = ui.createGroup("Group Example")
    ui.show(group, PointCloudExample, "Point Cloud Example")
    //ui.show(group, TriangleMeshExample, "TriangleMesh Example")
    ui.show(group, ScalarMeshFieldExample, "ScalarMeshField Example")
  }
}

As said, this is just a first draft, incomplete and probably not organized the way it should be. But IMO it's concise, reusable (so it can be leveraged in tests), and understandable (so it could also double as a simple hands-on example to get started).

What do you guys think about this matter?

Saving Images from UI changes type

Observation: A DiscreteScalarImage[_3D, Short] loaded into the UI (either trough the UI, or added with 'ui.show') gets transformed into a DiscreteScalarImage[_3D, Float] when saving it from the UI.

Expected behaviour: Scalar type should not change when saving the image.

Error opening model2009-publicmm1-bfm.h5

Hello,

I've cloned latest version and did sbt run
then tried to open official model2009-publicmm1-bfm.h5 model file which resulted in following error:
"Unable to load file: Expected H5ScalarDS when reading attribute"