moeyensj / difi

Did I Find It?

difi

Did I Find It?

About

difi is a simple package that takes pre-formatted linkage information from software such as MOPS, pytrax, or THOR and analyzes which objects have been found given a set of known labels (or truths). A key performance criteria is that difi needs to be fast by avoiding Python for loops and instead uses clever pandas.DataFrame manipulation.

Installation

Released Versions

Anaconda

difi can be downloaded directly from anaconda:
conda install -c moeyensj difi

Or, if preferred, installed into its own environment via:
conda create -n difi_py310 -c moeyensj difi python=3.10

Pip

difi is also available from the Python package index:
pip install difi

Docker

A Docker container with the latest version of the code can be pulled using:
docker pull moeyensj/difi:latest

To run the container:
docker run -it moeyensj/difi:latest

The difi code is installed the /projects directory, and is by default also installed in the container's Python installation.

Latest From Source

Anaconda

Clone this repository using either ssh or https. Once cloned and downloaded, cd into the repository.

To install difi in its own conda environment please do the following:
conda create -n difi_py310 -c defaults -c conda-forge --file requirements.txt python=3.10

Or, to install difi in a pre-existing conda environment called difi_py310:
conda activate difi_py310
conda install -c defaults -c conda-forge --file requirements.txt

Pip

Or, to install difi software using pip:
pip install .

Or, if you would like to make an editable install then:
pip install -e .[tests]

You should now be able to start Python and import difi.

Docker Compose

After cloning this repository, you can build a docker image that will allow you to develop the source code:

docker compose build difi

To run the docker container interatively with a terminal:

docker compose run -it difi

Developing

If you would like to contribute to difi, please make sure to initialize pre-commit. Pre-commit will automatically lint and format the source code after any changes have been staged for a commit. To load the appropriate hooks please run:

pre-commit install

Example and Tutorial

The example below can be found in greater detail in this Jupyter Notebook.

difi's Issues

Make difi take into account 15 day detection window

difi's findability criterion currently doesn't require observations to occur within a 15 day window. We should make sure it does. (Made this with @mjuric)

E.g. currently difi would say that one of my objects that has 2 observations on each of nights 266, 287, 288 and 312 is findable but it isn't really because those don't fit in 15 days.

Add Docker

Add support for Docker containers, update README with appropriate installation instructions.

Make difi callable from the command-line

Add examples for mops, pytrax and thor

Add false positive and unknown truths handling

Make difi support being run via multiprocessing

difi currently will run a multiprocessing pool if the user wants to, however, if others want to use difi in a pipeline that itself is using multiple processes then the current shared_memory implementation will break.

Fix truth dtype mismatch

If the linkageMembers truth column has a different dtype than the observations truth column, pandas joins will not work as intended. This should be fixed by insuring all dtypes are the same and warning the user if not. A keyword can be added to convert the truth column to the "object" dtype.

Use quivr instead of pandas

Quivr: https://www.github.com/spenczar/quivr

Add cuDF acceleration and test performance gain

Add population statistics

For use on minor planet linking it would be useful to have a set of user-defined object level population statistics.

• How many NEOs are findable and were linked, etc...
• How many KBOs...

Add config.py

difi should support a configuration file / class with defaults defining metric parameters and column mapping dictionaries.

Add generic linkage plotting code

Linkage plotting code: given some set of linkage_ids, plot them!

Add LSST eups table

Add LSST eups build table for compatibility with the LSST stack

Add findability metrics

Add option for the user to define and pass a findability metric to analyzeObservations.

Package code for conda

Warn user when not all of the unique truths appear in the given truth classes

When the user gives a column name or dictionary of the classes to which every observation belongs to, there should be a warning thrown if not all of the unique truths in the data set appear in the given classes.

Add versioning with setuptools_scm

Add "should I find it" code

Add analysis code for observations that given a set of algorithmic constrains (such as: the cadence between observations, maximum and minimum linkage ranges, etc) will calculate if known objects within that set of observations should be findable.

Add package build and publishing to PyPi and conda

• Add build worflow
• Add build + packaging worklow for PyPi and conda
• Add conda recipe

Add summary statistics

Add statistics on the number of objects found, the number of objects findable, the number of linkages that are contaminated, etc...