the-ludwig / panama Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 1.0 20.57 MB

A python toolkit for CORSIKA7

Home Page: https://panama.readthedocs.io/en/latest/

License: MIT License

Python 87.49% C 11.81% Shell 0.70%

corsika python

panama's People

Contributors

Stargazers

Watchers

Forkers

jean1995

panama's Issues

Road to 1.0

I am working on the very last issues and I think after updating the documentation, this package is ready for a 1.0 release.

I have a lot of ideas for this package which might improve and extend it, but I think the current scope is very reasonable.
For a 2.0 release some features may be:

panama install 77420 asks for the corsika password and downloads and installs/compiles the given corsika version to a given location (this would be a modularization and extention of the script in the admin folder. But may be we don't want to steal coconut the show)
yaml steering: instead of steering cards, write one steering card where everything is replaced by a template, which is then filled in by values from a yaml file. optionally override this on the command line.

@Jean1995 as you and I are currently the only ones using this, I guess (maybe @pgutjahr?): Are there any more features you might want to see for a 1.0?

use corsikaio/noparse

@maxnoe created the branch corsikaio/noparse in pycorsikaio for me.
Implementing that should dramatically increase loading speeds.

stop copying the whole run folder

It should be possible to "trick" CORSIKA7. Symlinking most files in the run directory should be enough. For example the IceCube runner does this.
This would dramaticly increase startup time. Most of it is wasted to copying CORSIKA7

Fix numpy typing

I use a very bad way of giving type hints for numpy arrays (writing : np.ndarray). I should switch to the methods explained here: https://numpy.org/devdocs/reference/typing.html

extrem slow down of corsika in the case of low simulated energies

If mostly low energies are simulated, the io becomes to heavy for the package and it can't read corsikas stdout fast enough. This makes it basically come to a halt.

more tests

obvious.

Installaton not possible on arm architecture

I am unable to install PANAMA on my M1 MacOS, it fails with the error

  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [23 lines of output]
      Error in sitecustomize; set PYTHONVERBOSE for traceback:
      AssertionError:
      cpuinfo failed, assuming no CPU features: 'flags'
      * Using Python 3.11.1 (main, Dec 23 2022, 09:28:24) [Clang 14.0.0 (clang-1400.0.29.202)]
      Traceback (most recent call last):
        File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/homebrew/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/opt/homebrew/Cellar/[email protected]/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/homebrew/Cellar/[email protected]/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/opt/homebrew/Cellar/[email protected]/3.11.1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 265, in <module>
        File "<string>", line 132, in get_blosc2_directories
      NotADirectoryError: Install directory for blosc2 not found in /opt/homebrew/Cellar/[email protected]/3.11.1/Frameworks/Python.framework/Versions/3.11
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Might be related to this issue

switch to pdm

Upon further research and experience, I no longer think poetry is the build system to use for this project. pdm seems far better suited. Migration should not take too long.

Unneeded Print

PANAMA/panama/weights.py

Line 52 in 6fc0da7

print(pd.concat(weights))

Move panama/fluxes to own package

I implemented numpy/pandas compatible versions of multiple cr-flux models for this package.
This is not really what this package is designed to do (PANdas And Multicore utils for corsikA7) and could be useful also outside this package.

fix test of created files

In tests/files/compare there are the files I produced by running the test_run script on my local machines. I provide them in the hope, that we can corss-check if the corsika runner produces the same files on different machines, when using the same seed.
This, sadly, does not seem to be the case

`read_DAT` broken for newest pandas release

I'm trying to read files with the read_DAT option, my command is

run_header, _, particles_C7 = pn.read_DAT(glob=f"{C7_PATH}/DAT*[!.long]", mother_columns=True, additional_columns=True, max_events=MAX_EVENTS_C7)

If I use the most recent version of pandas (2.1.1), I get an error:

  File "/home/xxxxx/xxxxx/xxxxx/create_history_plots.py", line 36, in <module>
    run_header, _, particles_C7 = pn.read_DAT(glob=f"{C7_PATH}/DAT*[!.long]", mother_columns=True, additional_columns=True, max_events=MAX_EVENTS_C7)
  File "/usr/local/lib/python3.10/dist-packages/panama/read.py", line 306, in read_DAT
    add_mother_columns(df_particles, pdgids)
  File "/usr/local/lib/python3.10/dist-packages/panama/read.py", line 360, in add_mother_columns
    df_particles.loc[~df_particles["has_mother"].array, "mother_hadr_gen"] = pd.NA
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 881, in __setitem__
    indexer = self._get_setitem_indexer(key)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 758, in _get_setitem_indexer
    return self._convert_tuple(key)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 963, in _convert_tuple
    keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)]
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 963, in <listcomp>
    keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)]
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 1484, in _convert_to_indexer
    return self._get_listlike_indexer(key, axis)[1]
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py", line 1520, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py", line 6114, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py", line 6175, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index([ True,  True,  True,  True,  True,  True,  True,  True,  True,  True,\n       ...\n       False,  True,  True,  True, False,  True,  True,  True, False,  True],\n      dtype='bool', length=1119605)] are in the [index]"

However, this error vanished when I used pandas 2.0.3 before, so something in pandas must have changed which broke the script.

fix test protons_only

I test the weighting and corsika reading via the following mechanisms:

fit the weighted primary spectrum -> spectral index should be around 2.7
fit the weighted muon spectrum of conventional muons -> spectral index should be around 3.7
fit the weighted muon spectrum of prompt muons -> spectral index should be around 2.7

I do this for weighting with the HillasGaisserH3a model, which works and I do this with the Thunman model. This does not work, since thunman only has protons as primaries, and then we have a really low statistic. This should be fixed

Start with different EVTNR

I believe at the moment, the first event always starts with the event number 1

It would be nice if one could optionally specific a different start number, e.g., if one wants to organize the events in a specific way :)

Save STDOUT

Right now, the stdout of the CORSIKA files is not saved, this should be added

hdf as optional dependency

Saving as hdf is only used at one point of the code, so we can make this an optional dependency

HDF5 installation not found Mac M1 Pro Ventura

The installation using pip install corsika-panama fails on a Mac M1 Pro with Ventura 13.6.1 because the local installation of HDF5 is not found.

This can be fixed by setting the correct installation path to
export HDF5_DIR=/opt/homebrew/opt/hdf5

which of course requires the installation of HDF5 using brew install hdf5.

maximal job mode

Currently, panama run --jobs N starts N jobs per primary particle. So if we have 5 components, we potentially have 5*N particles.
On a multi-user system we want to be able to limit the number of total jobs we have. So we need to implement a --max-job option to run the components after each other, not all at the same time.
While at it, a class-based refactoring of run.py is needed. See branch max-threads

Traceback (most recent call last):
  File "/usr/local/bin/panama", line 5, in <module>
    from panama.cli import cli
  File "/usr/local/lib/python3.10/dist-packages/panama/__init__.py", line 19, in <module>
    __version__ = get_version()
  File "/usr/local/lib/python3.10/dist-packages/panama/__init__.py", line 14, in get_version
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/dist-packages/pyproject.toml'

for Linux and

Traceback (most recent call last):
  File "/opt/homebrew/bin/panama", line 5, in <module>
    from panama.cli import cli
  File "/opt/homebrew/lib/python3.11/site-packages/panama/__init__.py", line 19, in <module>
    __version__ = get_version()
                  ^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/panama/__init__.py", line 14, in get_version
    with open(path) as f:
         ^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/opt/homebrew/lib/python3.11/site-packages/pyproject.toml'

for MacOS.

Update documentation

The documentation is heavily out of date. Especially the examples from the readme need to be updated

Add version to module

This bugged me for a while: add a version variable to the module

the-ludwig / panama Goto Github PK

panama's People

Contributors

Stargazers

Watchers

Forkers

panama's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs