The installation instructions should mention two facts:

1. It's now possible to install litebird_sim using PIP, as the package has been registered on PyPI.
2. Unless there is the need to update the list of dependencies, developers can install and use litebird_sim without the need to install poetry.

In file scanning.py there is a wrong usage of the numpy.arctan2 function, as x and y coordinates are reversed.

The tutorial is not tested in our CI builds, because of the well-known problem of the lack of a «public IMO» to use in these test cases. This led to some leftovers of the old API still present in the documentation (e.g., Instrument instead of InstrumentInfo).

Apparently, when copying the path to a data file included in a release, the name of the data file is included in the path. This is not fully compliant with the IMO specifications, which state that if a release tag is included, the data file part should be dropped because it's redundant.

We're surely going to incorporate full-fledged solutions, like libmadam or TOAST , but as a start we should implement a simple binning algorithm, to test the interface with Simulation objects.

I'm having problems in the installation with the intel compiler 2018. The issue seems to be with ducc. Probably worth opening an issue also there.
gcc 7.3.0 goes smooth, instead
Here the log

• Installing ducc0 (0.4.0 d86009e)

[EnvCommandError]
Command ['/galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/bin/pip', 'install', '--no-deps', '-U', '-e', '/galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/src/ducc0'] errored with the following return code 1, and output:
Obtaining file:///galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/src/ducc0
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Installing collected packages: ducc0
Running setup.py develop for ducc0
Complete output from command /galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/bin/python -c "import setuptools, tokenize;file='/galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/src/ducc0/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" develop --no-deps:
running develop
running egg_info
writing ducc0.egg-info/PKG-INFO
writing dependency_links to ducc0.egg-info/dependency_links.txt
writing requirements to ducc0.egg-info/requires.txt
writing top-level names to ducc0.egg-info/top_level.txt
reading manifest file 'ducc0.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching '.'
writing manifest file 'ducc0.egg-info/SOURCES.txt'
running build_ext
building 'ducc0' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/python
icc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPKGNAME=ducc0 -DPKGVERSION="0.4.0" -I. -I./src/ -I/tmp/pip-build-env-w12ypczt/overlay/lib/python3.7/site-packages/pybind11/include -I/tmp/pip-build-env-w12ypczt/overlay/lib/python3.7/site-packages/pybind11/include -I/galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/include -I/cineca/prod/opt/tools/anaconda/2019.07/python--3.6.4/include/python3.7m -c python/ducc.cc -o build/temp.linux-x86_64-3.7/python/ducc.o -std=c++17 -march=native -ffast-math -O3 -Wfatal-errors -Wfloat-conversion -W -Wall -Wstrict-aliasing=2 -Wwrite-strings -Wredundant-decls -Woverloaded-virtual -Wcast-qual -Wcast-align -Wpointer-arith
icc: command line warning #10006: ignoring unknown option '-Wfloat-conversion'
icc: command line warning #10006: ignoring unknown option '-Wredundant-decls'
icc: command line warning #10006: ignoring unknown option '-Wcast-align'
In file included from python/ducc.cc(3):
./src/ducc0/infra/threading.cc(343): error: identifier "rethrow_exception" is undefined
rethrow_exception(ex);
^

compilation aborted for python/ducc.cc (code 4)
error: command 'icc' failed with exit status 4

----------------------------------------

Command "/galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/bin/python -c "import setuptools, tokenize;file='/galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/src/ducc0/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" develop --no-deps" failed with error code 1 in /galileo/home/userexternal/lpagano0/.cache/pypoetry/virtualenvs/litebird-sim-k6pebD1r-py3.7/src/ducc0/
You are using pip version 19.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

File pyproject.toml should be updated with the list of people that want to be mentioned in the PyPI (Python Package Index) entry for the LiteBIRD Simulation Framework. Currently there are only two entries:

authors = [
	"The LiteBIRD Simulation Team",
	"Maurizio Tomasi <[email protected]>",
]

but the plan is to make the generic item The LiteBIRD Simulation Team disappear and have the proper list of authors listed here. Whoever wants to be listed, please modify the file/open a PR and add your name and email to the file. You can make this change within the browser using this link: https://github.com/litebird/litebird_sim/blob/master/pyproject.toml . Click on the pencil icon on the topright corner of the file to edit it:

Please keep alphabetical order!

N.B. It is not necessary that you contributed to this very repository to be listed as an author, nor that you are part of the Simulation Team! If you contributed in some way to the overall effort, you are entitled to opt in.

Currently, litebird_sim automatically includes the output of git diff in the reports produced by Simulation.flush. This helps to ensure traceability of outputs (which is mandated by the Specification Document agreed by the LiteBIRD collaboration).

However, in the last months it has happened more than once that some people (usually with poor knowledge of how git works) have had serious problems in running simulations because they were keeping all the results of their simulations in the same git repository hosting the source code. In some cases, the git diff output was very long because it included diffs of SVG plots, and this slowed Simulation.flush to one minute and more.

There should be an option to make the inclusion of git diff optional.

Since litebird_sim will most likely be the heaviest user of this new library in the near future, it makes a lot of sense to have at least a fork of the code withing the litebird Github organization, so I plan to add the code soon.
Technical question: does it make more sense to provide it as a standalone repository within https://github.com/litebird/, or should it be a subtree of https://github.com/litebird/litebird_sim/?
@ziotom78 what do you think?

Hi, shall we agree on how to simulate the CMB monopole and dipole? I can then add them in the map-based-sim module.

I'm having an issue getting 'poetry install' to work properly. Most of the dependencies build successfully but for these two packages I get errors:
backports-datetime-fromisoformat
ducc0
I do not think it is the same problem as in this issue: #34 , because the output of 'conda env list' is:

conda environments:

base * /astro/local/anaconda
py38 /astro/local/anaconda/envs/py38
py39 /astro/local/anaconda/envs/py39

It looks to me like poetry is trying to use options for the wrong versions of compilers, but I'm not 100% sure. I have attached a full error log of the two packages that fail.
poetry error log

Hi all, I believe that the Detector class adds more complications than benefits. Yes, I know that this topic was already discussed in #42 , but let me try again or at least summariz that discussion in a dedicated issue (even I had a hard time looking up the #42 PR)

Benefits are (@ziotom78 please feel free to add/amend in case a missed or misunderstood anything)

1. detectors are an important concept, they deserve a class
2. the class can provide custom __repr__ and __str__
3. the class can enforce the attribute names, which helps the coherence across the framework
4. since the Detector stores all the possible piece of information about the detectors, the user always have them at his/her disposal (for example when adding a new stage to an existing pipeline)
5. In spite of 4., at construction time we can specify only the parameters we need: all the rest is defaulted to None

Compared to a simple dictionary, I found the following complications

1. I think that the benefits of 4 are largely outweighed by the fact that every user will be interested in a small subset of quantities, probably different for every user. Still, if any user at any time will need a detector quantity, we will have to add it to the `Detector` attributes. The result is that *we force all the users to carry a lot of quantities even if they only care about a few*.
2. I think that *the dictionary is a more natural and transparent data structure*, which the user find more comfortable with. The following points are examples of this general idea
3. Sometimes you want to iterate over the quantities: it is more natural if the detector is a dictionary
4. Getting all the available detector quantities is much more natural ( det.keys() ) if det is a dictionary
5. it happens that the name of the quantity you want to access is stored in a variable, I prefer det[variable] over getattr(det, variable)

I've tried to install the library from the imointerface branch with poetry install but I get the following error:

(base) niki@mowgli:~/Library/LiteBIRD/litebird_sim$ poetry install --extras=jupyter --extras=docs
Installing dependencies from lock file


Package operations: 2 installs, 0 updates, 0 removals

  - Installing backports-datetime-fromisoformat (1.0.0)

[EnvCommandError]
Command ['/Users/niki/Library/Caches/pypoetry/virtualenvs/litebird-sim-O-5o24kS-py3.6/bin/pip', 'install', '--no-deps', 'backports-datetime-fromisoformat==1.0.0'] errored with the following return code 1, and output:
Collecting backports-datetime-fromisoformat==1.0.0
  Using cached https://files.pythonhosted.org/packages/20/15/4bc39da78d00da480ff627398ad25770be1f0c3cf40ea9bc5e9030b441fb/backports-datetime-fromisoformat-1.0.0.tar.gz
Installing collected packages: backports-datetime-fromisoformat
  Running setup.py install for backports-datetime-fromisoformat: started
    Running setup.py install for backports-datetime-fromisoformat: finished with status 'error'
    Complete output from command /Users/niki/Library/Caches/pypoetry/virtualenvs/litebird-sim-O-5o24kS-py3.6/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/dt/p1fypct95wzccmzqv68w6f5c0000gn/T/pip-build-bqi2p8lv/backports-datetime-fromisoformat/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/dt/p1fypct95wzccmzqv68w6f5c0000gn/T/pip-qf4kmji1-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/niki/Library/Caches/pypoetry/virtualenvs/litebird-sim-O-5o24kS-py3.6/include/site/python3.6/backports-datetime-fromisoformat:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.7-x86_64-3.6
    creating build/lib.macosx-10.7-x86_64-3.6/backports
    copying backports/__init__.py -> build/lib.macosx-10.7-x86_64-3.6/backports
    creating build/lib.macosx-10.7-x86_64-3.6/backports/datetime_fromisoformat
    copying backports/datetime_fromisoformat/__init__.py -> build/lib.macosx-10.7-x86_64-3.6/backports/datetime_fromisoformat
    running build_ext
    building 'backports._datetime_fromisoformat' extension
    creating build/temp.macosx-10.7-x86_64-3.6
    creating build/temp.macosx-10.7-x86_64-3.6/backports
    creating build/temp.macosx-10.7-x86_64-3.6/backports/datetime_fromisoformat
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/niki/Library/miniconda3/include -arch x86_64 -I/Users/niki/Library/miniconda3/include -arch x86_64 -I/Users/niki/Library/libsharp-master/auto/include -I/Users/niki/Library/Healpix_3.40/include -I/Users/niki/Library/llvm/build/include -I/Users/niki/Library/Caches/pypoetry/virtualenvs/litebird-sim-O-5o24kS-py3.6/include -I/Users/niki/Library/miniconda3/include/python3.6m -c backports/datetime_fromisoformat/module.c -o build/temp.macosx-10.7-x86_64-3.6/backports/datetime_fromisoformat/module.o
    In file included from /usr/local/Cellar/[email protected]/4.9.4_1/lib/gcc/4.9/gcc/x86_64-apple-darwin17.3.0/4.9.4/include-fixed/syslimits.h:7:0,
                     from /usr/local/Cellar/[email protected]/4.9.4_1/lib/gcc/4.9/gcc/x86_64-apple-darwin17.3.0/4.9.4/include-fixed/limits.h:34,
                     from /Users/niki/Library/miniconda3/include/python3.6m/Python.h:11,
                     from backports/datetime_fromisoformat/module.c:1:
    /usr/local/Cellar/[email protected]/4.9.4_1/lib/gcc/4.9/gcc/x86_64-apple-darwin17.3.0/4.9.4/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
     #include_next <limits.h>  /* recurse down to the real one */
                                                                 ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/Users/niki/Library/Caches/pypoetry/virtualenvs/litebird-sim-O-5o24kS-py3.6/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/dt/p1fypct95wzccmzqv68w6f5c0000gn/T/pip-build-bqi2p8lv/backports-datetime-fromisoformat/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/dt/p1fypct95wzccmzqv68w6f5c0000gn/T/pip-qf4kmji1-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/niki/Library/Caches/pypoetry/virtualenvs/litebird-sim-O-5o24kS-py3.6/include/site/python3.6/backports-datetime-fromisoformat" failed with error code 1 in /private/var/folders/dt/p1fypct95wzccmzqv68w6f5c0000gn/T/pip-build-bqi2p8lv/backports-datetime-fromisoformat/
You are using pip version 9.0.3, however version 20.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

everything works fine on the master

make_bin_map requires obs.pixel and obs.psi which are note defined properly.

Args:
    obss (list of :class:`Observations`): observations to be mapped. They
        are required to have the following attributes as arrays of identical
        shapes
        * `tod`: the time-ordered data to be mapped
        * `pixel`: the index of the pixel observed for each tod sample in a
          HEALpix map at nside `nside`
        * `psi`: the polarization angle (in radians) for each tod sample

Actually obs.pixel contains the detector id in the wafer

        det = lbs.DetectorInfo.from_imo(url="/releases/v1.0/satellite/MFT/M1-140/M03_030_QA_140T/detector_info",imo=imo)
        obs, = sim.create_observations(detectors=[det])
        print(obs.pixel)

gives 30

should obs.get_pointings() fill them?

Anyway, if I allocate and fill them with the output of obs.get_pointings, still I get an error when I run lbs.make_bin_map

TypingError Traceback (most recent call last)
in
----> 1 m=lbs.make_bin_map([obs], nsideout)

~/Documents/Universita/litebird/simteam/codes/hwp/litebird_sim/litebird_sim/mapping.py in make_bin_map(obss, nside)
70
71 for obs in obss:
---> 72 _accumulate_map_and_info(obs.tod, obs.pixel, obs.psi, info)
73
74 if all([obs.comm is None for obs in obss]) or not mpi.MPI_ENABLED:

~/Library/Caches/pypoetry/virtualenvs/litebird-sim-iPzjGDqs-py3.7/lib/python3.7/site-packages/numba/core/dispatcher.py in _compile_for_args(self, *args, **kws)
413 e.patch_message(msg)
414
--> 415 error_rewrite(e, 'typing')
416 except errors.UnsupportedError as e:
417 # Something unsupported is present in the user code, add help info

~/Library/Caches/pypoetry/virtualenvs/litebird-sim-iPzjGDqs-py3.7/lib/python3.7/site-packages/numba/core/dispatcher.py in error_rewrite(e, issue_type)
356 raise e
357 else:
--> 358 reraise(type(e), e, None)
359
360 argtypes = []

~/Library/Caches/pypoetry/virtualenvs/litebird-sim-iPzjGDqs-py3.7/lib/python3.7/site-packages/numba/core/utils.py in reraise(tp, value, tb)
78 value = tp()
79 if value.traceback is not tb:
---> 80 raise value.with_traceback(tb)
81 raise value
82

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function() found for signature:

getitem(array(float64, 3d, C), float64)

There are 22 candidate implementations:

• Of which 20 did not match due to:
  Overload of function 'getitem': File: : Line N/A.
  With argument(s): '(array(float64, 3d, C), float64)':
  No match.
• Of which 2 did not match due to:
  Overload in function 'GetItemBuffer.generic': File: numba/core/typing/arraydecl.py: Line 162.
  With argument(s): '(array(float64, 3d, C), float64)':
  Rejected as the implementation raised a specific error:
  TypeError: unsupported array index type float64 in [float64]
  raised from /Users/luca/Library/Caches/pypoetry/virtualenvs/litebird-sim-iPzjGDqs-py3.7/lib/python3.7/site-packages/numba/core/typing/arraydecl.py:69

During: typing of intrinsic-call at /Users/luca/Documents/Universita/litebird/simteam/codes/hwp/litebird_sim/litebird_sim/mapping.py (22)

File "../codes/hwp/litebird_sim/litebird_sim/mapping.py", line 22:
def _accumulate_map_and_info(tod, pix, psi, info):

sin = np.sin(2 * a)
info_pix = info[p]
^

Latest news: https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

TLDR: Travis cannot be used any longer for public repositories. We're receiving 1000 "free trial" minutes, and then we either pay for Travis, or we move to other solutions. (Also, they are going to charge Mac OS X support a lot more on, but this is reasonable, as they're forced to use costly hardware to run builds for that platform.)

I see that in the last few months many organisations have moved from Travis to GitHub Actions; other options are Circle CI, Drone CI. I have no strong preference, as I never used anything else than Travis. Do you have any good/bad experience to share about other CI systems?

The Simulation class should provide a seamless way to interface with LiteBIRD's IMO. This should enable users to retrieve parameters for simulation modules using a REST API, with the possibility to download contents locally and caching them before running the actual computation.

While the software stack for the IMO is being designed and built, we might use as reference InstrumentDB, which implements a similar way of structuring things (entities/quantities/data files).

When a flat-file IMO file is read from a JSON/YAML file, the "parent" field of entities is always set to None.

What will be the preferred way to pass rotation/orientation information (e.g. satellite orientation, detector pointings) to/from subpackages?
Quaternions appear like a natural choice; their only disadvantage is that they need a bit more memory than strictly necessary (4 floating point numbers instead of 3). If that is an issue, then any set of Euler angles could be used as well, but in that case I suggest to document the exact conversion between these angles and quaternions by a piece of pseudocode, so that no misunderstandings are possible.

Currently we're testing our codes against Python 3.6 and Python 3.7. As the default version installed by Anaconda is now 3.8, I think it would be worth to add it to the Travis build.

Not sure what to do with 3.7. If we keep 3.6 and add 3.8, is it worth to test an intermediate version? I fear the increase in the time required to complete a CI script.

(My experience with the Julia package ecosystem is that they always test the minimum compatible version and the most recent nominal version, and they always skip every version in-between.)

Sould the Instrument class have a from_imo function? Something like the one we have for the Detector class

Provide a C++ library (with Python interface), which generates a total convolution data cube from a set of sky and beam a_lm and computes interpolated values for a given list of detector pointings.

Development of this code happens in the directory interpol_ng of https://gitlab.mpcdf.mpg.de/mtr/cxxbase.

Algorithmic details:

• the code uses libsharp SHTs to compute the data cube, which is more efficient than the libconviqt approach
• shared-memory parallelization is provided via standard C++ threads.
• for interpolation, the algorithm and kernel described in https://arxiv.org/abs/1808.06736 are used. This allows very efficient interpolation with user-adjustable accuracy.
• no MPI support is planned at the moment, because I do not expect this to scale very well for LiteBIRD-typical resolutions. For massively parallel computation, I recommend processing different detectors simultaneously on different MPI tasks.

External dependencies:

• pybind11 (https://github.com/pybind/pybind11) for the Python glue code

Mapmaking requires information about the detector pointing and polarization angle. Have we defined how this information can be retrieved? Sorry if I missed

Currently the README is outdated: it suggests people to use poetry to install the framework, but this should be done only by the developers of the framework itself. The documentation describes the correct procedure (using pip).

As discussed at the S2S meeting in April 2020, I think it would be important to prepare a set of guidelines on all the steps needed to import existing codes in this repo.

There exist a bunch of codes already developed by collaborators that we probably want to import here. Releasing a tutorial on this would also help in making clearer to the collaboration the scope of this team, which, as we said several times, is not to develop the full pipeline but just to have a common and coherent infrastructure.

One example is the simple pipeline that I developed to produce map based simulations, which is available here https://github.com/NicolettaK/LB_mbs. This is a private repo at the moment as there are infos about the instrument, but I can give you access if you want to have a look.

Any idea on how we should proceed on this?

c.f. #90, ducc0 uses too much memory for readthedocs, and is only called once.
Should we make it optional and put it somewhere in [tool.poetry.extras] in pyproject.toml?

The script litebird_sim/install_imo does not properly create the path ~/.config/litebird_imo before trying to save the file imo.toml.

This path is created if the user chooses to download the IMO from the GitHub repository, as in that case the code creates the path and saves the downloaded file right there; the error happens only if the user configures a local copy of the IMO.

The litebrid_sim github page currently displays only the documentation of the master branch.
This has several limitation

• If, for any reason, a user wants to use a previous version he does not have a documentation for that
• We can not share documentation for among developers about developing branches
• We can not preview how the documentation will render exactly until we merge with the master branch

Possible decision

1. We accept these constraints
2. We use readthedocs
3. We keep using gihub pages but point the website to the gh-pages branch instead of the docs folder of the master branch. Then each branch have to write its documentation to a folder named after the branch in gh-pages

A few of the PRs that have been merged have broken the tests ran by bin/run_tests.sh (the ones employing black and flake8). It would be advisable to fix these inconsistencies before releasing 0.2.0.

Hi there, I'm trying to use mpi with sim.create_observations. I cannot understand how to pass correctly the communicator. Setting it in the initialization of lbs.Simulation is not working. Am I supposed to initialize obs only with lbs.Observation for handling mpi runs? It should not be the case, right?
Another small thing, reading the documentation, lbs.Simulation should have the flag use_mpi, but it doesn't exist anymore.

I'm trying to implement parallelization in the Mbs class. I can't enable mpi4py though.
I've followed the instruction here: https://github.com/litebird/litebird_sim/blob/master/docs/source/mpi.rst and set the environment variable with export LITEBIRD_SIM_PI=1, however lbs.MPI_ENABLED is still False and I don't get any error (which I think I should get as if I do import mpi4py it doesn't work)

I'm trying to install litebird_sim locally, but get:

martin@debian:~/codes/litebird_sim$ poetry install --extras=jupyter --extras=docs
/home/martin/.poetry/lib/poetry/_vendor/py2.7/subprocess32.py:149: RuntimeWarning: The _posixsubprocess module is not being used. Child process reliability may suffer if your program uses threads.
  "program uses threads.", RuntimeWarning)
The currently activated Python version 2.7.18 is not supported by the project (^3.6).
Trying to find and use a compatible version. 
Using python3 (3.8.5)
Creating virtualenv litebird-sim-p-FMZbmZ-py3.8 in /home/martin/.cache/pypoetry/virtualenvs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'PosixPath' is not defined
Updating dependencies
Resolving dependencies... (4.4s)

[RuntimeError]
Unable to retrieve the package version for /tmp/pypoetry-git-ducc4GP15X

This used to work some time ago, and the Travis builds run as well, so I'm a bit at a loss what's going wrong here. Any ideas?

[Edit: add missing command line]

When a simulation spans very long time scales or uses many detectors, parallelization on via MPI is probably the best way to go. The Simulation object should provide some tools to ease the task of distributing computing tasks among MPI processes. This will require the merging of #6 .

The constructor of Observations calls

This layer should contain functions that can be called after the generation of a map or after a comp sep layer. We can start with a function that calls anafast, deconvolves internally for beam and pixel window function, and applies normalizations. The next upgrade might be done using pymaster, and writing a simple interface.

It would be extremely good (and timely) to take an explicit decision about the data layout.

Current structure

Every simulation owns a list of observations. Every observation owns a detector from which you can get a timestream array, a quaternion, a PSD array, a white noise level etc. Firstly, I would reduce the number of indirections from simulation.observations[3].detector.get_tod() to simulation.observations[3].tod. Anyway, even in the latter case, the paradigm is

sim.observations[3].tod  # 1D np.array
sim.observations[3].wn_level  # float
print(sim.observations[3])

The advantage is that the code reflects the logical hierarchy and organization of the data we are storing. On the practical side, it allows to easily plug in and remove detectors, print or dump to disk all the attributes of the detector. This model is well suited for random operations on random detectors.

Proposal

I propose the following alternative paradigm. The simulation directly owns an array of quaternions, a 2d array of TODs, a 2d array of PSDs, a 1d array of white noise levels etc. These arrays stack the data of the individual detectors.

sim.tod  # 2D np.ndarray: first index for detectors, second for time
sim.tod[3]  # same result as before
sim.wn_level[3]  # same result as before
sim.print_detector(3)  # same result as before

Pros

Operations that involve more than one detector are likely to be easier to write. This is sure if the operation involves all the detectors.
If these operations are done by compiled codes, the interface is much easier (pass one big array, instead of many small values/arrays)
All numpy functions integrate seamlessly with ndarrays
...many other technical/philosophical things that probably shouldn’t drive the decision...

Examples

Standard deviations of the TOD of each detector: sim.tod.std(1)
TOD of the standard deviation across the focal plane: sim.tod.std(0)
TOD of the mean signal across the focal plane: sim.tod.mean(0)
Add white noise to the TOD:
sim.tod += np.random.normal(size=sim.tod.shape) * sim.wn_level[:, np.newaxis]
Add cross-talks given by a crosstalk matrix C: sim.tod = C @ sim.tod

Note

If we think it is necessary, we could still implement our current API with a caveat: sim.observations[3].tod has to be a view of sim.tod[3] (same for other attributes). In practice, it means that sim.observations[3].tod = … is not allowed (you have to do sim.observations[3].tod[:] = …) and sim.observations[3].wn_level is a 1-element array.

You got to the end, you're a hero. One last effort: comment!

We should have a general discussion and, possibly, guidelines about how to interface with the existing libraries that we want to plug in the pipeline. In particular, we should decide when add them as a dependency and import them and when we move them inside litebird_sim continue the development here.

Here is a proposal

Import when

• it is a general purpose library developed independently from the simulation team

Integrate when

• only a small and stable piece of a complex library is needed
• the main case of use of the library is the LiteBIRD pipeline or LiteBIRD-related activities

I've spent some time in the last months trying to fix issue #25 in healpy, to support Windows. Alas, the more I dig, the more difficult the fix seems to be. I explored Julia's awesome BinaryBuilder as a way to automatically generate DLLs for libsharp and libhealpix, but even producing these binaries would require deep changes in the way Healpy's setup.py works. (In the current version, it downloads a patched version of the source code for these libraries and triggers a compilation using Autotools.)

I have discovered that recently AppVeyor added support for WSL, provided that Visual Studio 2019 is used. I would like to move our script appveyor.yml to use WSL and only support it. Being an Ubuntu-like system, I expect WSL to be easy to support.

If we manage to move to WSL, then we could add healpy as a dependency to litebird_sim. What do you think?

I followed the installation instructions from README.md and encountered a small problem:
While running bin/run_tests.sh the script complained about missing black and flake8, although they had been installed during the poetry install step. Installing these packages system-wide did help, but that's probably not the preferred solution.

Another point: I know it's fashionable these days, but I really feel bad about encouraging people to execute commands like
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
(with or without sudo). It's extremely bad practice and dangerous. Personally I will never run such a command, unless I'm on a throw-away virtual machine. Is there a reason to prefer this over, say, pip3 install --user poetry?

Hi guys, I've tried to install the package on my Mac but I got the following error

(py36n) niki@mowgli:~/Library/LiteBIRD/litebird_sim$ poetry install --extras=jupyter --extras=docs
Installing dependencies from lock file


Package operations: 20 installs, 77 updates, 0 removals

  - Updating decorator (4.3.0 /Users/niki/Library/miniconda3/envs/py36n/lib/python3.6/site-packages -> 4.4.2)
  - Updating ipython-genutils (0.2.0 /Users/niki/Library/miniconda3/envs/py36n/lib/python3.6/site-packages -> 0.2.0)

[EnvCommandError]
Command ['/Users/niki/Library/miniconda3/envs/py36n/bin/pip', 'uninstall', 'ipython-genutils', '-y'] errored with the following return code 1, and output:
Cannot uninstall 'ipython-genutils'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Here probably the easiest starting point is to implement an inverse wishart distribution likelihood for BB only. So, it takes a scalar and tensor input power spectra, rescales them internally to build the theoretical power spectrum, computes the likelihood, and provides posteriors as output. For a quick evaluation of the impact of some residual systematics, this is probably the best way. A user can quickly run a simulation, take the difference with the input map, run the power spectrum and compute bias on r.

Implement code which performs the adjoint operation of what's done in #12. While not necessary for time stream generation, it is an important ingredient in deconvolution map mapmaking.

I am proposing two breaking changes in the code that might improve the consistency of some parts.

Rename Instrument to InstrumentInfo

There is a class named DetectorInfo in the framework, which closely matches the quantity detector_info in the IMO, and PR #60 is going to include a class FreqChannelInfo, which matches the quantity channel_info. As the Instrument class (currently used by the scanning strategy simulator) is meant to match the newly created instrument_info IMO quantity, for consistency it would be advisable to rename Instrument to InstrumentInfo.

Of course, this introduces a breaking change in the code, which will require to bump the version number from 0.1.0 to 0.2.0.

Angles are always expressed in radians

The current implementation of the SpinningScanningStrategy class accepts angles in degrees in the __init__ constructor; these are immediately converted in radians for efficiency reasons (NumPy and Numba trigonometric functions work with radians, so it's better to measure every angle using this unit). However, this leads to a few weirdnesses:

import litebird_sim as lbs
import numpy as np

sstr = lbs.SpinningScanningStrategy(
    # Specify the angle in degrees and spare an explicit call to np.deg2rad
    spin_sun_angle_deg=10.0,
    ...
)

# Surprise! There is no "spin_sun_angle_deg" field in "sstr"!
print(sstr.spin_sun_angle_rad) # You get the angle in radians

# If I want to modify the angle, I must now use "deg2rad" explicitly
sstr.spin_sun_angle_rad += np.deg2rad(5.0)

When I implemented SpinningScanningStrategy, I chose to use degrees in the constructor because angles in the IMO are expressed in this unit for the sake of readability and to avoid roundoff errors (when expressed in degrees, most of them are integer numbers).

However, while implementing PR #60, I had a similar problem for the InstrumentInfo class (which contains the angles specifying the orientation of the focal plane). This time I chose an alternate and more straightforward approach, and I required to use radians even in the constructor. (The only place where degrees are used is just when reading an object from the IMO, because of course angles in the IMO are in degrees!) Therefore, here is the code to create a InstrumentInfo object:

import numpy as np

instr = lbs.InstrumentInfo(
    spin_boresight_angle_rad=np.deg2rad(10.0),
    ...
)

# No surprise here: you get what you gave
print(instr.spin_boresight_angle_rad)

# We have to use deg2rad, but it's ok, as it is consistent with the code above
instr.spin_boresight_angle_rad += np.deg2rad(5.0)

The more I use this latter implementation, the more I find it pleasant. It's true that you always must call np.deg2rad, but it has a few advantages:

1. The keywords used for __init__ are the same as the fields of the class, and this prevent surprises;
2. If you need to modify the angle later, you keep using the same units as in the constructor;
3. It's not a real PITA, as most of the time people create these objects using InstrumentInfo.from_imo, which performs the degree-to-radians conversion automatically (the IMO reports these angles in degrees).

Being a breaking change, this will require a version bump too, but we might take two birds with one stone and implement the two changes together for 0.2.0.

Basically, the proposal is to change the declaration of SpinningScanningStrategy.__init__ so that it always accepts only angles in radians. (Thus, keyword names will be changed from spin_sun_angle_deg to spin_sun_angle_rad, etc.)

I would like to hear your opinions!

to be compatible with healpy read_map
https://github.com/litebird/litebird_sim/blob/master/litebird_sim/healpix.py#L337

For a telescope with a half-wave-plate, the classic 4pi convolution output (which consists of a single measurement value for every detector pointing) is not sufficient; the code should rather provide three values corresponding to the unpolarized, perfectly co-polarized and perfectly cross-polarized beam, respectively.
I do not expect great difficulties when adding this to interpol_ng, but I will need help with the exact definition of the individual beam components.

Should we add hwp_rpm in the Instrument class? Surely it doesn't enter directly in the pointing definition but I think here is the right place where to input it, what do you think?

Hi,

it would be good to have instructions on how to use the litebird_sim poetry env as kernel on JupyterHub at NERSC. Anyone knows how to do that?

Some simulation modules may require input in some specific units different from the standard units used by the rest of the pipeline. This may create conflicts between modules.
As an example, to fully simulate the detector response, one may want the optical input (sky map) in units of power rather than the more common temperature or intensity (Jy/sr) units produced by pysm. See the following presentation for some details about detector simulations: TES Simulation.pdf.

Should we:

1. Allow all modules to process data in all possible units needed by the users? As discussed during the telecon, this may be not a very efficient way.
2. Provide a module/function that converts between units needed by the users? If we are going to follow this line we should get some feedback from the collaboration about the units we should consider. Then get conversion factors, input models, etc. and agree on who and how is in charge of producing them.
3. Just specify the units for each module and leave it to the user to convert according to his/her own needs?
4. Other options?

Comments are more than welcome.

Consider the following script:

import litebird_sim as lbs

sim = lbs.Simulation(
    base_path="./simulation_dir", start_time=1.0, duration_s=11.0
)
det = lbs.Detector("dummy", sampling_rate_hz=15)
obs_list = sim.create_observations(detectors=[det], num_of_obs_per_detector=5)

tod_duration_s = int(obs_list[-1].get_times()[-1] - obs_list[0].get_times()[0])
print(f"The duration of the simulation is {tod_duration_s} s")

This prints:

The duration of the simulation is 1982 s

which is wrong: the simulation should last 11 seconds, as specified in the call to lbs.Simulation.

This is not a terribly urgent question, but we will have to address it at some point.

I'm pretty sure that my C++ components can be adjusted without many problems, but performance may be degraded quite severely, since at least initially there will be no support for vector instructions.