j-dr / skyfactory

Set up and run large suites of mock sky simulations.

SkyFactory

Set up and run large suites of mock sky simulations.

Setup

Clone this repository: git clone https://github.com/j-dr/SkyFactory.git

To set up all the other pieces of code required to run the simulations, just run

sh setup.sh <email>

Running jobs

If running on NERSC, you should run module load python3. To setup a job, run the command python setup_sky.py chinchilla {num} cori-haswell. Then cd {SkyFactoryDir}/chinchilla-herd/Chinchilla-{num}.

First you will need to move the preprocessed input files off of HPSS. If you have never accessed HPSS before, you will first need to run the command hsi. This will open an interface to HPSS. You can exit from it immediately. Once you have done that, run cd unarchivepreprocess && module load esslurm && sbatch job.unarchivepreprocess.sh. This will begin a job transferring the nbody simulation from HPSS. You will receive an email when it starts and finishes.

Once the transfer has completed, navigate to cd {SkyFactoryDir}/chinchilla-herd/Chinchilla-{num} and run sbatch job.all.sh. This will submit the full job. You will again receive an email when the job starts and finishes. Once it has completed, notify Joe and once you have the okay, then run cd archive && sbatch job.archive.sh in the {SkyFactoryDir}/chinchilla-herd/Chinchilla-{num}/archive directory. This will move the simulation to HPSS.

Finding Outputs

1. Outputs for a given simulation go to the global output dir in the system config file, followed by the simulation name, followed by the simulation number like this {global output dir}/{sim name}-{sim number}.
2. Outputs for a given task always go to the global output dir for the sim from 1 followed by the task name in in all lower case (i.e., {global output dir}/{sim name}-{sim number}/calcrnn for task CalcRnn).

Notes on Adding New Tasks

1. Make a file for the task under templates/{taskname}.py
2. In the file from 1, make a child class of BaseTemplate which looks like

from .basetemplate import BaseTemplate

class MyTask(BaseTemplate):
    def write_config(self, data_output_path, box_size):
        """
        write the job configs
        
        write job config to self.sysparams['JobBase']
        any outputs go to data_output_path
        """

    def write_jobscript(self, data_output_path, box_size):
        """
        write the job submission script
        
        write job submission script to self.sysparams['JobBase']
        outputs got to data_output_path
        """

If you need to run the task only once, instead of for all N-body boxes, add the following to code above

    def __init__(self, simnum, system, cosmo):
        super(DensMap, self).__init__(simnum, system, cosmo, allboxes=True)

3. Make sure to import the class from 2 in templates/__init__.py
4. Make sure to write and put your job submission script in systems/{all systems}/MyTask.{Sched} where {Sched} is the corresponding parameter in the {system}.yaml file.
5. Make sure to add task to setup_sky.py script

skyfactory's Issues

edison scripts buggy

• need to have #!/bin/bash -l instead of #!/bin/bash
• a lot of them have cd {OPath} in them, but now edison scripts start in the submission dir, so no cd needed

add calclens

• fill in base info
• put in paths to input pixLC files
• put in paths to input galaxy catalogs

add in module/code to make healpix maps of density fields

• want to make the maps at a different Nside than in the pixLC files
• want to combine radial shells in pairs (so 50 Mpc/h thick)

add another task to setup global job

remove template cfg files in favor of def right in class

sep template job subclasses into sep files

Remove intermediate addgals files in addgals postprocessing.

Delete the z/pix cell files after they have been combined (or at least add option to do this).

add calclens-postprocess

• put scripts to do this in another repo
• add config and template files
• do cleanup of duplicate outputs

create methods in base class to extract job script names, standardize them too

accept command line file to override paths for the system configs

copy exec files and source into sim specific job

set base job output path per sim in init of base class

It should look like this

jobbase = os.path.join(self.sysparams['JobBase'],'{0}-{1}'.format(self.cosmopars['SimName'],self.simnum))

pixLC needs to do all 8 light cones

finish setup_sky script

change unarchive to pull out all each boxes for each light cone

need to symlink shells from each box to mock up a single box

remove rockstar exec from repo

Don't require pixLC to be installed. Need to point to absolute path.

move config scripts per box to a separate repo

make restartable