facebookresearch / hydra

Hydra is a framework for elegantly configuring complex applications

A framework for elegantly configuring complex applications.

Check the website for more information,
or click the thumbnail below for a one-minute video introduction to Hydra.

---

Releases

Stable

Hydra 1.3 is the stable version of Hydra.

• Documentation
• Installation : pip install hydra-core --upgrade

See the NEWS.md file for a summary of recent changes to Hydra.

License

Hydra is licensed under MIT License.

Hydra Ecosystem

Check out these third-party libraries that build on Hydra's functionality:

• hydra-zen: Pythonic utilities for working with Hydra. Dynamic config generation capabilities, enhanced config store features, a Python API for launching Hydra jobs, and more.
• lightning-hydra-template: user-friendly template combining Hydra with Pytorch-Lightning for ML experimentation.
• hydra-torch: configen-generated configuration classes enabling type-safe PyTorch configuration for Hydra apps.
• NVIDIA's DeepLearningExamples repository contains a Hydra Launcher plugin, the distributed_launcher, which makes use of the pytorch distributed.launch API.

Ask questions in Github Discussions or StackOverflow (Use the tag #fb-hydra or #omegaconf):

• Github Discussions
• StackOverflow
• Twitter

Check out the Meta AI blog post to learn about how Hydra fits into Meta's efforts to reengineer deep learning platforms for interoperability.

Citing Hydra

If you use Hydra in your research please use the following BibTeX entry:

@Misc{Yadan2019Hydra,
  author =       {Omry Yadan},
  title =        {Hydra - A framework for elegantly configuring complex applications},
  howpublished = {Github},
  year =         {2019},
  url =          {https://github.com/facebookresearch/hydra}
}

hydra's Issues

Add plugin metadata for minimum python version

this can be validated when instantiating a plugin that needs python 3 on python 2.

interpolation of list configs

I have two config files for which merging/interpolation doesn't work because the config argument is not a primitive type (but a list)

robot:
class: learning_control.simulation.bullet_sim.BulletSimulation
params:
joint_limits: [2.96705972839,
2.09439510239,
2.96705972839,
2.09439510239,
2.96705972839,
2.09439510239,
3.05432619099]
torque_limits: [80, 80, 40, 40, 9, 9, 9]

dynamics_model:
class: learning_control.dynamics.learned_dynamics.LearnedDynamicsModel
params:
torque_limits: ${robot.params.torque_limits}
joint_limits: ${robot.params.joint_limits}

this breaks with:
File "/nfs/home/fmeier/.conda/envs/learning-control-py36/lib/python3.6/site-packages/omegaconf/omegaconf.py", line 343, in resolve_value
raise ValueError("String interpolation key '{}' refer a config node".format(inter_key))
ValueError: String interpolation key 'robot.params.torque_limits' refer a config node

Investigate integrating https://pypi.org/project/towncrier/

Town crier can manage automatic change list based on news files.

Allow setting environment variable automatically for jobs

This is low priority at the moment, please vote up if you will find it useful.

Control of env variables for launched jobs.
This could be based on a special node in the job config:

env:
  OMP_NUM_THREADS: 1 
  # copy env from running machine
  USER: ${env:USER} 

TO BE CLEAR:
You can already use environment variables in configs:
See the OmegaConf documentation.

This issue is about allowing easier control of the environment variable of Hydra apps for other purposes (for example, setting OMP_NUM_THREADS automatically to 1).

Crash after slurm training.

(pytorch-c) yuandong@devfair0306:~/dl_theory2$ python two_layer_new2.py -s
[2019-08-12 11:28:50,983][hydra_plugins.fairtask.fairtask_launcher][INFO] - Sweep output dir : /checkpoint/yuandong/outputs/2019-08-12_11-28-50
[2019-08-12 11:28:51,039][hydra_plugins.fairtask.fairtask_launcher][INFO] - Launching 1 jobs to slurm queue
[2019-08-12 11:28:51,039][hydra_plugins.fairtask.fairtask_launcher][INFO] -     #0 :
/private/home/yuandong/.local/lib/python3.7/site-packages/distributed/deploy/local.py:112: UserWarning: The ip keyword has been moved to host
  warnings.warn("The ip keyword has been moved to host")
Dask dashboard for "slurm" at http://localhost:8006.
Traceback (most recent call last):
  File "two_layer_new2.py", line 222, in <module>
    main()
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra/hydra.py", line 165, in decorated_main
    run_hydra(task_function, config_path, strict)
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra/hydra.py", line 145, in run_hydra
    hydra.sweep(overrides=args.overrides)
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra/hydra.py", line 88, in sweep
    return sweeper.sweep(arguments=cfg.hydra.overrides.task)
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra/sweeper/step_sweeper.py", line 61, in sweep
    results = self.launcher.launch(batch)
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra_plugins/fairtask/fairtask_launcher.py", line 96, in launch
    return loop.run_until_complete(self.run_sweep(queue, job_overrides))
  File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/hydra_plugins/fairtask/fairtask_launcher.py", line 78, in run_sweep
    return await gatherl(runs)
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/fairtask/util.py", line 26, in gatherl
    return await multi(iters)
  File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/tornado/gen.py", line 501, in callback
    result_list.append(f.result())
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/fairtask/queue.py", line 418, in __call__
    return await self.submit_fn(*args, **kwargs)
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/fairtask/queue.py", line 248, in submit
    pure=pure,
  File "/private/home/yuandong/.local/lib/python3.7/site-packages/distributed/client.py", line 1412, in submit
    key = funcname(func) + "-" + tokenize(func, kwargs, *args)
  File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/dask/base.py", line 661, in tokenize
    return md5(str(tuple(map(normalize_token, args))).encode()).hexdigest()
  File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/dask/utils.py", line 486, in __call__
    return meth(arg, *args, **kwargs)
  File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/dask/base.py", line 705, in normalize_object
    method = getattr(o, "__dask_tokenize__", None)
  File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 72, in __getattr__
    return self.get(key=key, default_value=None)
  File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 83, in get
    return self._resolve_with_default(key=key, value=self.get_node(key), default_value=default_value)
  File "/private/home/yuandong/miniconda3/envs/pytorch-c/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 88, in get_node
    raise KeyError("Accessing unknown key in a struct : {}".format(self.get_full_key(key)))
KeyError: 'Accessing unknown key in a struct : __dask_tokenize__'

Improve test coverage

hit 100%?

dynamic command line help

Try to incorporate user config families into --help dynamically.

Tests are leaving log files in the current directory

Move non-public APIs into hydra._internal

To discourage usage of non public APIs.

Integrate logo into website

move hydra generated files into an output directory relative to the job dir

automatically generated files like config.yaml, overrides.yaml, and others can go here.
Also additional debug information about the hydra config composition order.

Document using config to specify experiments

Docusaurus 2 TODO list

• Finish porting documentation from demos into websides/docs.
• Figure out a way to show a split screen file browser for demo code and configs
• Deploy website to gh pages or similar

Clean up duplication of output directory pattern

We have hydra.run.dir and hydra.sweep.dir.
consider eliminating one of them.

Document how to override subdir pattern

And generally document how to configure hydra

hydra:
     subdir: ${job:num}_${job:id}_${job:override_dirname}

Document Hydra in cluster users docs

Missing copyright headers

Hydra returns empty config for .yml files.

Hydra produces an empty config when the file has a .yml extension as opposed to a .yaml extension.

A minimal reproduction is here:
hydra_test.py

import hydra
class A:
    def __init__(self, **kwargs):
        print(kwargs)


@hydra.main('./config_test.yml')
def main(cfg):
    print(cfg)
    print(cfg.obj_test)
    print("ASDFASDFASDFAFSD")
    hydra.utils.instantiate(cfg.obj_test)


if __name__ == '__main__':
    main()

config_test.yml / config_test.yaml

obj_test:
  class: hydra_test.A
  params:
    a: 123
    b: 421

Put both of these files into the same folder and run with python hydra_test.py

submitit support

Work in progress, waiting on some of submitit's fixes to get pushed.
that's most of it:

https://github.com/fairinternal/submitit/pull/120

Also reactivating local queue:
https://github.com/fairinternal/submitit/issues/121

try colorlog

https://pypi.org/project/colorlog/

Flow to create an empty hydra app for quick start

for example:
hydra init --name demos/4_sweep/sweep_example.py
can create an initial tree with working code like:

demos/4_sweep/
├── conf
│   ├── config.yaml
│   └── hydra.yaml
└── sweep_example.py

Support Windows

CI for mac

Integrate with bash completion

see mbrl for example

Option to raise an error when a config value is not defined?

Doing args.x where x is not defined in the config silently results in None -- is there a way to make this throw an error instead? Potentially by changing the resolver of an OmegaConf object so that it throws an error? My current hack is to just pull out the dict from the OmegaConf object and put it into a new object that just uses a non-default dict. Shamelessly sharing that:

@hydra.main(config_path='./config.yaml')
def main(args):
    class hack_args(object):
        def __init__(self, args):
            self.__dict__ = args.__dict__['content']
    args = hack_args(args)

support user provided sweep list

Unexpected keyword arg

calebh@devfair020:~/downloads/tactile-manipulation-master/video_prediction$ python train_model.py -s model.params.num_masks=5,10,20 model.params.latent_dim=4,
8,16 num_context_frames=2,4 lr=2e-3,2e-4,2e-5
Traceback (most recent call last):
  File "train_model.py", line 60, in <module>
    sys.exit(experiment())
  File "/private/home/calebh/miniconda3/envs/tacticle-manipulation/lib/python3.7/site-packages/hydra/hydra.py", line 190, in decorated_main
    run_hydra(task_function, config_path, strict)
  File "/private/home/calebh/miniconda3/envs/tacticle-manipulation/lib/python3.7/site-packages/hydra/hydra.py", line 150, in run_hydra
    hydra.sweep(overrides=args.overrides)
  File "/private/home/calebh/miniconda3/envs/tacticle-manipulation/lib/python3.7/site-packages/hydra/hydra.py", line 89, in sweep
    verbose=self.verbose)
  File "/private/home/calebh/miniconda3/envs/tacticle-manipulation/lib/python3.7/site-packages/hydra/plugins.py", line 38, in instantiate_sweeper
    verbose=verbose)
  File "/private/home/calebh/miniconda3/envs/tacticle-manipulation/lib/python3.7/site-packages/hydra/sweeper/step_sweeper.py", line 28, in setup
    verbose=verbose
  File "/private/home/calebh/miniconda3/envs/tacticle-manipulation/lib/python3.7/site-packages/hydra/plugins.py", line 50, in instantiate_launcher
    verbose=verbose)
TypeError: setup() got an unexpected keyword argument 'hydra_cfg'

I think its because 2c116a0 changed the parameter names of FAIRTaskLauncher.setup

Document: debugging methods

1. How to debug config composition problems?
2. How to debug job launch issues?

Allow using the job arguments in the output directory pattern of a local run (and not just a sweep)

It looks like there is no way to change the name of an output folder to include parameter values in hydra's configure. I can only see the parameter value in the sweeper's subfolder, as demonstrated in https://github.com/fairinternal/hydra/tree/master/demos/99_hydra_configuration/workdir

Fix CI testing of Python 2.7 on Windows

https://circleci.com/docs/2.0/hello-world-windows/

Hydra client API

Create Hydra client API to allow jobs to interact with Hydra.
one immediate need is to allow executed code to find out what is the original working directory is was executed from.

test for demos/4_sweep

4_sweep test is partial, improve it

Document and test : custom log configuration

Add automated lint to CI

Should also check copyright headers.

https://our.intern.facebook.com/intern/wiki/Open_Source/Tooling/Automated_Checkup/Copyright_Headers_Exist/

Update website config to conform to an objective oriented approach.

HYDRA EX: Simple command line
… as the number of arguments grow become pain

HYDRA EX: single config file

… want to override

HYDRA EX: config w/command line

… want to use two different methods

HYDRA EX: file w/alternatives via a variable
(why non-overridable defaults?)

Config groups:
Path 1:
Config getting big -> split using config splitting (adam for example)
We want to try nesterov: add nesterov, put in an optimizer dir and change defaults to select one of them. Override from command line.

Path 2:
Adam exists in the config file, we want try nesterov as well -> split adam out into optimizer/adam.yaml and create optimizer/netsterov.yaml and put in the defaults: switch to select adam. Override from command line

Create a pip package in the fair cluster

Customize config search path

Add a method to influence the config search path.

Adding symlink inside the subdir to job.err file.

When you run a big sweep and some of your jobs fail early, it is usefully to look at the job.err file. The problem is that these files are stored in the sweep root folder, and it is pretty hard to find the corresponding *.err file for a particular jobs. Maybe we can add a symlink to this file and store it in the particular job directory?

automate website deloyment

Automate website deployment through circleci.
requires enabling docusaurus bot with commit access to gh-pages branch

consider using mypy for static analysis of the code

Continuous integration

• integrate with tox and travis (or circleci)

consider xar integration

This can be interesting as a method to launch self contained jobs.
Can we automate the packaging somehow?

https://github.com/facebookincubator/xar and https://pypi.org/project/pex/

break --sweep into it's base components

--sweep right now changes both the launcher and activate a sweeper.
should separate it into two things.

--run-all:
Launch with the configured sweeper.

launcher=basic|submitit|fairtask:
Which launcher to use. basic is the default and can only run jobs locally and serially.

Overriding the hydra config/custom workdirs

I want to have multiple experiment scripts in a directory and have custom hydra config (and workdir directory) for each of them. From this example and the source here it seems like .hydra/hydra.yaml is currently hard-coded in as the path to look for overriding hydra config. What do you think of making this more configurable? I want to have something like this:

+ `exp_1.py`
+ `exp_1.yaml`
+ `exp_1_hydra.yaml`
+ `exp_2.py`
+ `exp_2.yaml`
+ `exp_2_hydra.yaml`
...

where in the exp_N.py points to the relevant *_hydra.yaml config for over-riding.

Or is there an alternative to have custom workdir names? Maybe it could be added to the main decorator with a default value of outputs so that the hydra default config doesn't need to be explicitly over-loaded just for this?

Modifications to job directory in task config not incorporated

Currently, changes to the job directory cannot be specified from the task configuration. It seems like the job directory is specified completely by the hydra config before the task config is merged in, so that the modifications to the job directory from the task configuration are not incorporated.

Related lines:

hydra/hydra/hydra.py

Lines 59 to 66 in 8da2927

	hydra_cfg = self.config_loader.load_hydra_cfg(overrides)
	return utils.run_job(config_loader=self.config_loader,
	hydra_cfg=hydra_cfg,
	task_function=self.task_function,
	overrides=overrides,
	verbose=self.verbose,
	job_dir=hydra_cfg.hydra.run.dir,
	job_subdir_key=None)

hydra/hydra/utils.py

Lines 167 to 182 in 8da2927

	def run_job(config_loader, hydra_cfg, task_function, overrides, verbose, job_dir, job_subdir_key):
	filtered_overrides = filter_overrides(overrides)
	JobRuntime().set('override_dirname', get_overrides_dirname(filtered_overrides))
	task_cfg = config_loader.load_task_cfg(overrides)
	# merge with task to allow user to change the behavior of the working directory/subdir from the task itself.
	# this can be useful for having output subdir that depends on random_seed, for example.
	hydra_and_task_cfg = OmegaConf.merge(hydra_cfg, task_cfg)
	JobRuntime().set('name', hydra_and_task_cfg.hydra.name)
	old_cwd = os.getcwd()
	working_dir = job_dir
	if job_subdir_key is not None:
	# evaluate job_subdir_key lazily.
	# this is running on the client side in sweep and contains things such as job:id which
	# are only available there.
	subdir = hydra_and_task_cfg.select(job_subdir_key)
	working_dir = os.path.join(working_dir, str(subdir))

evaluate black formatter

https://github.com/psf/black

Demo: specializing configuration depending on multiple selected config families

add option to set sweep parameters into the working directory of a job

Move website to real domain once it's established

Demo: instantiate classes from config

Create a demo with instantiating classes from config