bbopt / hypernomad

A library for the hyperparameter optimization of deep neural networks

---

Hyperparameter optimization of deep neural networks with HyperNOMAD

---

HyperNOMAD is a C++ and Python package dedicated to the hyperparameter optimization of deep neural networks. The package contains a blackbox specifically designed for this problematic and provides a link with the NOMAD software used for the optimization. The blackbox takes as inputs a list of hyperparameters, builds a corresponding deep neural network in order to train, validate and test it on a specific data set before returning the test error as a mesure of performance. NOMAD is then used to minimize this error. The following appendix provides an overview of how to use the HyperNOMAD package.

The following tutorial shows the different steps to take in order to run HyperNOMAD on a first example. The complete functionalities of HyperNOMAD are described in the documentation.

Prerequisites

In order to run HyperNOMAD correctly, please make sure to have:

• A compiled version of NOMAD 3. Please note that HyperNomad is not compatible with NOMAD 4.
• Python > 3.6
• PyTorch
• GCC > 9.0

installation of HyperNOMAD

First build the executable by running the following command.

make

    building HyperNOMAD ...

    To be able to run the example
    the HYPERNOMAD_HOME environment variable
    must be set to the HyperNOMAD home directory
    

When the compilation is successful, a message appears asking to set an environment variable 'HYPERNOMAD_HOME'. This can be done by adding a line in the file .profile or .bashrc :

    export HYPERNOMAD_HOME=hypernomad_directory

The executable hypernomad.exe is located in the bin directory. You can check that the installation is successful by trying to run the commad

    $HYPERNOMAD_HOME/bin/./hypernomad.exe -i

which should return the following informations:

    --------------------------------------------------
      HyperNomad - version 1.0
    --------------------------------------------------
      Using Nomad version 3.9.0 - www.gerad.ca/nomad
    --------------------------------------------------

    Run           : hypernomad.exe parameters_file
    Info          : hypernomad.exe -i
    Help          : hypernomad.exe -h
    Version       : hypernomad.exe -v
    Usage         : hypernomad.exe -u
    Neighboors    : hypernomad.exe -n parameters_file

Getting started

The next phase is to create a parameter file that contains the necessary informations to specify the classification problem, the search space and the initial starting point. HyperNOMAD allows for a good flexibility of tuning a convolutional network by considering multiple aspects of a network at once such as the architecture, the dropout rate, the choice of the optimizer and the hyperparameters related to the optimization aspect (learning rate, weight decay, momentum, ...), the batch size, etc. The user can choose to optimize all these aspects or select a few and fixe the others to certain values. The user can also change the default range of each hyperparameter.

This information is passed through the parameter file by using a specific synthax:

  KEYWORD   INITIAL_VALUE   LOWER_BOUND   UPPER_BOUND   FIXED/VAR

Here is an example of an acceptable parameter file. First, the dataset MNIST is choosen and we specify that HyperNOMAD is allowed to try a maximum of 100 configurations. Then, the number of convolutional layers is fixed throught the optimization to 5, the two '-' appearing after the '5' mean that the default lower and upper bounds are not changed. The kernels, number of fully connected layers and activation function are respectively initialized at 3, 6, and 2 (Sigmoid) and the dropout rate is initialized at 0.6 with a new lower bound of 0.3 and upper bound of 0.8

Finally, all the remaining hyperparameters that are not explicitly mentioned in this file are fixed to their default values during the optimization.

DATASET                 MNIST
MAX_BB_EVAL             100

# Optional information
NUM_CON_LAYERS          5  -  -  FIXED # The initial value is fixed
                                       # lower and upper bounds have no
                                       # influence when parameter 
                                       # is fixed.

KERNELS                 3              # Only the initial value is set (not fixed)
                                       # the lower bound and upper bound
                                       # have default values.

NUM_FC_LAYERS           6
ACTIVATION_FUNCTION     2
DROPOUT_RATE            0.6  0.3 0.8  # The lower and upper bounds 
                                      # are set to values that are not 
                                      # the default ones
REMAINING_HPS           FIXED

More details are provided in the user guide section of the documentation.

Running an optimization

The optimization starts by executing the command:

$HYPERNOMAD_HOME/bin/./hypernomad.exe parameter_file.txt

Multiple examples of parameter files are provided in the folder examples. One uses CIFAR-10 starting from the default starting point and the other use MNIST with different configurations.

To use these files, the cammand is:

$HYPERNOMAD_HOME/bin/./hypernomad.exe $HYPERNOMAD_HOME/examples/cifar10_default.txt

or

$HYPERNOMAD_HOME/bin/./hypernomad.exe $HYPERNOMAD_HOME/examples/mnist_fc_optim.txt

Citing HyperNOMAD

If you use HyperNOMAD, please cite the following paper.

@article{lakhmiri2021hypernomad,
  title={HyperNOMAD: Hyperparameter Optimization of Deep Neural Networks Using Mesh Adaptive Direct Search},
  author={Lakhmiri, Dounia and Digabel, S{\'e}bastien Le and Tribes, Christophe},
  journal={ACM Transactions on Mathematical Software (TOMS)},
  volume={47},
  number={3},
  pages={1--27},
  year={2021},
  publisher={ACM New York, NY, USA}
}

hypernomad's Issues

Mettre un commentaire sur le extended poll trigger

https://github.com/DouniaLakhmiri/HYPERNOMAD/blob/c6894ea5931979c4fe3ccadd2f789653d4bd3b4d/hypernomad.cpp#L93

Memory leak

In create_extended_points we should not create new signature without delete.

The starting point cannot be evaluated

If I try to use HyperNOMAD I have the following problem

caterina99@pc0425:/mnt/c/Users/calessi/Desktop/HyperNomad_linux/examples$ hypernomad.exe mnist_x0.txt

WARNING: hyperparameters explicitely set by name are superseded
by settings done using X0, LOWER_BOUND and UPPER_BOUND.

BLOCKS OF HYPERPARAMETERS
Each block has a head hyperparameters and possibly
several groups of associated hyperameters.

Convolutionnal layers {

Head of block NUM_CON_LAYERS -> x0=2, lb=0, ub=100, is VARIABLE
Multiple times associated hyperparameters: 2 groups
Group #0
NUM_OUTPUT_LAYERS -> x0=6, lb=1, ub=1000, is VARIABLE
KERNELS -> x0=5, lb=1, ub=20, is FIXED
STRIDES -> x0=1, lb=1, ub=3, is VARIABLE
PADDINGS -> x0=0, lb=0, ub=2, is VARIABLE
POOLING_SIZE -> x0=1, lb=1, ub=5, is VARIABLE
Group #1
NUM_OUTPUT_LAYERS -> x0=16, lb=1, ub=1000, is VARIABLE
KERNELS -> x0=5, lb=1, ub=20, is FIXED
STRIDES -> x0=1, lb=1, ub=3, is VARIABLE
PADDINGS -> x0=0, lb=0, ub=2, is VARIABLE
POOLING_SIZE -> x0=1, lb=1, ub=5, is VARIABLE
}

Full layers {

Head of block NUM_FC_LAYERS -> x0=2, lb=0, ub=500, is VARIABLE
Multiple times associated hyperparameters: 2 groups
Group #0
SIZE_FC_LAYER -> x0=128, lb=1, ub=1000, is VARIABLE
Group #1
SIZE_FC_LAYER -> x0=84, lb=1, ub=1000, is VARIABLE
}

Batch size {

Head of block BATCH_SIZE -> x0=128, lb=1, ub=400, is VARIABLE
No associated hyperparameters
}

Optimizer {

Head of block OPTIMIZER_CHOICE -> x0=3, lb=1, ub=4, is VARIABLE
One time associated hyperparameters (always 1 group)
Group #0
OPT_PARAM_1 -> x0=0.1, lb=0, ub=1, is VARIABLE
OPT_PARAM_2 -> x0=0.9, lb=0, ub=1, is VARIABLE
OPT_PARAM_3 -> x0=0.0005, lb=0, ub=1, is VARIABLE
OPT_PARAM_4 -> x0=0, lb=0, ub=1, is VARIABLE
}

Dropout rate {

Head of block DROPOUT_RATE -> x0=0.2, lb=0, ub=0.95, is FIXED
No associated hyperparameters
}

Activation function {

Head of block ACTIVATION_FUNCTION -> x0=1, lb=1, ub=3, is VARIABLE
No associated hyperparameters
}

Warning: {
Model use is disabled for problem with categorical variables.
}
Warning: {
Setting granularity different than 0 is disabled for problem with categorical variables.
}
Warning: {
Default anisotropic mesh is disabled with categorical and binary variables.
}

HyperNomad - version 1.0

Using Nomad version 3.9.1 - www.gerad.ca/nomad

Nomad parameters {

    dimension            : n=22
    lower bounds         : (  NaN    1    1    1    0    1    1    1    1    0    1  NaN    1    1    1  NaN    0    0    0    0    0    1 )
    upper bounds         : (  NaN 1000   20    3    2    5 1000   20    3    2    5  NaN 1000 1000  400  NaN    1    1    1    1 0.95    3 )
    fixed variables      : (  NaN  NaN    5  NaN  NaN  NaN  NaN    5  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  0.2  NaN )
    blackbox input types : (   cat(C)   int(I)   int(I)   int(I)   int(I)   int(I)   int(I)   int(I)   int(I)   int(I)   int(I)   cat(C)   int(I)   int(I)   int(I)   cat(C)  cont(R)  cont(R)  cont(R)  cont(R)  cont(R)   int(I) )
    extended poll trigger: 10
    variable groups {
            group #0 {
                    indexes: { 14 }
                    directions {
                            n             : 1
                            types         : { [Ortho-MADS n+1 NEG] }
                            sec poll types: { }
                            int poll types: { }
                            seed          : 0
                    }
            }
            group #1 {
                    indexes: { 21 }
                    directions {
                            n             : 1
                            types         : { [Ortho-MADS n+1 NEG] }
                            sec poll types: { }
                            int poll types: { }
                            seed          : 0
                    }
            }
            group #2 {
                    indexes: { 12 13 }
                    directions {
                            n             : 2
                            types         : { [Ortho-MADS n+1 NEG] }
                            sec poll types: { }
                            int poll types: { }
                            seed          : 0
                    }
            }
            group #3 {
                    indexes: { 0 11 15 }
                    no directions (categorical variables)
            }
            group #4 {
                    indexes: { 16 17 18 19 }
                    directions {
                            n             : 4
                            types         : { [Ortho-MADS n+1 NEG] }
                            sec poll types: { }
                            int poll types: { }
                            seed          : 0
                    }
            }
            group #5 {
                    indexes: { 1 3 4 5 6 8 9 10 }
                    directions {
                            n             : 8
                            types         : { [Ortho-MADS n+1 NEG] }
                            sec poll types: { }
                            int poll types: { }
                            seed          : 0
                    }
            }
    }
    blackbox outputs (m=1) {
            #0          OBJ $python /mnt/c/Users/calessi/Desktop/HyperNomad_linux/src/blackbox/pytorch_bb.py MNIST $python /mnt/c/Users/calessi/Desktop/HyperNomad_linux/src/blackbox/pytorch_sgte.py MNIST
    }
    signature                       : standard
    has surrogate                   : yes
    sort trial points with surrogate: yes
    surrogate cost                  : none
    sort trial points randomly      : no
    add seed to output file names   : yes
    solution file                   : none
    history file                    : history.0.txt
    stats file                      : (stats.0.txt) BBE ( SOL ) OBJ
    cache file                      : none
    x0                              : (    2    6    5    1    0    1   16    5    1    0    1    2  128   84  128    3  0.1  0.9 0.0005    0  0.2    1 )
    directions                      : Ortho-MADS n+1 NEG
    gmesh (isotropic) {
            granularity      :  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
            coarsening exponent    :   1
            refining exponent      :  -1
            initial mesh index     :   0
            initial mesh size      : (    1   10    1    1    1    1   10    1    1    1    1    1   10   10   10    1    1    1    1    1    1    1 )
            initial poll size      : (    1   50    5    5    5    5   50    5    5    5    5    1   50   50   50    1    1    1    1    1    1    5 )
    }
    snap to bounds                   : yes
    opportunistic evaluations        : yes
    use models (search and sort)     : no
    speculative search               : yes
    VNS search                       : no
    NelderMead (NM) search {
            max_trial_pts_nfactor: 80
            gamma (shrink):0.5
            delta_oc (outside contraction):0.5
            delta_ic (inside contraction):-0.5
            delta_e (expansion):2
            intensive: no
            opportunistic: no
            use_only_Y: no
            init_Y_best_von: no
            use_short_Y0: no
            include_factor: 8       }
    Latin-Hypercube (LH) search      : no
    cache search                     : no
    random seed / run id             : 0
    epsilon                          : 1e-13
    undefined string                 : NaN
    infinity string                  : inf
    display degrees {
            general  : full (3)
            search   : full (3)
            poll     : full (3)
            iterative: full (3)
    }
    display stats                : BBE ( SOL ) OBJ
    display all evaluations      : no
    point display limit          : no limit
    max number of blackbox eval. : 100
    max cache memory             : 2000 MB

}

MADS run {

    starting point evaluation {

            x0 eval point: (  2  6  5  1  0  1 16  5  1  0  1  2 128 84 128  3 0.1 0.9 0.0005  0 0.2  1 )

            list of points evaluation (x0 evaluation) {

                    submitted evaluation 1/1 {

                            point #0   (  2  6  5  1  0  1 16  5  1  0  1  2 128 84 128  3 0.1 0.9 0.0005  0 0.2  1 )
                            mesh indices   : (  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 )

                            not in cache
                    }

sh: 1: python: not found

                    evaluation 1/1 {

                            point #0
                            x0 evaluation unsuccessful point #0: evaluation failed (you may need to check the source of the problem).
                    }

            } end of evaluations


            terminate MADS       : yes
            termination cause    : problem with starting point evaluation
            iteration status     : unsuccessful
            new feas. incumbent  : none
            new infeas. incumbent: none

    } end of starting point evaluation

} end of run (problem with starting point evaluation)

NOMAD final display {

    cache {
            number of cache points: 1
            size in memory        : 619 B
            cache file            : -
    }

    stats {
            MADS iterations                 : 0
            blackbox evaluations            : 1

            simulated blackbox evaluations  : 0
            evaluations                     : 1
            failed evaluations              : 1 (all evaluations failed)
            interrupted sequences of eval.  : 0
            cache hits                      : 0
            number of poll searches         : 0
            dyn. direction successes        : 0
            number of speculative searches  : 0
            number of user searches         : 0
            number of LH searches           : 0
            number of NM searches           : 0
            number of TM line searches      : 0
            number of cache searches        : 0
            number of VNS searches          : 0
            no model has been constructed
            wall-clock time                 : 0s
    }

    miscellaneous {
            mesh indices                             : min= ( 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 ), max = ( 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 ), last= (  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0 )
            best feasible solution                   : no feasible solution has been found
    }

}

The starting point cannot be evaluated. Please verify that the Pytorch script is available and runs correctly. The default setting for bbExe is $python /mnt/c/Users/calessi/Desktop/HyperNomad_linux/src/blackbox/pytorch_bb.py MNIST. Make sure it works correctly on its own.