ksettaluri6 / autockt

Deep Reinforcement Learning of Analog Circuit Designs

UPDATE: please see the journal paper for additional results on designing a two-stage folded-cascode and two-stage amplifier (https://ieeexplore.ieee.org/abstract/document/9576505).

AutoCkt: Deep Reinforcement Learning of Analog Circuit Designs

Code for Deep Reinforcement Learning of Analog Circuit Designs, presented at Design Automation and Test in Europe, 2020. Note that the results shown in the paper include those from NGSpice and Spectre. NGSpice is free and can be installed online (see Setup). Spectre requires a license, as well as access to the particular technology; the code for this will be open sourced at a later time.

Setup

This setup requires Anaconda. In order to obtain the required packages, run the command below from the top level directory of the repo to install the Anaconda environment:

conda env create -f environment.yml

To activate the environment run:

source activate autockt

You might need to install some packages further using pip if necessary. To ensure the right versions, look at the environment.yml file.

NGspice 2.7 needs to be installed separately, via this installation link. Page 607 of the pdf manual on the website has instructions on how to install. Note that you might need to remove some of the flags to get it to install correctly for your machine.

Code Setup

The code is setup as follows:

The top level directory contains two sub-directories:

• AutoCkt/: contains all of the reinforcement code
  • val_autobag_ray.py: top level RL script, used to set hyperparameters and run training
  • rollout.py: used for validation of the trained agent, see file for how to run
  • envs/ directory: contains all OpenAI Gym environments. These function as the agent in the RL loop and contain information about parameter space, valid action steps and reward.
• eval_engines/: Contains all of the code pertaining to simulators
  • ngspice/: this directory runs all NGSpice related scripts.
    • netlist_templates: the exported netlist file modified using Jinja to update any MOS parameters
    • specs_test/: a directory containing a unique yaml file for each circuit with information about design specifications, parameter ranges, and how to normalize.
    • script_test/: directory with files that test functionality of interface scripts

Training AutoCkt

Make sure that you are in the Anaconda environment. Before running training, the circuit netlist must be modified in order to point to the right library files in your directory. To do this, run the following command:

python eval_engines/ngspice/ngspice_inputs/correct_inputs.py 

To generate the design specifications that the agent trains on, run:

python autockt/gen_specs.py --num_specs ##

The result is a pickle file dumped to the gen_specs/ folder.

To train the agent, open ipython from the top level directory and then:

run autockt/val_autobag_ray.py

The training checkpoints will be saved in your home directory under ray_results. Tensorboard can be used to load reward and loss plots using the command:

tensorboard --logdir path/to/checkpoint

To replicate the results from the paper, num_specs 350 was used (only 50 were selected for each CPU worker). Ray parallelizes according to number of CPUs available, that affects training time.

Validating AutoCkt

The rollout script takes the trained agent and gives it new specs that the agent has never seen before. To generate new design specs, run the gen_specs.py file again with your desired number of specs to validate on. To run validation, open ipython:

run autockt/rollout.py /path/to/ray/checkpoint --run PPO --env opamp-v0 --num_val_specs ### --traj_len ## --no-render

• num_val_specs: the number of untrained objectives to test on
• traj_len: the length of each trajectory

Two pickle files will be updated: opamp_obs_reached_test and opamp_obs_nreached_test. These will contain all the reached and unreached specs, respectively.

Results

Please note that results vary greatly based on random seed and spec generation (both for testing and validation). An example spec file is provided that was used to generate the results below.

The rollout generalization results will show up as pickle files opamp_obs_reached_test and opamp_obs_nreached_test. For this particular run, we obtained 938/1000. Additional runs were also conducted, and we found that the results varied from 80%-96% depending on the generated specs during rollout, and the specs that were changed during training. Our results were obtained by running on an 8 core machine, we've found that running on anything below 2 cores results in weird training behavior.

autockt's Issues

1

No available node types can fulfill resource request.

INFO worker.py:1544 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
INFO algorithm_config.py:2899 -- Your framework setting is 'tf', meaning you are using static-graph mode. Set framework='tf2' to enable eager execution with tf2.x. You may also then want to set eager_tracing=True in order to reach similar execution speed as with static-graph mode.
INFO algorithm_config.py:2899 -- Your framework setting is 'tf', meaning you are using static-graph mode. Set framework='tf2' to enable eager execution with tf2.x. You may also then want to set eager_tracing=True in order to reach similar execution speed as with static-graph mode.
(autoscaler +10s) Tip: use ray status to view detailed cluster status. To disable these messages, set RAY_SCHEDULER_EVENTS=0.
(autoscaler +10s) Error: {'GPU': 1.0, 'CPU': 1.0}. Add suitable node types to this cluster to resolve this issue.
== Status ==
Current time: 2023-04-05 03:03:10 (running for 00:00:05.11)
Memory usage on this node: 4.7/7.5 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/4 CPUs, 0/0 GPUs, 0.0/1.98 GiB heap, 0.0/0.99 GiB objects
Result logdir: /home/Labexam/ray_results/train_45nm_ngspice
Number of trials: 1/1 (1 PENDING)
+-----------------------------+----------+-------+
| Trial name | status | loc |
|-----------------------------+----------+-------|
| PPO_TwoStageAmp_e98f0_00000 | PENDING | |
+-----------------------------+----------+-------+

Questions about the choice of parameters initializations and normalization values

Hi, I wanted to understand some choices made in the code

from my understanding of the paper each parameter should be initialized to mid-range , but instead these "magic" ? values are used at every reset:

AutoCkt/autockt/envs/ngspice_vanilla_opamp.py

Line 157 in 2d01bc6

self.cur_params_idx = np.array([33, 33, 33, 33, 33, 14, 20])

Also I understand that normalization is generally useful in RL , can you explain why these values are chosen for normalization?

AutoCkt/eval_engines/ngspice/ngspice_inputs/yaml_files/two_stage_opamp.yaml

Line 15 in 2d01bc6

normalize: !!python/tuple [350, 0.001, 60, 950000.0]

can you explain the choice of these values and what effect this might have?

Thanks.

ac/dc file doesn't exist

when execute "run val_autobag_ray “ , it says,ac/dc file doesn't exist.When the following changes are made, it should work. This may be because ngspice has updated some functionality.
TwoStageClass.py:

ac_fname = os.path.join(output_path, 'ac.csv.data')
dc_fname = os.path.join(output_path, 'dc.csv.data')
......
ac_raw_outputs = np.genfromtxt(ac_fname, skip_header=0)
dc_raw_outputs = np.genfromtxt(dc_fname, skip_header=0)

Question about "run autockt/val_autobag_ray.py" when training the agent

Hello, I am studying your open source code, but when I run according to the operation instructions of the reinforcement learning "training agent" part in the readme file, I have the following problem. I don't know how to solve it. I would like to trouble you to reply in time. Very grateful!
1.run autockt/val_autobag_ray.py
(PPOTrainer pid=120010) ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::RolloutWorker.init() (pid=119960, ip=192.168.10.12)
(PPOTrainer pid=120010) File "/home/wangyuan/.local/lib/python3.9/site-packages/ray/rllib/evaluation/rollout_worker.py", line 464, in init
(PPOTrainer pid=120010) _validate_env(self.env, env_context=self.env_context)
(PPOTrainer pid=120010) File "/home/wangyuan/.local/lib/python3.9/site-packages/ray/rllib/evaluation/rollout_worker.py", line 1702, in _validate_env
(PPOTrainer pid=120010) raise EnvError(
(PPOTrainer pid=120010) ray.rllib.utils.error.EnvError: Env's observation_space Box([-1. -1. -1. -1. -1. -1. -1. -1. 1. 1. 1. 1. 1. 1. 1.], [0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 1. 1. 1. 1.], (15,), float32) does not contain returned observation after a reset ([ 9.9401027e-03 -8.5833985e-01 -7.7145004e-01 8.2071406e-01
(PPOTrainer pid=120010) -1.5965167e-02 5.8320427e-01 0.0000000e+00 6.1827123e-01
(PPOTrainer pid=120010) 3.3000000e+01 3.3000000e+01 3.3000000e+01 3.3000000e+01
(PPOTrainer pid=120010) 3.3000000e+01 1.4000000e+01 2.0000000e+01])!

Looking forward to your reply!

ValueError

Hi,
I followed the steps to run the code. But I got the ValueError when running the
run autockt/val_autobag_ray.py in the ipython.
ValueError: ('Observation outside expected value range', Box(15,), array([ 9.94010248e-03, -8.58339853e-01, -7.71450042e-01, 8.20714082e-01,
-1.80438449e-01, 5.84788222e-01, 0.00000000e+00, 8.28439265e-01,
3.30000000e+01, 3.30000000e+01, 3.30000000e+01, 3.30000000e+01,
3.30000000e+01, 1.40000000e+01, 2.00000000e+01]))
Since I'm new to circuit design, I cannot understand the error. Could be the problem with ngspice?

Facing error in execution of code

We are trying to execute the code by following the Read.me file. Installed all the packages and libraries mentioned, but during the installation of libraries and packages we are facing issues in version. Either there is no version available which has been mentioned or we are unable to find a few packages which are required with the proper version.If we are installing the version which is mentioned and then installing some package then this package is changing the version of the 1st package.

After following all the process mentioned in read.me file and avoiding the package versions. We are here with this error:

TypeError Traceback (most recent call last)
~/anaconda3/lib/python3.9/site-packages/ray/tune/experiment/experiment.py in from_json(cls, name, spec)
306 try:
--> 307 exp = cls(name, run_value, **spec)
308 except TypeError as e:

TypeError: init() got an unexpected keyword argument 'checkpoint_freq'

During handling of the above exception, another exception occurred:

TuneError Traceback (most recent call last)
~/AutoCkt/AutoCkt-master/autockt/val_autobag_ray.py in
31 #If checkpoint fails for any reason, training can be restored
32 if not args.checkpoint_dir:
---> 33 trials = tune.run_experiments({
34 "train_45nm_ngspice": {
35 "checkpoint_freq":1,

~/anaconda3/lib/python3.9/site-packages/ray/tune/tune.py in run_experiments(experiments, scheduler, server_port, verbose, progress_reporter, resume, reuse_actors, trial_executor, raise_on_failed_trial, concurrent, callbacks, _remote)
847 # because it schematize the experiments
848 # and it conducts the implicit registration.
--> 849 experiments = _convert_to_experiment_list(experiments)
850
851 if concurrent:

~/anaconda3/lib/python3.9/site-packages/ray/tune/experiment/experiment.py in _convert_to_experiment_list(experiments)
492 exp_list = [experiments]
493 elif type(experiments) is dict:
--> 494 exp_list = [
495 Experiment.from_json(name, spec) for name, spec in experiments.items()
496 ]

~/anaconda3/lib/python3.9/site-packages/ray/tune/experiment/experiment.py in (.0)
493 elif type(experiments) is dict:
494 exp_list = [
--> 495 Experiment.from_json(name, spec) for name, spec in experiments.items()
496 ]
497

~/anaconda3/lib/python3.9/site-packages/ray/tune/experiment/experiment.py in from_json(cls, name, spec)
307 exp = cls(name, run_value, **spec)
308 except TypeError as e:
--> 309 raise TuneError(
310 f"Failed to load the following Tune experiment "
311 f"specification:\n\n {pp.pformat(spec)}.\n\n"

TuneError: Failed to load the following Tune experiment specification:

{'checkpoint_freq': 1,
'config': {'env': <class 'autockt.envs.ngspice_vanilla_opamp.TwoStageAmp'>,
'env_config': {'generalize': True, 'run_valid': False},
'horizon': 30,
'model': {'fcnet_hiddens': [64, 64]},
'num_gpus': 0,
'num_workers': 6,
'train_batch_size': 1200},
'stop': {'episode_reward_mean': -0.02}}.

Please check that the arguments are valid. Experiment creation failed with the following error:
init() got an unexpected keyword argument 'checkpoint_freq'

Additionally adding the snapshot of error: