stochss / sciope

Python3 Toolkit for ML-assisted inference, optimization and parameter exploration.

README

Scalable inference, optimization and parameter exploration (sciope) is a Python 3 package for performing machine learning-assisted inference and model exploration by large-scale parameter sweeps. Please see the documentation for examples.

What can the sciope toolbox do?

• Surrogate Modeling:

  • train fast metamodels of computationally expensive problems
  • perform surrogate-assisted model reduction for large-scale models/simulators (e.g., biochemical reaction networks)

• Inference:

  • perform likelihood-free parameter inference using parallel ABC
  • train surrogate models (ANNs) as expressive summary statistics for likelihood-free inference
  • perform efficient parameter sweeps based on statistical designs and sampling techniques

• Optimization:

  • optimize a specified objective function or surrogate model using a variety of approaches

• Model exploration:

  • perform large distributed parameter sweep applications for any black-box model/simulator which output time series data
  • generates time series features/summary statistics on simulation output and visualize parameter points in feature space
  • interactive labeling of paramater points in feature space according to the users preferences over the diversity of model behaviors
  • supports semi-supervised learning and downstream classifiers

• Version 0.4

How do I get set up?

Please see the documentation for instructions to install and examples. The easiest way to start using Sciope is through the StochSS online platform (https://app.stochss.org).

Steps to a successful contribution

1. Fork Sciope (https://help.github.com/articles/fork-a-repo/)
2. Make the changes to the source code in your fork.
3. Check your code with PEP8 or pylint. Please limit text to 80 columns wide.
4. Each feature or bugfix commit should consist of the corresponding code, tests, and documentation.
5. Create a pull request to the develop branch in Sciope.
6. Please feel free to use the comments section to communicate with us, and raise issues as appropriate.
7. The pull request gets accepted and your new feature will soon be integrated into Sciope!

Who do I talk to?

• Prashant Singh ([email protected])
• Fredrik Wrede ([email protected])
• Andreas Hellander ([email protected])

Citing Sciope

To cite Sciope, please reference the Bioinformatics application note. Sample Bibtex is given below:

@article{sciope,
    author = {Singh, Prashant and Wrede, Fredrik and Hellander, Andreas},
    title = "{Scalable machine learning-assisted model exploration and inference using Sciope}",
    journal = {Bioinformatics},
    year = {2020},
    month = {07},
    issn = {1367-4803},
    doi = {10.1093/bioinformatics/btaa673},
    url = {https://doi.org/10.1093/bioinformatics/btaa673},
    note = {btaa673},
    eprint = {https://academic.oup.com/bioinformatics/advance-article-pdf/doi/10.1093/bioinformatics/btaa673/33529616/btaa673.pdf},
}

sciope's Issues

Singular Matrix Error

devils_v0_month.ipynb.zip

Data File:
DFDT Population
Devil Population

We are consistently getting the following error using the above model:

---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
<ipython-input-15-a92b38b04305> in <module>
      7     # First compute the fixed(observed) mean
      8     abc.compute_fixed_mean(chunk_size=2)
----> 9     res = abc.infer(num_samples=100, batch_size=10, chunk_size=2)

/opt/conda/lib/python3.8/site-packages/sciope/inference/rep_smc_abc.py in infer(self, num_samples, alpha, R_trial, c, p_min, batch_size, chunk_size)
    208                 for i in range(n_cull, num_samples):
    209                     perturb_tasks.append(self._perturb_resample(population[i, :], distances[i], R_trial, tol))
--> 210                 res, = dask.compute(perturb_tasks)
    211 
    212                 # Update the population with the perturbed population

/opt/conda/lib/python3.8/site-packages/dask/base.py in compute(*args, **kwargs)
    566         postcomputes.append(x.__dask_postcompute__())
    567 
--> 568     results = schedule(dsk, keys, **kwargs)
    569     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    570 

/opt/conda/lib/python3.8/site-packages/distributed/client.py in get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2669                     should_rejoin = False
   2670             try:
-> 2671                 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2672             finally:
   2673                 for f in futures.values():

/opt/conda/lib/python3.8/site-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
   1946             else:
   1947                 local_worker = None
-> 1948             return self.sync(
   1949                 self._gather,
   1950                 futures,

/opt/conda/lib/python3.8/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    843             return future
    844         else:
--> 845             return sync(
    846                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    847             )

/opt/conda/lib/python3.8/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    323     if error[0]:
    324         typ, exc, tb = error[0]
--> 325         raise exc.with_traceback(tb)
    326     else:
    327         return result[0]

/opt/conda/lib/python3.8/site-packages/distributed/utils.py in f()
    306             if callback_timeout is not None:
    307                 future = asyncio.wait_for(future, callback_timeout)
--> 308             result[0] = yield future
    309         except Exception:
    310             error[0] = sys.exc_info()

/opt/conda/lib/python3.8/site-packages/tornado/gen.py in run(self)
    733 
    734                     try:
--> 735                         value = future.result()
    736                     except Exception:
    737                         exc_info = sys.exc_info()

/opt/conda/lib/python3.8/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1811                             exc = CancelledError(key)
   1812                         else:
-> 1813                             raise exception.with_traceback(traceback)
   1814                         raise exc
   1815                     if errors == "skip":

/opt/conda/lib/python3.8/site-packages/sciope/inference/rep_smc_abc.py in _perturb_resample()
    113             # Metropolis Perturbation
    114             if nume > 0:
--> 115                 nume = nume * self.perturbation_kernel.pdf(param.reshape(1, -1), proposal.reshape(1, -1))[0, 0,]
    116                 dnm = self.prior_function.pdf(param) * \
    117                       self.perturbation_kernel.pdf(proposal.reshape(1, -1), param.reshape(1, -1))[0, 0]

/opt/conda/lib/python3.8/site-packages/sciope/utilities/perturbationkernels/multivariate_normal.py in pdf()
     43                 pdfs.append(multivariate_normal.logpdf(x, x0[i], self.cov))
     44             else:
---> 45                 pdfs.append(multivariate_normal.pdf(x, x0[i], self.cov))
     46         return np.vstack(pdfs)
     47 

/opt/conda/lib/python3.8/site-packages/scipy/stats/_multivariate.py in pdf()
    514         dim, mean, cov = self._process_parameters(None, mean, cov)
    515         x = self._process_quantiles(x, dim)
--> 516         psd = _PSD(cov, allow_singular=allow_singular)
    517         out = np.exp(self._logpdf(x, mean, psd.U, psd.log_pdet, psd.rank))
    518         return _squeeze_output(out)

/opt/conda/lib/python3.8/site-packages/scipy/stats/_multivariate.py in __init__()
    163         d = s[s > eps]
    164         if len(d) < len(s) and not allow_singular:
--> 165             raise np.linalg.LinAlgError('singular matrix')
    166         s_pinv = _pinv_1d(s, eps)
    167         U = np.multiply(u, np.sqrt(s_pinv))

LinAlgError: singular matrix

After 2 days of simulation, throws Singular Matrix Error and no results are given. Instead return last good result

Please add error handling, so that when a run errors it does not crash but instead returns the last result calculated.

Vilar examples breaks due to error in gillespy2.StochMLDocument

Related to issue #512 in gillespy2

How do I get set up

Hello, I tried to get the documents about how to set up sciope, but the link 'Documentation' in 'How do I get set up' part did not work.It showed Error 404 saying 'There is not a Github pages site here'. So, I want to know if there is something wrong about the link of documentation and how to get the documents about sciope.Thank You!

SMC ABC doesn't stop when max_rounds == 1

Proposed fixes

• RelativeEpsilonSelector constructor should raise an error if max_rounds == 0 and the doc string should indicate the required value.
• relative_epsilon_selector.py line 55: should return self.max_rounds == 1 for the has_more return value.
• smc_abc.py line 182: round should have an initial value of 2.
• relative_epsilon_selector.py line 81: should return self.max_rounds and round >= self.max_rounds for the terminate return value.

intro_core notebook

intro_core.ipynb not readable

Prevent infinite rejection sampling within SMC rounds

There should be an option to specify the maximum number of allowed rejection sampling iterations.

SMC ABC Inference doesn't support 1D parameter space

If this is an intended feature we should raise an error early.

Error

Statistics
Compute Environment: Local
2023-04-07 19:12:35,614 - GillesPy2 - ERROR - Job errors: tuple index out of range
Traceback (most recent call last):
File "/stochss/stochss/handlers/util/scripts/start_job.py", line 120, in
job.run(verbose=args.verbose)
File "/stochss/stochss/handlers/util/model_inference.py", line 943, in run
results = smc_abc_inference.infer(*infer_args, **infer_kwargs)
File "/opt/conda/lib/python3.8/site-packages/sciope/inference/smc_abc.py", line 210, in infer
abc_results = abc_instance.infer(num_samples=t,
File "/opt/conda/lib/python3.8/site-packages/sciope/inference/abc_inference.py", line 354, in infer
return self.rejection_sampling(num_samples, batch_size, chunk_size, ensemble_size, normalize)
File "/opt/conda/lib/python3.8/site-packages/sciope/inference/abc_inference.py", line 210, in rejection_sampling
for f, dist in as_completed(futures_dist, with_results=True):
File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 5199, in __next__
return self._get_and_raise()
File "/opt/conda/lib/python3.8/site-packages/distributed/client.py", line 5188, in _get_and_raise
raise exc.with_traceback(tb)
File "/opt/conda/lib/python3.8/site-packages/sciope/inference/smc_abc.py", line 72, in _weighted_draw_perturb
if self.ref_prior.pdf(sz) > 0:
File "/opt/conda/lib/python3.8/site-packages/sciope/utilities/priors/uniform_prior.py", line 82, in pdf
for i in range(z.shape[0]):
IndexError: tuple index out of range

Model

def create_michaelis_menten(parameter_values=None):
    # Initialize Model
    model = gillespy2.Model(name="Michaelis_Menten")

    # Define Variables (GillesPy2.Species)
    A = gillespy2.Species(name='Substrate', initial_value=301)
    B = gillespy2.Species(name='Enzyme', initial_value=120)
    C = gillespy2.Species(name='Enzyme_Substrate_Complex', initial_value=0)
    D = gillespy2.Species(name='Product', initial_value=0)
    
    # Add Variables to Model
    model.add_species([A, B, C, D])

    # Define Parameters
    rate1 = gillespy2.Parameter(name='rate1', expression=0.0017)
    rate2 = gillespy2.Parameter(name='rate2', expression=0.5)
    rate3 = gillespy2.Parameter(name='rate3', expression=0.1)
    
    # Add Parameters to Model
    model.add_parameter([rate1, rate2, rate3])

    # Define Reactions
    r1 = gillespy2.Reaction(
        name="r1", reactants={'Substrate': 1, 'Enzyme': 1}, products={'Enzyme_Substrate_Complex': 1}, rate='rate1'
    )
    r2 = gillespy2.Reaction(
        name="r2", reactants={'Enzyme_Substrate_Complex': 1}, products={'Substrate': 1, 'Enzyme': 1}, rate='rate2'
    )
    r3 = gillespy2.Reaction(
        name="r3", reactants={'Enzyme_Substrate_Complex': 1}, products={'Enzyme': 1, 'Product': 1}, rate='rate3'
    )
    
    # Add Reactions to Model
    model.add_reaction([r1, r2, r3])
    
    # Define Timespan
    tspan = gillespy2.TimeSpan.linspace(t=100, num_points=101)
    
    # Set Model Timespan
    model.timespan(tspan)
    return model

Inference

def configure_simulation(trajectories=1):
    solver = gillespy2.SSACSolver(model=model, delete_directory=False)
    kwargs = {
        'number_of_trajectories': trajectories,
        'seed': None,
        'solver': solver,
    }
    return kwargs

kwargs = configure_simulation()
results = model.run(**kwargs)

unshaped_obs_data = results.to_array()
shaped_obs_data = unshaped_obs_data.swapaxes(1, 2)
obs_data = shaped_obs_data[:,1:, :]
print(obs_data.shape)

def process(raw_results):
    return raw_results.to_array().swapaxes(1, 2)[:,1:, :]

def simulator(parameter_point):
    model = create_michaelis_menten()

    labels = [
        'rate1'
    ]
    for ndx, parameter in enumerate(parameter_point):
        model.listOfParameters[labels[ndx]].expression = str(parameter)

    kwargs = configure_simulation()
    raw_results = model.run(**kwargs)

    return process(raw_results)

values = numpy.array([
    0.0017
])
parameter_names = [
    'rate1'
]

lower_bounds = numpy.array([
    0.00085
])
upper_bounds = numpy.array([
    0.00255
])

uni_prior = uniform_prior.UniformPrior(lower_bounds, upper_bounds)

summ_func = auto_tsfresh.SummariesTSFRESH()

eps_selector = RelativeEpsilonSelector(20, max_rounds=2)

c = Client()

smc_abc_inference = smc_abc.SMCABC(
    obs_data, sim=simulator, prior_function=uni_prior, summaries_function=summ_func.compute
)

smc_abc_results = smc_abc_inference.infer(
    num_samples=100, batch_size=10, eps_selector=eps_selector, chunk_size=10
)

LinAlgError raised when running SMC ABC

Error

Traceback (most recent call last):
  File "/stochss/stochss/handlers/util/scripts/start_job.py", line 120, in <module>
    job.run(verbose=args.verbose)
  File "/stochss/stochss/handlers/util/model_inference.py", line 958, in run
    results = smc_abc_inference.infer(*infer_args, **infer_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/sciope/inference/smc_abc.py", line 220, in infer
    kweights = self.perturbation_kernel.pdf(population, new_samples)
  File "/opt/conda/lib/python3.8/site-packages/sciope/utilities/perturbationkernels/multivariate_normal.py", line 45, in pdf
    pdfs.append(multivariate_normal.pdf(x, x0[i], self.cov))
  File "/opt/conda/lib/python3.8/site-packages/scipy/stats/_multivariate.py", line 580, in pdf
    params = self._process_parameters(mean, cov, allow_singular)
  File "/opt/conda/lib/python3.8/site-packages/scipy/stats/_multivariate.py", line 417, in _process_parameters
    psd = _PSD(cov, allow_singular=allow_singular)
  File "/opt/conda/lib/python3.8/site-packages/scipy/stats/_multivariate.py", line 172, in __init__
    raise np.linalg.LinAlgError(msg)
numpy.linalg.LinAlgError: When `allow_singular is False`, the input matrix must be symmetric positive definite.

Model

def create_genetic_toggle_switch(parameter_values=None):
    model = gillespy2.Model(name='Genetic_Toggle_Switch')
    model.volume = 1

    # Variables
    U = gillespy2.Species(name='U', initial_value=10, mode='discrete')
    V = gillespy2.Species(name='V', initial_value=10, mode='discrete')
    model.add_species([
        U, V
    ])

    # Parameters
    alpha1 = gillespy2.Parameter(name='alpha1', expression='1')
    alpha2 = gillespy2.Parameter(name='alpha2', expression='1')
    beta = gillespy2.Parameter(name='beta', expression='2')
    gamma = gillespy2.Parameter(name='gamma', expression='2')
    mu = gillespy2.Parameter(name='mu', expression='1')
    model.add_parameter([
        alpha1, alpha2, beta, gamma, mu
    ])

    # Reactions
    cu = gillespy2.Reaction(
        name='cu',
        reactants={}, products={'U': 1},
        propensity_function='alpha1/(1+pow(V,beta))',
        ode_propensity_function='alpha1/(1+pow(V,beta))'
    )
    cv = gillespy2.Reaction(
        name='cv',
        reactants={}, products={'V': 1},
        propensity_function='alpha2/(1+pow(U,gamma))',
        ode_propensity_function='alpha2/(1+pow(U,gamma))'
    )
    du = gillespy2.Reaction(
        name='du', rate='mu',
        reactants={'U': 1}, products={}
    )
    dv = gillespy2.Reaction(
        name='dv', rate='mu',
        reactants={'V': 1}, products={}
    )
    model.add_reaction([
        cu, cv, du, dv
    ])

    # Timespan
    tspan = gillespy2.TimeSpan.arange(1, t=100)
    model.timespan(tspan)
    return model

Inference

def configure_simulation(trajectories=1):
    solver = gillespy2.SSACSolver(model=model, delete_directory=False)
    kwargs = {
        'number_of_trajectories': trajectories,
        'seed': None,
        'solver': solver,
    }
    return kwargs

kwargs = configure_simulation(100)
results = model.run(**kwargs)

unshaped_obs_data = results.to_array()
shaped_obs_data = unshaped_obs_data.swapaxes(1, 2)
obs_data = shaped_obs_data[:,1:, :]
print(obs_data.shape)

def process(raw_results):
    return raw_results.to_array().swapaxes(1, 2)[:,1:, :]

def simulator(parameter_point):
    model = create_genetic_toggle_switch()

    labels = [
        'alpha1', 'alpha2', 'beta', 'gamma', 'mu'
    ]
    for ndx, parameter in enumerate(parameter_point):
        model.listOfParameters[labels[ndx]].expression = str(parameter)

    kwargs = configure_simulation()
    raw_results = model.run(**kwargs)

    return process(raw_results)

values = numpy.array([
    1.0, 1.0, 2.0, 2.0, 1.0
])
parameter_names = [
    'alpha1', 'alpha2', 'beta', 'gamma', 'mu'
]

lower_bounds = numpy.array([
    0.5, 0.5, 1, 1, 0.5
])
upper_bounds = numpy.array([
    1.5, 1.5, 3, 3, 1.5
])

uni_prior = uniform_prior.UniformPrior(lower_bounds, upper_bounds)

summ_func = identity.Identity()

eps_selector = RelativeEpsilonSelector(20, max_rounds=1)

c = Client()

smc_abc_inference = smc_abc.SMCABC(
    obs_data, sim=simulator, prior_function=uni_prior, summaries_function=summ_func.compute
)

smc_abc_results = smc_abc_inference.infer(
    num_samples=10, batch_size=10, eps_selector=eps_selector, chunk_size=10
)