The nomad from stan-dev

Manual Draft

Finish up a first draft of the manual,

Automatic Differentiation
Implementation
Using Nomad
Nomad Reference Guide

Should we have a separate development team or just go with the Stan development team for attribution?

Templated Function Signatures and Vectorization

One of the nice things about the new autodiff system is that it incorporates operands and partials directly, which will make implementing functions and implementing distributions identical.

An immediate feature is the ability to construct all possible function signatures like Stan does for the distributions. For example, a binary function taking in two autodiff vars would be written as

template <short autodiff_order>
inline var<autodiff_order> binary_function(const var<autodiff_order>& v1,
                                           const var<autodiff_order>& v2) {

  const short partials_order = 3;
  const unsigned int n_inputs = 2;

  next_inputs_delta = n_inputs;
  next_partials_delta =
    var_body<autodiff_order, partials_order>::n_partials(n_inputs);

  new var_body<autodiff_order, partials_order>(n_inputs);

  double x = v1.first_val();
  double y = v2.first_val();

  push_dual_numbers<autodiff_order>(binary_function(x, y));

  push_inputs(v1.dual_numbers());
  push_inputs(v2.dual_numbers());

  if (autodiff_order >= 1) {
    push_partials(df_dx);
    push_partials(df_dy);
  }
  if (autodiff_order >= 2) {
    push_partials(df2_dx2);
    push_partials(df2_dxdy);
    push_partials(df2_dy2);
  }
  if (autodiff_order >= 3) {
    push_partials(df3_dx3);
    push_partials(df3_dx2dy);
    push_partials(df3_dxdy2);
    push_partials(df3_dy3);
  }
  return var<autodiff_order>(next_body_idx_ - 1);
}

and we'd have to add separate implementations for the (double, var) and (var, double) signatures. But we should be able to template the arguments out and do something like

template <short autodiff_order>
inline var<autodiff_order> binary_function(const T1& v1,
                                           const T2& v2) {

  const short partials_order = 3;
  const unsigned int n_inputs = T1.is_var() + T2.is_var();

  next_inputs_delta = n_inputs;
  next_partials_delta =
    var_body<autodiff_order, partials_order>::n_partials(n_inputs);

  new var_body<autodiff_order, partials_order>(n_inputs);

  double x = v1.first_val();
  double y = v2.first_val();

  push_dual_numbers<autodiff_order>(binary_function(x, y));

  push_inputs(v1.dual_numbers());
  push_inputs(v2.dual_numbers());

  if (autodiff_order >= 1) {
    if (T1.is_var()) push_partials(df_dx);
    if (T2.is_var()) push_partials(df_dy);
  }
  if (autodiff_order >= 2) {
    if (T1.is_var()) push_partials(df2_dx2);
    if (T1.is_var() && T2.is_var()) push_partials(df2_dxdy);
    if (T2.is_var()) push_partials(df2_dy2);
  }
  if (autodiff_order >= 3) {
    if (T1.is_var()) push_partials(df3_dx3);
    if (T1.is_var() && T2.is_var()) push_partials(df3_dx2dy);
    if (T1.is_var() && T2.is_var()) push_partials(df3_dxdy2);
    if (T2.is_var()) push_partials(df3_dy3);
  }
  return var<autodiff_order>(next_body_idx_ - 1);
}

Of course we'd have to be careful regarding some optimizations (especially if we want to use specialized var_body implementations).

Vectorization should work the same way, too, if we had a n_vars method in a template array wrapper.

Thoughts?

Discontinuous Functions

What should we do with functions without proper derivatives? Stan includes them and ignores the discontinuities, but this does lead to unintended problems when people try to autodiff through them.

http://www.cplusplus.com/reference/cmath/

Comparison operators are very bad for autodiff because they admit the construction of discontinuous functions. Equals operators are particularly bad because they can't even be tested with finite differences, as the perturbations cause the comparison to fail. Not sure how these operators would be tested.

src/scalar/operators/nonsmooth_operators/operator_equal_to
src/scalar/operators/nonsmooth_operators/operator_not_equal_to
src/scalar/operators/nonsmooth_operators/operator_unary_not

Implement Eigen Operations

Right now we have an efficient matrix multiplication operation that admits expression template arguments but it only works for functions and not for operator overloading. Investigate this issue and see if we can have proper operator overloading.

var_body.hpp: static int n_partials() doesn't have a default return

There's no default return for this function.

Porting nomad and autodiff to ArrayFire

Does anyone know how hard it is to port to ArrayFire? ArrayFire has OpenCL, CUDA and CPU backends and many nice language bindings as well.

Implement Remaining Operators

Domain Errors

What should we do with domain errors? The easiest solution would be to use std::nan() for the values and gradients, but then errors could not be traced back to their source. Alternatively, we can throw domain exceptions and be careful about wrapping the autodiff methods in try/catch blocks.

Functions with constrained domain:

To Do:

Select and implement error strategy.
Domain tests following finite difference functor pattern.

Partial specializations of unary and binary var bodies

Partially specialize unary and binary var bodies for partial_order = 1, partial_order = 2, to avoid storing unnecessary higher-order partials.

Implement Polygamma

Need the polygamma function in order to compute derivatives of the gamma function and its derivatives. Should be able to produce a reasonable implementation by leveraging the generic recursion relation to transform the domain to (0, 1) and expand the polygamma around 0. Bonus points for going the other way and using the asymptotic expansion for large arguments.

With a polygamma implementation we can then implement the last cmath functions,
http://www.cplusplus.com/reference/cmath/

tgamma
lgamma

Implement Special Functions

Miscellaneous special functions that have been implemented in Stan, some requiring the Boost libraries and some requiring the polygamma functions #3.

stan-dev / nomad Goto Github PK

nomad's People

Contributors

Stargazers

Watchers

Forkers

nomad's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs