bredelings / mcon Goto Github PK

For TSV files, it seems common enough to add comment lines starting with # and perhaps [.
MCON doesn't allow these.
We can make a "comment": <string> field in the header.
Should there be any convention for how this is structured?

We can always separate JSON values by a tab character, since JSON never contains unescaped tabs. Lets call such a format "TSJ".

However, this isn't necessarily valid TSV -- a JSON string would need to be mangled by TSV-quoting in order to be valid TSV.

How should we handle this?

Packages to modify:

• RevBayes - https://github.com/revbayes/revbayes
• Tracer - https://github.com/beast-dev/tracer
  • TracerFrame, Skyline, Extended Skyline, Skyride, LTT, TTT,
• R packages
  • coda - https://github.com/cran/coda
  • posterior
  • mcmcse
• bayesplot - https://github.com/stan-dev/bayesplot

MCMC diagnostics for some objects may depend on what the object means. Similarly, it may be possible to perform visualizations for objects of a certain type.

Examples:

• suppose we approximate a function on [0,1] using a piecewise-linear form with an unknown number of pieces. We should be able to generate posterior credible intervals for {f(x)| x in [0,1]}.
• the JSON value for variable N might describe a population size history through time for a coalescent model.
• the JSON value could describe a discrete distribution with a weighted list of values. If the number of values is constant, we want to sort by both values and weight by the values. If the number of values is not constant, we can still compute the median, variance, etc.

In order to represent more complex objects than arrays and objects, we introduce a special notation.

If a field value contains the keys @$record and @$value then we consider it to represent a record type.
The value for the key @$value must be an object, and its keys represent the fields for that object.

Thus if we have::

    "rates": {"@$record": "DiscreteDistribution",
              "@$value":  {"weights": [0.2, 0.3, 0.5],
                           "values": [0.2, 1.1, 3.4] } }

Then we consider this to represent a record shape DiscreteDistribution with fields weights and values.
In order to multiple record shapes to be part of the same data type, we allow an additional key @$type.
In languages like C++ or Java, the record shape would be considered a type.
However, in languages with algebraic data types (such as Rust), a data type can include multiple record shapes.

The purpose of this feature is to indicate the meaning of the values in each Monte Carlo sample so that appropriate summary measures can be computed.

Parquet - https://parquet.apache.org/

• Stan is talking about addout Parquet output.

Arrow - https://arrow.apache.org/

• See here for Stan branches using Arrow https://discourse.mc-stan.org/t/usage-of-arrow-with-stan/29835

HDF5 - https://www.hdfgroup.org/solutions/hdf5/

CODA format? Produced by OpenBugs and JAGS.

JSON doesn't allow for Inf, -Inf, or NaN.

Python's implementation of JSON allows reading (Infinity,-Infinity,NaN) which is how javascript represents these values but is not valid according to the JSON spec.

Boost.JSON writes (1e99999, -1e99999, null) by default, but allows writing (Infinity,-Infinity,NaN). It can read both versions.

Some packages (i.e. nlohmann:json) write null for these values.

See boostorg/json#397

Tracer reads TSV, but other software reads or writes CSV.

Q1. How should we flag keys like generation number and related indices in the header field?

For MCMC, the "iteration" or "generation" field is different than other fields. It is always increasing, and is meta-data about the sample, not part of the sample itself. Currently we can arrange to place that field first when converting to TSV, but it is otherwise not identified in the header.

If there are multiple MCMC chains, we could combine the samples by marking the sample with which chain it comes from.

For SMC, the "generation" field might indicate which distribution the sample is from, and we'd also need a "particle" number to distinguish samples.

Q2. We could have additional meta-information that is continuous. For example, with multiple heated chains, we could annotate each chain with a temperature.

Currently, MCON can be written by:

• bali-phy 4.0 beta
• revbayes, with PR revbayes/revbayes#377

To add:

• BayesCode https://github.com/bayesiancook/bayescode
• PhyloBayes https://github.com/bayesiancook/pbmpi
• BEAST - https://github.com/beast-dev/beast-mcmc/
• BEAST2 - https://github.com/CompEvol/beast2
• JAGS - https://sourceforge.net/p/mcmc-jags/code-0/ci/default/tree/
• STAN - https://github.com/stan-dev
• NIMBLE - https://r-nimble.org/writing-reversible-jump-mcmc-in-nimble