Original report by Sy-Miin Chow (Bitbucket: symiin, GitHub: symiin).

Prep.regimes doesn't seem to be working correctly with covariates.

I specified this model where the covariates x1 and x2 are used to predict the LO parameters associated with the (1,1) and (2,1) elements of the transition probability matrix. When I printed out the specified model, I got
Slot "covariates":
character(0)

When optimizing the full model (RS-linearODE), the parameters for the covariates don't seem to be updated at all. Looking at prep.regimes, I also didn't see how the covariates were actually used inside prep.regimes.

#!R
# Regime-switching function
# The RS model assumes that each element of the transition probability 
# matrix (TPM) can be expressed as a linear predictor (lp).
# LPM = 
# lp(p11) ~ 1 + x1 + x2 + ... + xn,   lp(p12) ~ 1 + x1 + x2 + ... + xn
# lp(p21) ~ 1 + x1 + x2 + ... + xn,   lp(p22) ~ 1 + x1 + x2 + ... + xn
# Here I am specifying lp(p11) and lp(p21); the remaining elements
# lp(p12) and lp(p22) are fixed at zero.

regimes <- prep.regimes(
  values=matrix(c(6,.5,-.3,rep(0,3),
                  -3,-1.5,-1,rep(0,3)), 
                nrow=2, ncol=6,byrow=T), # nrow=numRegimes, ncol=numRegimes*(numCovariates+1)
  params=matrix(c("a_{11}","d_{11,1}","d_{11,2}",rep("fixed",3),
                  "a_{21}","d_{21,1}","d_{21,2}",rep("fixed",3),), 
                nrow=2, ncol=6,byrow=T), covariates=c('x1', 'x2'))

Original report by Sy-Miin Chow (Bitbucket: symiin, GitHub: symiin).

I pushed a "partially working" RSLinearODE.r" to github. In implementing these recipe functions. The following issues need to be fixed. prep.tfun doesn't seem to be working correctly (it ran fine but cat(writeCcode(trans)$c.string) generated the error we saw earlier)

#1. Suppress warnings about "In matrix(as.numeric(x), numRow, numCol) : NAs introduced by coercion"

#28. Add intercepts and allow for regime-dependent intercepts and Lambda

#29. prep.matrixDynamics should be allowed to be regime-dependent

#34. Allow regime-dependent covariate functions in prep.formulaDynamics (may take more time - do we want to do this?)

#31. Allow regime-dependent covariate effects in prep.matrixDynamics covered in #29

#32. Allow regime-dependent covariate effects in prep.loadings covered in #28

33. dynr.model doesn't run yet with regime-switching models. The order in which the modeling functions are compiled needs reorganizing. Lu is fixing this.

#27. Can we set params = NULL as the default? Weird to give parameter names to fixed parameters.

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

start(), ub(), lb(), etc

Original report by Sy-Miin Chow (Bitbucket: symiin, GitHub: symiin).

dyn=printex(dynm)
generated the error:

Error in paste0("\(", mulpatn, "\)/", sigpatn) :
object 'sigpatn' not found

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Or rather treat a missing params argument as a matrix of 0s of the appropriate size, i.e. all fixed parameters.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Write alternative function that does not use KFAS::ldl

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

They should be variations on

#!R
prep.measurement()
prep.dynamics() # also contains Jacobian information
prep.regimes()
prep.noise()
prep.initial()

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

e.g. make nightly

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Use the R package roxygen2 to automatically build some documentation files.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

As per the 2016-04-01 meeting, what is currently called dynr.run should be renamed dynr.cook.

It should have arguments model, data, and control.

#!R

dynr.cook <- function(model, data, control){
#   ...
}

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

See this code from Sy-Miin

#!R
#If you are uncomfortable with differentiation, R has symbolic differentiation
#functions you can use
D(expression(a*x+b), 'x')

#Run this function - gives you a way to obtain symbolic differentiation of any order
DD <- function(expr, name, order = 1) {
  if(order < 1) stop("'order' must be >= 1")
  if(order == 1) D(expr, name)
  else DD(D(expr, name), name, order - 1)
}

DD(expression(a*x+b), "x", 2) #second derivative

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Make sure that Lu gets email notifications for issues.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Like the loadings and dynamics, the initial state and initial covariance should be allowed to be regime-switching.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Move them to DESCRIPTION.in

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

#!R
matrix(c(x~y, z~y), 2, 1)
#     [,1]      
#[1,] Expression
#[2,] Expression

# Should look like
matrix(c(x~y, z~y), 2, 1)
#     [,1]      
#[1,] x~y
#[2,] z~y

# Use something like
formula2string <- function(formula){
    if(is.list(formula) && length(formula) == 1){
        formula <- formula[[1]]
    }
    Reduce(paste, deparse(formula))
}
mat <- matrix(c(x~y, z~y), 2, 1)
apply(mat, c(1, 2), formula2string)
#     [,1]   
#[1,] "x ~ y"
#[2,] "z ~ y"

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Number 5: In one-regime model, some users find the empty call dynr.regime()
confusing. Better to not have to do it at all. Perhaps make one-regime model
default.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Users cannot compile on the fly functions that don't exist locally.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

The loadings should be regime dependent, but I need to add intercepts that are regime dependent.

Also, add covariates and make them regime-dependent.

Lastly, test these.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Use environment variable $GSL_LIB instead of the local 64-bit GSL in dynr upon installation.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

A model is a collection of recipes, one of each type. E.g.

#!R

setClass(Class =  "dynrModel",
         representation = representation(
           dynamics =  "dynrDynamics",
           measurement = "dynrMeasurement",
           initial = "dynrInitial",
          # ...
         )
)

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Let the transition probability matrix operate on the row-space instead of the column-space

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

There's a protect stack imbalance upon returning to R. Likely culprits are the most recent commits here: https://bitbucket.org/mhunter/dynr/history-node/2710418a907a/src/mainR.c?at=master

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Something in how dynrTrans is handled kills the optimization/Hessian of dynr/R/demo/LinearSDE.R

Original report by Sy-Miin Chow (Bitbucket: symiin, GitHub: symiin).

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

In case a users tries to say values=matrix('fixed', 1, 1), throw a friendly error message.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Number 13: Print method to make things look pretty in LaTeX.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

# Complete the printex functions and allow them to be regime-specific equations

#36.1. Implement the printex method for formula specifications. Allow using latex symbols in parameter names "\eta_{i}". - Assigned to Lu

# 36.2. Consolidate dynamic formula and matrix specifications so it outputs univariate system of R functions. - Lu, but in coordination with Mike

#36.3. Plug in starting and estimated values in the pretex-ed equations - Assigned to Lu

Think we may only need to output latex equations showing the parameter names. Equations with starting and estimated values plugged in may be better formulated as R expressions to be plotted in R. Open to discussion.

# 36.4. Implement the printex method for measurement model - Mike

# 36.5. Implement the printex method for the remaining recipes (initial, noise) - Mike and/or Lu, depending on time and availability

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

Updated on 3/31 New issue:

res <- dynr.run(model, data)

Error in .Call("main_R", model, data, debug_flag, outall_flag, PACKAGE = "dynr") :
"main_R" not available for .Call() for package "dynr"

With the new version of dynr installed and loaded, a bunch of errors of "could not find function XXX" occur. I had the problem with all demo files on my lab mac. See below (Sorry I can't upload any attachment). It seems that R was not looking into the namespace dynr. Are all the functions explicitly exported? I think I know where the problem is now. On the machine, the "make clean install" modifies the NAMESPACE. See below.

****roxygen2 has been updated to version 5.x.x on CRAN. Make sure you grab the newest version of roxygen2 in testing. My laptop (mac) with roxygen2 version 4.x.x worked fine.

For a function to be usable outside of your package, you must export it. By default roxygen2 doesn’t export anything from your package. If you want an object to be publically available, you must explicitly tag it with "export".

In terminal,

After git pull --rebase

$ cat NAMESPACE

exportPattern("^[[:alpha:]]+")

exportPattern("^[^\\.]")

useDynLib(dynr)

After make clean install

$ cat NAMESPACE

Generated by roxygen2: do not edit by hand

In R

getAnywhere(dynr.data)

A single object matching ‘dynr.data’ was found
It was found in the following places
namespace:dynr
with value

dynr:::dynr.data()

Error in unique(dataframe[, id]) :
argument "dataframe" is missing, with no default

dynr.data()

Error: could not find function "dynr.data"

library(dynr)

dynr.data()

Error: could not find function "dynr.data"

dynr::dynr.data()

Error: 'dynr.data' is not an exported object from 'namespace:dynr'

Original report by Sy-Miin Chow (Bitbucket: symiin, GitHub: symiin).

Need to start building roxygen comments. Sukruth, if dynr.model is too daunting, I would suggest that you start with prep.regimes. I added some text and comments but didn't bother "beautifying" the layout. Please look into the following website to learn how to use more sophisticated commands to make the roxygen comments look nice.

https://cran.r-project.org/web/packages/roxygen2/vignettes/rd.html

To all: it may be difficult for Sukruth to write roxygen contents from scratch given that he doesn't know the underlying algorithms and set-up of the recipes well. I would suggest that we add basic text as we go - and when you feel that the help comments are more-or-less complete, tell Sukruth and let him do the beautifying.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Number 4: Values.inistate = c(“Freed”, 1) (value 1 is fixed) ; params.inistate = c(5,
“Fixed”). Better distinguish between the label for a parameter (Freed vs.
fixed) and possible starting value.

Original report by Sy-Miin Chow (Bitbucket: symiin, GitHub: symiin).

I have been working with Linying and Meng to set up another examples that uses dynr (a RS linear SDE) and we came across the following issues that need to be fixed/tightened:

Putting down Mike as the assignee for now but can discuss delegation later.

1. Dynamic formula now only works with prep.loadings because it needs latent variable names. But prep.measurement has the latest regime-specific functionality. We should only keep one, not both given that we always just allow for linear normal measurement model. All of us universally prefer prep.measurement to prep.loadings. But latent variable names need to be added as a slot to this function.

2. The dynamic formula and jacobian C translation functions often have false cases of mistaking parameter names as latent variables because of the use of approximate matching. For instance, if my latent variables are called mom and baby, and my parameters contain something like eta_mom or eta_{mom} or etamom, these functions mistake part of these parameter names as latent variables and mess up the automatic code translation, despite having eta_{mom} etc. already provided as parameter names. Need better ways of handling this - perhaps enabling only exact matching.

3. The help documentation for prep.loadings is hard to understand. Other help docs are generally hard to understand.

4. If we are keeping prep.loadings, do we allow for idvar=NULL (no loadings being fixed for identification purposes).

5. linearSDE.r needs to be updated later when all functions have been finalized (e.g., it currently doesn't use parameter names; specify input argument names explicitly; tfun needs to be replaced with prep.tfun).

6. Check to see if prep.tfun works with prep.matrixDynamics

7. Updates to installation instructions:
   a. Add xtable to the list of required packages
   b. 2.4.1 Hello.r not provided
   c. section 2.4, point 5. Move "You may..." to the beginning of the point.
   d. section 2.2 point 5. Clarify where else users are allowed to install Rtools.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

dynr.model should do what dynr.prep currently does.

#!R
dynr.model <- function(outfile=tempfile(), ...,infile){
	#num_regime=1, dim_latent_var, isContinuousTime=TRUE
	# can be extracted from inputs, i.e. dynrRecipe objects
	
	# xstart, ub, lb, options=default.model.options,
	# compileLib=TRUE, verbose=TRUE
	# can be extracted from the dynrControl object

	inputs <- list(...)
	cparts <- sapply(inputs, slot, name='c.string')
	includes <- "#include <math.h>\n#include <gsl/gsl_matrix.h>\n#include <gsl/gsl_blas.h>\n"
	body <- paste(cparts, collapse="\n\n")
	if( length(grep("void function_regime_switch", body)) == 0 ){ # if regime-switching function isn't provided, fill in 1 regime model
		body <- paste(body, prep.regimes()$c.string, sep="\n\n")
	}
	glom <- paste(includes, body, prep.dP_dt, .cfunctions, sep="\n\n")
	cat(glom, file=outfile)

	# From dynr.model
	#returns a list of addresses of the compiled model functions
	func_address=dynr.funcaddress(isContinuousTime=isContinuousTime,infile=infile,outfile=outfile,verbose=verbose, compileLib=compileLib)
	#
	plist <- list(num_regime=as.integer(num_regime), dim_latent_var=as.integer(dim_latent_var), xstart=xstart, ub=ub, lb=lb, isContinuousTime=isContinuousTime, num_func_param=as.integer(length(xstart)),func_address=func_address, options=options)
	return(new("dynrModel", plist))
}

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Add a check in the frontend of dynr.run to verify that all function addresses are non-null before giving them to the backend.

#!R

if(any(is.null.pointer(addresses))){
    stop('Helpful message')
}

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

1. First attempt

Take the input of the functions (i.e., params and values matrix) below, create something that looks like what is in the attachment.

#!R
# define the differential equation
dynamics <- dynr.linearDynamics(
  params.dyn=matrix(c(0, 1, 0, 2), nrow=2, ncol=2),
  values.dyn=matrix(c(0, 0, 1, 0), nrow=2, ncol=2),
  time="continuous")
# Uses parameters 1 (spring constant) and 2 (damping)

# measurement
meas <- dynr.matrixLoadings(
  values=matrix(c(1, 0), nrow=1, ncol=2),
  params=matrix(c(0, 0), nrow=1, ncol=2))
# No free parameters

# observation and dynamic noise components
ecov <- dynr.matrixErrorCov(
  values.latent=diag(c(1e-5, 1)), params.latent=diag(c(0, 3)),
  values.observed=diag(1.5, nrow=1), params.observed=diag(4, nrow=1))
# Uses parameters 3 and 4

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Similar to the loadings.

Regime-dependent dynamics, covariates, and intercepts.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Number 2: For the zeta (process noise) covariance matrix, have users provide “0” as
opposed to “1e-5” then we do behind-the-scene conversion of 0 into a small
constant.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

We have lots of new functionality and little changes could break it. We should be testing these features.

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

Now fixed parameters are labelled as "0". Can they be changed to something like "fixed"?
Should we still give user the option to specify param numbers?

Original report by Lu Ou (Bitbucket: luou, GitHub: luou).

replace all covariate names by gsl_vector_get(co_variate,x) in the c.string.
then write it out to the c script.

Original report by Sy-Miin Chow (Bitbucket: symiin, GitHub: symiin).

The innovation vector and inverse covariance matrix output right now are regime-specific. Can't use them for diagnostic purposes across all regimes. Need to compute these guys as weighted across regimes.

Original report by Michael Hunter (Bitbucket: mhunter, GitHub: mhunter).

Remove unneeded (non .R non.c) files from demo directory

Original report by Sy-Miin Chow (Bitbucket: symiin, GitHub: symiin).

No param.names slot is assigned to dynrDebug or dynrOutall object, so they don't work with the summary function or other post-processing functions. Two issues to resolve/think about:

1. How to handle the above issue. Either we make sure all that all slots that will get used in post-processing also get assigned to dynrDebug and dynrOutall, or we find a way to store only the unique slots in dynrDebug and return both the dynrCook and dynrDebug objects (with only the unique elements) as a list.

2. Also unclear to me why we need both dynrDebug and dynrOutall. Can we not just keep one?

Please let me know your suggestion on the preferred way to do this given that you guys know overall flow of the post-processing aspects of dynr better.