Team discussion of next steps about template

[lboxell]

@arosenbe @Shun-Yang @stanfordquan @yuchuan2016, See above. @gentzkow @jmshapir: That works for me if it works for everyone else.

…

--- On Aug 1, 2017, at 5:38 AM, jmshapir <[email protected]<mailto:[email protected]>> wrote: i could probably do that — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub<#56 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AMbyaSJK2DavaonN78_5X9K7XiPoflvEks5sTxxFgaJpZM4Ok9o_>.

[lboxell]

Collecting my comments here:

• Make builders fail explicitly when targets are not build. See #55.
  • I don't think this used to not be an issue when we always ran the 'state_of_repo' logger as the final step in the build process with all previous steps as dependencies. In this scenario, any failed target build would cause scons to tell you it failed. With the 'atexit' function, we no longer get this safeguard. There are positives and negatives to both approaches.
• Consider adding code to reconfigure cache. See gslab-econ/gslab_python#71.
• Store python requirements for each template/repo, including gslab_tools, in a virtual environment. [As suggested by Toby]
  • If we do this, we should consider putting gslab_tools back on pypi. Because its more complex to download from a github repository (possibly requiring a separate step) instead of using straight pip.
  • Look into similar options for R.
• Fix naming scheme in gslab_scons debrief function and use in template. See #52 (comment) and https://github.com/gslab-econ/gslab_python/blob/dev_four/gslab_scons/misc.py#L14. Avoid use of 'env' and 'target' unless these are being used by SCons in a build step.
• Determine whether we want to use SCons env.PDF for latex or build our own builder.
• Investigate YAML tools for matlab.
  • I used https://code.google.com/archive/p/yamlmatlab/ in https://github.com/gslab-econ/divergence/pull/17 and it seemed to work well. Currently, I have it as a 'lib' in the divergence directory, but we may want to add to 'gslab_matlab'.
• Rethink use of logs and whether we (a) track them as targets and (b) version control them.
  • If you don't do (a), but do (b), then when you pull from cache, it looks like you 'delete' the log files from git's perspective. Should probably do both (a) and (b) or neither.
• Determine whether we want to have a standard set of integration tests we run. Currently, we just run "unit" tests that are platform independent and don't actually interact with R, Stata, etc.

[qlquanle]

Collecting my comments here.

• Figure out a controlled environment for the general purpose Command builder. My instinct is to build a Command wrapper than runs whatever the user input as command, but use the logging machinery instead of the current free for all. That way we get timestamps.

[lboxell]

@RoyalTS,

Feel free to add anything you've found confusing or that you think could be better implemented as you've played around with our template.

[RoyalTS]

Suggestions/comments/pointers from my side:

• I like the idea of making almost everything optional
• provide bash scripts to install all external dependencies (brew install ...
• I would switch to Rscript or littlr as the R command line interpreter. R CMD BATCH is outdated (https://stackoverflow.com/questions/21969145/why-or-when-is-rscript-or-littler-better-than-r-cmd-batch)
• the Python environment should be more tightly controlled. conda is the neatest approach to this but requires anaconda to be installed on the system. In its absence, virtualenvs and freezing them via pip may be the best option
• Have yet to find a truly great solution for more tightly controlling R packages and versions. packrat, pacman and checkpoint packages might all be options. No obvious best one among them.
• if you did want to create your own LaTeX builder, writing a thin wrapper around latexmk would probably get you pretty far
• and yes, a build failing when the targets didn't build seems like an absolutely essential feature of a build system

[lboxell]

@gentzkow,

Is the call something you were wanting to do once you get back from France? Or were you wanting to do it sooner?

[gentzkow]

Let's do it sooner. Could you guys do 9:00 PT / 18:00 france time today?

[jmshapir]

i could probably do that

[lboxell]

Closing task.

Summary:

• We discussed the above points in a call today.
• Created https://github.com/gslab-econ/admin/issues/115 to summarize the call and create the next steps.