revolutionanalytics / checkpoint Goto Github PK

This feature is apparently in packrat as well as GRAN.

Seems that packrat only snapshots the packages used in the repo, that is, the packrat/ dir in the user's repo. I think GRAN snapshots the whole repo, including the users files, etc.

This is basically what git does really well, so would be nice to just use git, but can we do something similar without the user having to have git? Ideas:

• We could store the users files and their rrt_manifest.yml file in a "versions" folder somewhere in the repo, named with repoid + version number, like 1944a6826bee36f5925f1d13f11c6dc6_v1. Then a user can switch between versions with a single function. I think this can easily be done, but we'd want to probably make a user go through a prompt to make sure they actually want to overwrite files.

A note on terminology: since we have the notion of "snapshots" in marmoset/mran, if we have the notion of versions of a user's RRT repositories on their machine, we should use a different term, perhaps "version"?

When users do this when they have started R in their RRT repo, the package installs in the library in the repo (this is good), but the source is just installed in a temporary dir and deleted at end of session. Perhaps look for these installed packages without source files whenever rrt_install or rrt_refresh are run, and download the sources of the approprate version

create method to step through /src/contrib/Archive/ folders

include in /MRAN/RRT directory/filesystem

• snapshots
• diffs
• pkg level metadata

If you try and run rrt_install on a repository that includes any folders OTHER THAN rrt or .git, it will fail.

> rrt_install()
Checking to make sure repository exists...
Checing to make sure rrt directory exists inside your repository...
Error in rrt_install() : rrt directory doesn't exist

The reason in this case is that my repository includes the .Rproj.user folder created by RStudio:

> repo=getwd()
> file.path(repo, "rrt", "lib", R.version$platform,
+ base::getRversion())
[1] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1.0"
> present <- list.dirs(repo)[-1]
> present
 [1] "/Users/david/RRT-example2/.Rproj.user"                                          
 [2] "/Users/david/RRT-example2/.Rproj.user/9CD34B73"                                 
 [3] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/ctx"                             
 [4] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/presentation"                    
 [5] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/sdb"                             
 [6] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/sdb/prop"                        
 [7] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/sdb/s-3B2F7D3D"                  
 [8] "/Users/david/RRT-example2/rrt"                                                  
 [9] "/Users/david/RRT-example2/rrt/lib"                                              
[10] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0"                    
[11] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1"                
[12] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1.0"              
[13] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1.0/src"          
[14] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1.0/src/contrib"  
[15] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate"     
[16] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate/help"
[17] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate/html"
[18] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate/Meta"
[19] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate/R"   
[20] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio"        
[21] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio/help"   
[22] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio/html"   
[23] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio/Meta"   
[24] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio/R"      

Which causes the following test in rrt_install to fail:

    if (!all(grepl("rrt", present))) {
        stop("rrt directory doesn't exist")
    }

Right now, we print progress for package downloads and installs.

Get rid of these and give much more compact messages to users.

@revodavid Do you want the first version to be v0.1 or v1.0 ?

provide login details to Scott

Probably makes sense to install binaries elsewhere, like bin/... something, then make sure to look for those there

If build fails in some cases, then not such a helpful vignette.

1. rrt_init()
2. Create some .R files, or .Rmd, etc. with some code, including some libraries
3. Then try rrt_refresh() - to look for new packages that will be downloaded
4. Try rrt_install() - to install packages in the repository
5. ...

• Send zfs list -t snapshot output to a text file that lives under marmoset server $WWW_HOME
• Make sure that it is updated after every snapshot using crontab

from discussion in #2

we need a field for snapshot ID too. I can add this to the programmatic manifest file changes when a user decides to use MRAN - when they do we can write a field called MRAN_ID or something like that

I'm familiar with Travis-CI, but is there a preference for Jenkins or something else? @cmosetick

There is quite good support for R packages now on Travis-CI (if we go the travis route), so perhaps structuring this repo as a package, even if not essential, would give us easy way to get CI.

Right now we have DateCreated but that actually updates to latest date it was modified, so have both DateCreated and DateModified or DateUpdated.

Also, signify time zone in dates.

related to #13

in the short term at least...Employ ability to locally test for package compatibility

Is there anything we can use from GRAN in RRT?

If any files present, find pkgs and write this file, if no files present simply write a blank file.

we can use https://github.com/ropensci/git2r if we want an R native git binding

Do we want to do this? If so, right away, or for later?

Using rsnapshot we can allow users to get snapshots of the marmoset repository. Wrap in RRT...

Currently testing on Windows. Seems to be a few bugs in how paths are found and used, looking at now.

Thoughts on this?

Proposed layout of a RRT repository

myrepo - |
         |- manifest.txt # user specified list of pkgs
         |- code.R # any number of user supplied files
         |- rrt # this dir created by rr_init()
            |- lib # holds packages in pkgs/
                |- pkgs - |
                          |-doMC
                          |-plyr
            |- manifest.lock # RRT generated metadata file

configure a rsync daemon on marmoset server so RRT can utilize the rsync protocol for file transfers to users systems.

Things to consider doing on the server

These tests should be more robust than those on CRAN obviously.

• r cmd check
• incorporate checks already done on CRAN?
• force run examples
• force run tests
• force run vignettes
• check downstream pkgs install and check fine
• check reverse dependencies
• as a separate longer term thing could test package x with all versions of its dependencies, not just current versions, so if user wants to use ver 1.2 of pkg y, we know id its compatible or not with pkg x

We talked about this on the call...While we can get dependencies, we don''t know if the packages actually work together. There is some filtering for this given the CRAN package submission process (but it's not strictly enforced)

Perhaps: A single function (and helpers) to test that packages are compatible with one another in the project, either

• locally, or
• on a server?

The problem with the local testing is that with a large number of packages this could take a long time, but will explore ways to get the time down.

Running R from an initialized repository where the only file (myscript.R) has the following content:

## Example script using packages
require(ggplot2)
require(plyr)
require(data.table)
require(knitr)
require(lattice)

print(sessionInfo())

I run rrt_refresh:

> rrt_refresh()
Warning messages:
1: In packageDescription(y, encoding = NA) : no package 'hexbin' was found
2: In packageDescription(y, encoding = NA) : no package 'Hmisc' was found
3: In packageDescription(y, encoding = NA) : no package 'mapproj' was found
4: In packageDescription(y, encoding = NA) : no package 'maps' was found
5: In packageDescription(y, encoding = NA) : no package 'maptools' was found
6: In packageDescription(y, encoding = NA) : no package 'multcomp' was found
7: In packageDescription(y, encoding = NA) : no package 'quantreg' was found
8: In packageDescription(y, encoding = NA) : no package 'abind' was found
9: In packageDescription(y, encoding = NA) : no package 'doMC' was found
10: In packageDescription(y, encoding = NA) : no package 'foreach' was found
11: In packageDescription(y, encoding = NA) : no package 'iterators' was found
12: In packageDescription(y, encoding = NA) : no package 'itertools' was found
13: In packageDescription(y, encoding = NA) : no package 'bit64' was found
14: In packageDescription(y, encoding = NA) : no package 'chron' was found
15: In packageDescription(y, encoding = NA) : no package 'data.table' was found
16: In packageDescription(y, encoding = NA) : no package 'fastmatch' was found
17: In packageDescription(y, encoding = NA) : no package 'reshape' was found
18: In packageDescription(y, encoding = NA) : no package 'xts' was found
19: In packageDescription(y, encoding = NA) : no package 'rgl' was found
20: In packageDescription(y, encoding = NA) : no package 'testit' was found

A number of options here to start off with:

• miniCRAN
• dependencies, by Gavin Simpson
• packrat, by RStudio

I'll have a peak around each of them and figure out what else is needed. Probably better now to not have too many dependencies, so new code I think is in order.

See gRAN for reference: https://raw.githubusercontent.com/gmbecker/gRAN/master/vignettes/ReportScreenshot.png

Include:

• results of local compat checks
• do any pkgs have available updates

• What to use for a manifest? Options include .R, .md, .txt, .json, .yml
• What to use for the markup? This is linked to the item above of course since we wouldn't do e.g., yaml in a .R file. Could have no markup and simply a list of packages like

plyr
ggplot2
...

or could use yaml markup as in a Makefile. I like the idea of using yaml as it could be extended to have other information besides package names, e.g. , urls for the project if needed, project owner, a license, etc.

I was trying to think of some example workflows - do these make sense? Any others?

Proposed Workflows

No project exists yet

Install and load the pkg

devtools::install_github("RevolutionAnalytics/RRT")
library("RRT")

Initialize a repo. Defaults to creating in the current working directory.

rrt_init()

Write some code...

Then refresh the repo by running rrt_refresh() which will look through the repository again and install any new packages needed, update internal manifest, etc.

rrt_refresh()

Project (folder) already exists

The same process as above, except that since repo already exists, when rrt_init() is run, an internal rrt directory is created with associated files within the directory given by user, and packages and deps collected from the files already in the directory.

sort of like install_github(), but for MRAN, we already have a fxn to download and a separate one to install packages, but not specifically from MRAN.

When a user wants to install all new packages this function can clean out the old packages for them, and print success message.

Most likely to be used before installing from snapshot.

Right now we get

rrt_repos_list()

## Warning: cannot open file
## '/Users/sacmac/mynewrepository/rrt/rrt_manifest.txt': No such file or
## directory
## Error: cannot open the connection

Check for existence and move on gracefully

User specify via one of: folder path, identifier (UUID?), repo name, url, etc., Parse from one parameter for user simplicity.

Look in the path to the list of repos in the .Renviron or .Rprofile file perhaps

Where to keep track of various repositories on a users machine.

Perhaps write to .Renviron a record like RRT_REPO_LIST="/path/to/list.txt" where list.txt has the list of paths to repos and their names

I started RStudio in a new project directory, and then tried rrt_init() to set it up as an RRT repository. Then I realised interactive=TRUE is the default. This is what I got:

rrt_init(interactive=FALSE)
Checking to see if repository exists already...
Checing to make sure rrt directory exists inside your repository...
Creating rrt directory /Users/david/R/RRT-example/rrt/lib/x86_64-apple-darwin13.1.0/3.1
Looking for packages used in your repository...
Error in if (repos["CRAN"] == "@cran@") { :
missing value where TRUE/FALSE needed

What's ideal tool to use if we have a command line command like RRT --optional flags...

I've written command line tools in Ruby and Python, but not R, but if we're trying to stick with R, maybe there is a way to do in R. Using Rub/Python I guess would introduce new deps

Create a short screencast demo of RRT 0.1 functionality. Maximum length of ~10 minutes, can be shorter. Due date: 6-24 by end of day.

I'll start working on this locally, then we can move to test on server

• Do all in R first - see how painful
• tools::write_PACKAGES to create metadata for each package
• convert to JSON? or not? if JSON (or other structured text), will be easier to add to, and parse downstream
• add additional metadata (with a schema predefined)
  • snapshot metadata
  • snapshot diff metadata
  • include compatability (down the road) - Define key-value pair, but = NULL for now

R packages aren't necessarily on CRAN, maybe don't deal with this right away, but would be nice to have a slick solution for this

Just using bower init this morning on the cli, they have an interactive bower init process that asks for input to set up a bower.json file

Seems this could be easily done for RRT, asking for input on where to setup the new repository, what packages to install, authors, license, online git/svn repo, etc.

bower init suggests defaults for each question, which could be done as well i think.

Right now these are stored in tmp files, write these somewhere else perhaps so if the user shuts down the R session, the files are still available to look at.

This is labeled with the next milestone (v0.2), but seems like an important feature.

Possible options include wrapping up into a compressed zip or tarball, or sharing to the web via github, etc.

for each package

Would have to either look at diff for the snapshot in question, or just compare for each package any differences. probaly diff will work fine. then insert message for changes.