revolutionanalytics / checkpoint Goto Github PK
View Code? Open in Web Editor NEWInstall R packages from snapshots on checkpoint-server
Install R packages from snapshots on checkpoint-server
This feature is apparently in packrat
as well as GRAN
.
Seems that packrat
only snapshots the packages used in the repo, that is, the packrat/
dir in the user's repo. I think GRAN
snapshots the whole repo, including the users files, etc.
This is basically what git does really well, so would be nice to just use git, but can we do something similar without the user having to have git? Ideas:
rrt_manifest.yml
file in a "versions" folder somewhere in the repo, named with repoid + version number, like 1944a6826bee36f5925f1d13f11c6dc6_v1
. Then a user can switch between versions with a single function. I think this can easily be done, but we'd want to probably make a user go through a prompt to make sure they actually want to overwrite files.A note on terminology: since we have the notion of "snapshots" in marmoset/mran, if we have the notion of versions of a user's RRT repositories on their machine, we should use a different term, perhaps "version"?
When users do this when they have started R in their RRT repo, the package installs in the library in the repo (this is good), but the source is just installed in a temporary dir and deleted at end of session. Perhaps look for these installed packages without source files whenever rrt_install
or rrt_refresh
are run, and download the sources of the approprate version
create method to step through /src/contrib/Archive/ folders
include in /MRAN/RRT directory/filesystem
If you try and run rrt_install
on a repository that includes any folders OTHER THAN rrt or .git, it will fail.
> rrt_install()
Checking to make sure repository exists...
Checing to make sure rrt directory exists inside your repository...
Error in rrt_install() : rrt directory doesn't exist
The reason in this case is that my repository includes the .Rproj.user folder created by RStudio:
> repo=getwd()
> file.path(repo, "rrt", "lib", R.version$platform,
+ base::getRversion())
[1] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1.0"
> present <- list.dirs(repo)[-1]
> present
[1] "/Users/david/RRT-example2/.Rproj.user"
[2] "/Users/david/RRT-example2/.Rproj.user/9CD34B73"
[3] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/ctx"
[4] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/presentation"
[5] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/sdb"
[6] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/sdb/prop"
[7] "/Users/david/RRT-example2/.Rproj.user/9CD34B73/sdb/s-3B2F7D3D"
[8] "/Users/david/RRT-example2/rrt"
[9] "/Users/david/RRT-example2/rrt/lib"
[10] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0"
[11] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1"
[12] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1.0"
[13] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1.0/src"
[14] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1.0/src/contrib"
[15] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate"
[16] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate/help"
[17] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate/html"
[18] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate/Meta"
[19] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/manipulate/R"
[20] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio"
[21] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio/help"
[22] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio/html"
[23] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio/Meta"
[24] "/Users/david/RRT-example2/rrt/lib/x86_64-apple-darwin13.1.0/3.1/rstudio/R"
Which causes the following test in rrt_install to fail:
if (!all(grepl("rrt", present))) {
stop("rrt directory doesn't exist")
}
Right now, we print progress for package downloads and installs.
Get rid of these and give much more compact messages to users.
@revodavid Do you want the first version to be v0.1
or v1.0
?
provide login details to Scott
Probably makes sense to install binaries elsewhere, like bin/...
something, then make sure to look for those there
If build fails in some cases, then not such a helpful vignette.
rrt_init()
.R
files, or .Rmd
, etc. with some code, including some librariesrrt_refresh()
- to look for new packages that will be downloadedrrt_install()
- to install packages in the repositoryzfs list -t snapshot
output to a text file that lives under marmoset server $WWW_HOME
crontab
from discussion in #2
we need a field for snapshot ID too. I can add this to the programmatic manifest file changes when a user decides to use MRAN - when they do we can write a field called MRAN_ID
or something like that
I'm familiar with Travis-CI, but is there a preference for Jenkins or something else? @cmosetick
There is quite good support for R packages now on Travis-CI (if we go the travis route), so perhaps structuring this repo as a package, even if not essential, would give us easy way to get CI.
Right now we have DateCreated
but that actually updates to latest date it was modified, so have both DateCreated
and DateModified
or DateUpdated
.
Also, signify time zone in dates.
related to #13
in the short term at least...Employ ability to locally test for package compatibility
Is there anything we can use from GRAN in RRT?
If any files present, find pkgs and write this file, if no files present simply write a blank file.
we can use https://github.com/ropensci/git2r if we want an R native git binding
Do we want to do this? If so, right away, or for later?
Using rsnapshot we can allow users to get snapshots of the marmoset repository. Wrap in RRT
...
Currently testing on Windows. Seems to be a few bugs in how paths are found and used, looking at now.
Thoughts on this?
myrepo - |
|- manifest.txt # user specified list of pkgs
|- code.R # any number of user supplied files
|- rrt # this dir created by rr_init()
|- lib # holds packages in pkgs/
|- pkgs - |
|-doMC
|-plyr
|- manifest.lock # RRT generated metadata file
configure a rsync daemon on marmoset server so RRT can utilize the rsync protocol for file transfers to users systems.
Things to consider doing on the server
These tests should be more robust than those on CRAN obviously.
We talked about this on the call...While we can get dependencies, we don''t know if the packages actually work together. There is some filtering for this given the CRAN package submission process (but it's not strictly enforced)
Perhaps: A single function (and helpers) to test that packages are compatible with one another in the project, either
The problem with the local testing is that with a large number of packages this could take a long time, but will explore ways to get the time down.
Running R from an initialized repository where the only file (myscript.R) has the following content:
## Example script using packages
require(ggplot2)
require(plyr)
require(data.table)
require(knitr)
require(lattice)
print(sessionInfo())
I run rrt_refresh:
> rrt_refresh()
Warning messages:
1: In packageDescription(y, encoding = NA) : no package 'hexbin' was found
2: In packageDescription(y, encoding = NA) : no package 'Hmisc' was found
3: In packageDescription(y, encoding = NA) : no package 'mapproj' was found
4: In packageDescription(y, encoding = NA) : no package 'maps' was found
5: In packageDescription(y, encoding = NA) : no package 'maptools' was found
6: In packageDescription(y, encoding = NA) : no package 'multcomp' was found
7: In packageDescription(y, encoding = NA) : no package 'quantreg' was found
8: In packageDescription(y, encoding = NA) : no package 'abind' was found
9: In packageDescription(y, encoding = NA) : no package 'doMC' was found
10: In packageDescription(y, encoding = NA) : no package 'foreach' was found
11: In packageDescription(y, encoding = NA) : no package 'iterators' was found
12: In packageDescription(y, encoding = NA) : no package 'itertools' was found
13: In packageDescription(y, encoding = NA) : no package 'bit64' was found
14: In packageDescription(y, encoding = NA) : no package 'chron' was found
15: In packageDescription(y, encoding = NA) : no package 'data.table' was found
16: In packageDescription(y, encoding = NA) : no package 'fastmatch' was found
17: In packageDescription(y, encoding = NA) : no package 'reshape' was found
18: In packageDescription(y, encoding = NA) : no package 'xts' was found
19: In packageDescription(y, encoding = NA) : no package 'rgl' was found
20: In packageDescription(y, encoding = NA) : no package 'testit' was found
A number of options here to start off with:
I'll have a peak around each of them and figure out what else is needed. Probably better now to not have too many dependencies, so new code I think is in order.
See gRAN for reference: https://raw.githubusercontent.com/gmbecker/gRAN/master/vignettes/ReportScreenshot.png
Include:
.R
, .md
, .txt
, .json
, .yml
.R
file. Could have no markup and simply a list of packages likeplyr
ggplot2
...
or could use yaml markup as in a Makefile. I like the idea of using yaml as it could be extended to have other information besides package names, e.g. , urls for the project if needed, project owner, a license, etc.
I was trying to think of some example workflows - do these make sense? Any others?
Install and load the pkg
devtools::install_github("RevolutionAnalytics/RRT")
library("RRT")
Initialize a repo. Defaults to creating in the current working directory.
rrt_init()
Write some code...
Then refresh the repo by running rrt_refresh()
which will look through the repository again and install any new packages needed, update internal manifest, etc.
rrt_refresh()
The same process as above, except that since repo already exists, when rrt_init()
is run, an internal rrt
directory is created with associated files within the directory given by user, and packages and deps collected from the files already in the directory.
sort of like install_github()
, but for MRAN, we already have a fxn to download and a separate one to install packages, but not specifically from MRAN.
When a user wants to install all new packages this function can clean out the old packages for them, and print success message.
Most likely to be used before installing from snapshot.
Right now we get
rrt_repos_list()
## Warning: cannot open file
## '/Users/sacmac/mynewrepository/rrt/rrt_manifest.txt': No such file or
## directory
## Error: cannot open the connection
Check for existence and move on gracefully
User specify via one of: folder path, identifier (UUID?), repo name, url, etc., Parse from one parameter for user simplicity.
Look in the path to the list of repos in the .Renviron
or .Rprofile
file perhaps
Where to keep track of various repositories on a users machine.
Perhaps write to .Renviron
a record like RRT_REPO_LIST="/path/to/list.txt"
where list.txt
has the list of paths to repos and their names
I started RStudio in a new project directory, and then tried rrt_init() to set it up as an RRT repository. Then I realised interactive=TRUE is the default. This is what I got:
rrt_init(interactive=FALSE)
Checking to see if repository exists already...
Checing to make sure rrt directory exists inside your repository...
Creating rrt directory /Users/david/R/RRT-example/rrt/lib/x86_64-apple-darwin13.1.0/3.1
Looking for packages used in your repository...
Error in if (repos["CRAN"] == "@cran@") { :
missing value where TRUE/FALSE needed
What's ideal tool to use if we have a command line command like RRT --optional flags...
I've written command line tools in Ruby and Python, but not R, but if we're trying to stick with R, maybe there is a way to do in R. Using Rub/Python I guess would introduce new deps
Create a short screencast demo of RRT 0.1 functionality. Maximum length of ~10 minutes, can be shorter. Due date: 6-24 by end of day.
I'll start working on this locally, then we can move to test on server
tools::write_PACKAGES
to create metadata for each packageNULL
for nowR packages aren't necessarily on CRAN, maybe don't deal with this right away, but would be nice to have a slick solution for this
Just using bower init
this morning on the cli, they have an interactive bower init
process that asks for input to set up a bower.json
file
Seems this could be easily done for RRT
, asking for input on where to setup the new repository, what packages to install, authors, license, online git/svn repo, etc.
bower init
suggests defaults for each question, which could be done as well i think.
Right now these are stored in tmp files, write these somewhere else perhaps so if the user shuts down the R session, the files are still available to look at.
This is labeled with the next milestone (v0.2
), but seems like an important feature.
Possible options include wrapping up into a compressed zip or tarball, or sharing to the web via github, etc.
for each package
Would have to either look at diff for the snapshot in question, or just compare for each package any differences. probaly diff will work fine. then insert message for changes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.