richfitz / drat.builder Goto Github PK

I've been playing around a bit with {drat.builder} for the past couple of days, mainly b/c for the second time now, I ran into the issue that my drat repo did not contain the required bin paths (see PRs #23 and #24 and with the imminent release of 4.2 we'll face this once again). In my fork of your repo I added functionality that can check with CRAN to get all current macOS and Windows paths (see master...nbenn:master). This exposes an argument check_cran, which when TRUE will get the directories from CRAN (with a bit of cleanup required, as there are also dirs like tools, base, etc. that we don't want) and update the drat repo accordingly. I'm not fetching versions from metacran, as this would not have been able to save us from #24 and the only way to achieve this that I could think of, was accessing CRAN itself.

Happy to submit a PR if this is of interest to you.

It looks like drat.builder now supports the package root not being in the git root directory, but it still complains if two packages are in the same repo (which in my experience is the main reason someone uses this pattern in the first place), e.g.

egonw/rrdf/rrdflibs
egonw/rrdf/rrdf

I realize the logic is a bit tricky here, since drat.builder doesn't want to attempt to clone the repo twice..

I often see fetch fail if a package has been updated since I last ran drat, and I still have a packages_src directory sitting around.

I haven't completely diagnosed this one yet, might be something in how you are doing the original clone command (e.g. perhaps you're doing shallow clones that don't pull the whole history? Of course that's nice for this context, but would create this issue I think).

e.g.:

Error in call_system(Sys_which("git"), c("-C", path, "fetch", "origin",  : 
  Running command:
  '/usr/bin/git' -C packages_src/knitr fetch origin *:*
had status 1
Program output:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
From https://github.com/yihui/knitr
 ! [rejected]        refs/pull/1073/merge -> refs/pull/1073/merge  (non-fast-forward)
 ! [rejected]        refs/pull/1080/merge -> refs/pull/1080/merge  (non-fast-forward)

I think the shallow clones are ideal (particularly since the nightly builds on circle are always doing fresh pulls anyway, so it's nice that's as fast as possible), and the ideal behaviour might be to catch this and delete the stored copy so we can shallow clone again. But otherwise full clones might be the quick fix. (Or maybe I'm way off and the issue is completely different).

In my drat repo, I call build("packages.txt") to install a list of hadleyverse packages, and then build("ropensci.txt") to build an automatically generated list of ropensci packages (using the ropkgs API).

Unfortunately, somehow the second call seems to be purging all the downloaded packages from the first list -- none of them are in packages any more, only the ropensci packages are. Meanwhile, drat.json seems to only have the entries for the packages in first list (packages.txt), and not the others. Consequently, an attempt to rebuild fails with:

Error in git2r::fetch(package_repo(p), "origin") : 
  error in evaluating the argument 'repo' in selecting a method for function 'fetch': Error in normalizePath(path, winslash = "/", mustWork = TRUE) : 
  path[1]="./packages/assertthat": No such file or directory
Calls: package_repo ... <Anonymous> -> <Anonymous> -> .local -> normalizePath
Calls: build ... fetch_package_sources -> fetch_package -> <Anonymous>
Execution halted

On the first package not to be in the packages list (e.g. assertthat here, after I manually clone a few of the previous to check). Deleting drat.json doesn't help, I still get this error. I have to manually remove packages_src to get rebuild going again.

Also noticed that somehow I ended up with a blank PACKAGES and PACKAGES.gz list in my repo -- not sure how that happened (but just in time for Prof to scold me for the packages not being there when I pushed a patch to RNeXML last night...).

Use full url, perhaps:

ropensci/lawn # comes from github
https://bitbucket.com/foo/bar # comes from bitbucket

Another option would be to use toml or yaml to break the file into different sources:

[github]
ropensci/lawn

[bitbucket]
foo/bar

[urls]
https://whatever....

But then, I don't have a need for this at present - for better or worse things seem pretty centralised on github...

cboettig/drat@9ee7fed

drat now supports binary packages, and CI infrastructure (e.g., GitHub Actions) typically support multilple OSes, so it's feasible to build binaries for Windows and macOS on the CI machines and push those to the drat repo.

It would be great if drat.builder could support this.

perhaps this package is only intended for CI use, so this may be a moot point...

I successfully setup a drat repository using drat.builder::build() but I am unable to run build() more than once.
Subsequent calls give the error below. Note, this also completely breaks the git submodule structure and breaks the repo.

> drat.builder::build()
*** [fetch] achubaty/amc@development
*** [fetch] PredictiveEcology/CBMutils@development
*** [fetch] PredictiveEcology/fireSenseUtils@development
*** [fetch] PredictiveEcology/LandR@development
*** [fetch] PredictiveEcology/LandWebUtils@development
*** [fetch] PredictiveEcology/map@development
*** [fetch] PredictiveEcology/pemisc@development
*** [fetch] PredictiveEcology/peutils@development
*** [fetch] PredictiveEcology/quickPlot@development
*** [fetch] PredictiveEcology/Require@development
*** [fetch] PredictiveEcology/reproducible@development
*** [fetch] PredictiveEcology/SpaDES.core@development
*** [fetch] PredictiveEcology/SpaDES.tools@development
Error in call_system(Sys_which("git"), args, ...) : Running command:
  '/usr/bin/git' clone -- packages_src/achubaty/amc ./packages/amc
had status 128
Program output:
----------------------------------------------------------------------------------------------------------
fatal: repository 'packages_src/achubaty/amc' does not exist
----------------------------------------------------------------------------------------------------------

same result on Ubuntu 20.04 and Windows 10.

vignette building is slow, adds dependencies and points of failure. It might be good to have the option to just skip the vignette building step? (e.g. --no-build-vignettes)

Currently a problem in traitecoevo/drat

So invalid entries in packages.txt will not throw errors

As awesome as the free hosting with super-stable GitHub servers and fast CDN and all is, there are some use cases for which it would make more sense to host elsewhere than on Github pages -- e.g. so we can parse server logs and compute download counts, etc. Seems the commit workflow is relatively hardwired into the top-level functions though. Probably just need a way to toggle off the call to update_drat in the build() call.

So many things needed that are not included now that accessing both ways is starting to feel a bit silly. So could drop the git2r dep and add a SystemDependencies for git (>= 1.8.0).

Should removal of a package from packages.txt remove the package? Does it already?

https://travis-ci.org/richfitz/drat.builder

Breaks parse_packages, easy enough to fix.

Change occurs in R ~3.3.0