pik-piam / lucode2 Goto Github PK
View Code? Open in Web Editor NEWA collection of tools which allow to manipulate and analyze code.
License: Other
A collection of tools which allow to manipulate and analyze code.
License: Other
If given something like this:
parameter
cm_FlexTaxFeedback "switch deciding whether flexibility tax feedback on buildlings and industry electricity prices is on"
*** cm_FlexTaxFeedback, switches on feedback of flexibility tax on buildings/industry.
*** That is, electricity price decrease for electrolysis has to be matched by electrictiy price increase in buildings/taxes.
*** This switch only has an effect if the flexibility tax is on by cm_flex_tax set to 1.
;
cm_FlexTaxFeedback = 0; !! def 0
and tasked with changing cm_FlexTaxFeedback
to 1, manipulateConfig
will detect the /
in the comment inside the parameter definition and will change everything between the slashes so the file ends up as:
parameter
cm_FlexTaxFeedback "switch deciding whether flexibility tax feedback on buildlings and industry electricity prices is on"
*** cm_FlexTaxFeedback, switches on feedback of flexibility tax on buildings/ 1 /taxes.
*** This switch only has an effect if the flexibility tax is on by cm_flex_tax set to 1.
;
cm_FlexTaxFeedback = 1; !! def 0
i.e. swallowing a good part of the comments.
Within the current framework of manipulateConfig
this is very hard to solve. manipulateConfig uses regular expressions, which are best for context-free grammars, but to properly detect what is happening, we have to detect two contexts - first, the definition context, which in the example goes from parameter
to the first ;
, but can also span multiple definitions, and then, second, the comment context within the definition context, which spans from ***
to the next end of line. While I think it should be theoretically possible to solve this in perl-compatible regular expressions (they are turing-complete, after all), practically, the regular expressions are completely unreadable already, and adding another context detection to them is practically impossible. For the record, this is the regex which detects parameter definitions currently, which does not handle comments properly:
paste0("((\\n|^)[\\t ]*(scalar|parameter|set|)s?[\\t ]*", key, "(|\\([^\\)]*\\))(/|[\\t ]+(\"[^\"]*\"|)[^\"/;]*/))[^/]*")
Hi,
I'm not quite sure if this is a bug or intended behaviour: If I call manipulateConfig
on an empty file, it will just be happy and not report any error or warning. This is a problem if there are typos in the config to be manipulated (or in my new config keys), or if I introduced a new config switch - both will just happily do nothing and report nothing for the affected keys.
Probably, we would get quite a lot of false positives if we added a warning at the moment because not all settings actually end up in main.gms
in normal operation, but I think we should try to find a solution, silently doing the wrong thing / nothing is a bad response to a typo or missing definition.
see attached, here for gms
package. Maybe a thing for @pfuehrlich-pik?
I just pushed directly to pik-piam/lucode2 master instead of to my fork. Stupid error, and I hope I undid it fast enough that no one noticed. However, shouldn't we just disallow that via branch protection rules?
@orichters wrote:
Can we add such a Makefile to all piam-packages, with bl = Rscript -e "lucode2::buildLibrary()", and testthat = Rscript -e "devtools::test(show_report = TRUE)" for example?
I think that is a good idea, will look into it.
This code
Lines 31 to 40 in 009b94c
I have the tests of the remind2
package failing for some time (on some lusweave
/pdflatex
issue), and buildLibrary()
happily ignore that. (Not that I'm eager to have remind2
fail on me several times a day, but … boyscout rule.)
Restarting R session...
> (lib <- getwd())
[1] "/home/pehl/PIK/pik-piam/remind2"
> { testResults <- callr::r(function(lib) {
+ withr::local_options(crayon.enabled = TRUE)
+ return(devtools::test(lib))
+ }, args = list(lib), show = TRUE)
+
+ if (sum(testResults[["failed"]]) > 0 || any(testResults[["error"]])) {
+ stop("Some tests failed, please fix them first.")
+ }
+ }
ℹ Loading remind2
Loading required package: magclass
Registered S3 method overwritten by 'quantmod':
method from
as.zoo.data.frame zoo
ℹ Testing remind2
✓ | F W S OK | Context
|
Executing reportFEnvGDX2mif
\
| | 1 0 | convGDX2mif
x | 1 0 | convGDX2mif [289.6s]
────────────────────────────────────────────────────────────────────────────────
Failure (test-convGDX2mif.R:95:3): Test if REMIND reporting is produced as it should and check data integrity
file.exists(scenarioComparisonPath) is not TRUE
`actual`: FALSE
`expected`: TRUE
────────────────────────────────────────────────────────────────────────────────
|
══ Results ═════════════════════════════════════════════════════════════════════
Duration: 289.6 s
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 0 ]
|
Frustration is a natural part of programming :)
>
The referenced fields do not exist
> testResults[["failed"]]
NULL
> testResults[["error"]]
NULL
so the condition is always FALSE
and no error thrown.
It seems as if testResult
is a list of all test files and their results, which again are lists of all test_that
calls and their final or terminating conditions
> str(testResults)
List of 1
$ :List of 7
..$ file : chr "test-convGDX2mif.R"
..$ context: chr "convGDX2mif"
..$ test : chr "Test if REMIND reporting is produced as it should and check data integrity"
..$ user : num 286
..$ system : num 3.42
..$ real : num 290
..$ results:List of 1
.. ..$ :List of 6
.. .. ..$ message : chr "file.exists(scenarioComparisonPath) is not TRUE\n\n`actual`: \033[32mFALSE\033[39m\n`expected`: \033[32mTRUE\033[39m "
.. .. ..$ srcref : 'srcref' int [1:8] 95 3 95 50 3 50 95 95
.. .. .. ..- attr(*, "srcfile")=Classes 'srcfilecopy', 'srcfile' <environment: 0x561dcb2d9e50>
.. .. ..$ trace : NULL
.. .. ..$ start_frame: int 49
.. .. ..$ end_frame : num 51
.. .. ..$ test : chr "Test if REMIND reporting is produced as it should and check data integrity"
.. .. ..- attr(*, "class")= chr [1:4] "expectation_failure" "expectation" "error" "condition"
- attr(*, "class")= chr "testthat_results"
Something like
> any(sapply(testResults,
+ function(t) {
+ sapply(t$result, inherits, what = "error")
+ }
+ ))
[1] TRUE
might work.
pik-piam/mrdrivers@15fe2e7
This automatic commit does some unecessary stuff (I think) but definitely causes an issue by removing the last line of tests/testthat/_snaps/snaphsots.md. Any ideas?
whattodo
(or similar):
message("Do you want to update your renv? y/n")
if (tolower(gms::getLine()) %in% c("y", "yes")) { renv::install('package'); renv::snapshot(prompt = FALSE) }
you may even install with renv::install('[email protected]')
just the required package version. I don't know if the snapshot part is necessary, you know better.
otherwise the update process crashes
In REMIND, @LaviniaBaumstark and I found that main.gms
defined
$setGlobal cm_EsubGrowth low
but the default.cfg defined cfg$gms$cm_esubGrowth
with a small e
. The substitution done by manipulateConfig
resp. manipulateFile
just dropped this setting without any notification or so, so the settings were simply not applied.
See mismatch here:
> cd /p/tmp/oliverr/NGFS_v3/2022-05-22/remind/output/C_h_ndc_bIT-rem-5
> grep cm_[eE]subGrowth cfg.txt full.gms
cfg.txt: cm_esubGrowth: middle
full.gms:$setGlobal cm_EsubGrowth low !! def = low
Looking at the structure of the code, I don't see an easy option to fix that and issue warnings etc., because manipulateConfig
simply defines lots of manipulations with the intent that lots of them will never be useful, so how could manipulateFile
where the manipulations are applied know which of them are problematic. But maybe @dklein-pik or @tscheypidi have an idea? Or may it be simply be useful to do the replacement not case-sensitive, given that GAMS is case insensitive anyway.
Still, the problem should be much reduced in REMIND simply because we don't store much in default.cfg
anymore and mismatches between main.gms
and default.cfg
should be substantially reduced.
When I am running buildLibrary
with abs or relative pathes rather than from my current working directory I get the following error messages even though the documentation for argument lib
says Path to the package.
Or am I missing something?
lucode2::buildLibrary("/p/projects/open/Jannes/repos/lucode2")
# Checking if your repo is up-to-date...
# Error:
# ! Path '/p/projects/open/Jannes/' does not appear to be inside a project or package.
lucode2::buildLibrary("./repos/lucode2")
# Checking if your repo is up-to-date...
# Error:
# ! Path '/p/projects/open/Jannes/' does not appear to be inside a project or package.
It seems it can only handle the current working directory. A traceback says ...
5: ui_stop("Path {ui_path(path)} does not appear to be inside a project or package.")
4: proj_set(path = path, force = force)
3: local_project(pathToRepo, quiet = TRUE)
2: checkRepoUpToDate(".", autoCheckRepoUpToDate)
1: lucode2::buildLibrary("./repos/lucode2")
When building a library with lucode2::buildlibrary(), the README file is overwritten following some standards. The only field that in my understanding can be customized is the Description
field in the DESCRIPTION file, but this field does not allow for e.g. inserting code chunks, going to a new line, including paragraphs... wouldn't it be better to have a "fully customizable" README section?
more info here: pik-piam/remind2#2 (comment)
Minimal example:
library(lucode2)
configFile <- file.path(tempdir(), 'test.gms')
x <- '* This is a test
scalars
foo "blah"
bar "either x/y or X/Y"
bazz "just x/y"
buzz "not x/y"
;
foo = 0
bar = 0
buzz = 0
bazz = 0
* End of the test
'
cfg <- list(foo = 1,
bar = 1,
buzz = 1,
bazz = 1)
writeLines(x, configFile)
manipulateConfig(configFile, cfg)
cat(paste(readLines(configFile), collapse = '\n'))
returns
* This is a test
scalars
foo "blah"
bar "either x/ 1 /Y"
bazz "just x/ 1 / 1
instead of the expected
* This is a test
scalars
foo "blah"
bar "either x/y or X/Y"
bazz "just x/y"
buzz "not x/y"
;
foo = 1
bar = 1
buzz = 1
bazz = 1
* End of the test
This is relevant to REMIND which employs slashes for units in explanatory text, like this
cm_build_H2costAddH2Inv "additional h2 distribution costs for low diffusion levels (default value: 6.5$/kg = 0.2$/Kwh)"
Although I didn't find any indication of that in the function, this is the observed behavior. Why does this happen?
Either it's using usethis::use_github_action()
wrong, or that is broken upstream.
$ Rscript -e "library(usethis); sessionInfo(); lucode2::buildLibrary(updateType = 'none')"
R version 4.0.4 (2021-02-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 21.04
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
locale:
[1] LC_CTYPE=en_IE.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_IE.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_IE.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] usethis_2.1.0
loaded via a namespace (and not attached):
[1] compiler_4.0.4 magrittr_2.0.1 fs_1.5.0 glue_1.4.2
[5] lifecycle_1.0.1 rlang_0.4.11 purrr_0.3.4
Checking for lucode2 update... You're running the newest lucode2 version.
Is your repository up-to-date? Did you pull immediately before running this check? (Y/n)
ℹ Updating lucode2 documentation
ℹ Loading lucode2
Writing NAMESPACE
Writing NAMESPACE
── Building ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── lucode2 ──
Setting env vars:
• CFLAGS : -Wall -pedantic -fdiagnostics-color=always
• CXXFLAGS : -Wall -pedantic -fdiagnostics-color=always
• CXX11FLAGS: -Wall -pedantic -fdiagnostics-color=always
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
✔ checking for file ‘/home/pehl/PIK/pik-piam/lucode2/DESCRIPTION’ ...
─ preparing ‘lucode2’:
✔ checking DESCRIPTION meta-information ...
─ checking for LF line-endings in source and make files and shell scripts
─ checking for empty or unneeded directories
Removed empty directory ‘lucode2/tests/testthat/_snaps’
─ building ‘lucode2_0.16.3.tar.gz’
── Checking ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── lucode2 ──
Setting env vars:
• _R_CHECK_CRAN_INCOMING_USE_ASPELL_: TRUE
• _R_CHECK_CRAN_INCOMING_REMOTE_ : FALSE
• _R_CHECK_CRAN_INCOMING_ : FALSE
• _R_CHECK_FORCE_SUGGESTS_ : FALSE
• NOT_CRAN : true
── R CMD check ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
─ using log directory ‘/tmp/RtmpNgcNM2/lucode2.Rcheck’ (344ms)
─ using R version 4.0.4 (2021-02-15)
─ using platform: x86_64-pc-linux-gnu (64-bit)
─ using session charset: UTF-8
─ using options ‘--no-tests --no-manual --as-cran’
✔ checking for file ‘lucode2/DESCRIPTION’ ...
─ checking extension type ... Package
─ this is package ‘lucode2’ version ‘0.16.3’
─ package encoding: UTF-8
✔ checking package namespace information ...
✔ checking package dependencies (2.2s)
✔ checking if this is a source package
✔ checking if there is a namespace
✔ checking for executable files (432ms)
✔ checking for hidden files and directories
✔ checking for portable file names
✔ checking for sufficient/correct file permissions
✔ checking serialization versions ...
✔ checking whether package ‘lucode2’ can be installed (7.4s)
✔ checking installed package size ...
✔ checking package directory ...
✔ checking for future file timestamps ...
✔ checking DESCRIPTION meta-information ...
✔ checking top-level files
✔ checking for left-over files ...
✔ checking index information
✔ checking package subdirectories ...
✔ checking R files for non-ASCII characters ...
✔ checking R files for syntax errors ... OK
✔ checking whether the package can be loaded (1.6s)
✔ checking whether the package can be loaded with stated dependencies (1.6s)
✔ checking whether the package can be unloaded cleanly (1.9s)
✔ checking whether the namespace can be loaded with stated dependencies (1.8s)
✔ checking whether the namespace can be unloaded cleanly (2.4s)
✔ checking loading without being on the library search path (1.7s)
✔ checking dependencies in R code (2.1s)
✔ checking S3 generic/method consistency (2.9s)
✔ checking replacement functions (1.9s)
✔ checking foreign function calls (2.1s)
✔ checking R code for possible problems (11.4s)
✔ checking Rd files ...
✔ checking Rd metadata ...
✔ checking Rd line widths ...
✔ checking Rd cross-references ...
✔ checking for missing documentation entries (1.5s)
✔ checking for code/documentation mismatches (4.5s)
✔ checking Rd \usage sections (3.2s)
✔ checking Rd contents ...
✔ checking for unstated dependencies in examples ...
✔ checking examples (7.3s)
✔ checking for unstated dependencies in ‘tests’ ...
─ checking tests ... SKIPPED
✔ checking for non-standard things in the check directory
✔ checking for detritus in the temp directory
── R CMD check results ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── lucode2 0.16.3 ────
Duration: 1m 1.4s
0 errors ✔ | 0 warnings ✔ | 0 notes ✔
ℹ Loading lucode2
ℹ Testing lucode2
✔ | F W S OK | Context
✔ | 10 | checkRequiredPackages [6.2s]
✔ | 10 | package2readme [0.8s]
══ Results ══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Duration: 7.0 s
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 20 ]
No linter warnings - great :D
✔ Setting active project to '/home/pehl/PIK/pik-piam/lucode2'
Error: `path` must be a string.
Execution halted
When developing an R package, buildLibrary()
updates the version number of the package in different places. Here the DESCRIPTION
is written with the new version number. Here it calls package2readme()
to recreate the README.md
with the new version number. In package2readme()
the template README_template.md is filled with infos from the DESCRIPTION
file. At the beginning of the template, [:version:]
is replaced by the correct new version number. At the end of the template, [:cite:]
is replaced by the result of fillCite()
, which seems to create the citation text from the installed version of the R package and thus uses an old wrong version number.
Here is an example of the result of buildLibrary()
on a README.md.
As already noticed beginning of December in madrat-verse on mattermost, lucode2::buildLibrary()
does not update the readme-file of mrremind
anymore. Running lucode2::package2readme()
alone (a suggested workaround) from mrremind
directory yields:
Error in file(con, "r") :
cannot open the connection to 'https://pik-piam.r-universe.dev/packages'
The command with the error is in line 105:
readLines("https://pik-piam.r-universe.dev/packages")
The problem appears to be the https: 'Peer certificate cannot be authenticated with given CA certificates'
. Changing the URL to http avoids it. Would be great if this was fixed.
pik-piam/mrremind#396 failed its automated test yesterday with two warnings:
- checking Rd files ... WARNING
checkRd: (7) convertADVANCE_WP2.Rd:18: Invalid URL: zotero://select/items/JP8X2QFK
checkRd: (7) readADVANCE_WP2.Rd:15: Invalid URL: zotero://select/items/JP8X2QFK
https://github.com/pik-piam/mrremind/actions/runs/4981508009/jobs/8915845019?pr=396
This URL
buildLibrary()
either locally or on the clusterNo idea what the differences in the setup of the Github Workflow and the cluster are.
I think 7c5242f breaks the building library routine for windows:
running lucode2::buildLibrary()
I get the error
Error in system("git status --porcelain", intern = TRUE) :
'git' not found
One of the AMT REMIND runs failed very early, just stating this in log.txt
:
`Global .Rprofile loaded! (R version 4.1.2 (2021-11-01))
Error:
! package or namespace load failed for 'lucode2':
.onLoad failed in loadNamespace() for 'styler', details:
call: mkdirs.default(path, mustWork = TRUE)
error: Failed to create directory (tried 5 times), most likely because of lack of file permissions (directory '/home/lavinia/.cache/R/R.cache/styler' exists but nothing beyond): /home/lavinia/.cache/R/R.cache/styler/1.10.0
Backtrace:
▆
1. └─base::library(lucode2, quietly = TRUE, warn.conflicts = FALSE)
2. └─base::tryCatch(...)
3. └─base (local) tryCatchList(expr, classes, parentenv, handlers)
4. └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
5. └─value[[3L]](cond)
Execution halted
see /p/projects/remind/modeltests/remind/output/SSP5-NDC-AMT_2023-06-03_07.50.43/log_beforeRestart.txt
This check job in pik-piam/modelstats failed with:
aborting due to test error:
Error : .onLoad failed in loadNamespace() for 'styler', details:
call: mkdirs.default(path, mustWork = TRUE)
error: Failed to create directory (tried 5 times), most likely because of lack of file permissions (directory '/home/runner/.cache/R/R.cache/styler' exists but nothing beyond): /home/runner/.cache/R/R.cache/styler/1.10.1
Error in lucode2::check(runLinter = FALSE) : lucode2::check failed
Execution halted
Error: Process completed with exit code 1.
That is a bit annoying, can we somehow make sure this does not happen anymore? Maybe @pfuehrlich-pik, do you have an idea?
Hey, if you have set allowLinterWarnings: no
in R/.buildlibrary
, then linter warnings break the completion of buildLibrary()
.
It would be fantastic to tell the reader in the stop message in L61 of check.R
that the option to set allowLinterWarnings: yes
exists.
Lines 24 to 37 in b7865d5
usethis::git_remotes()
returns all remotes in alphabetic ordergetGitHubRepo()
returns the vector of all remotes,fillGithubActions()
and fillCodecov()
to pass vectors to fillTemplate()
,gsub()
(In gsub(paste0("[:", what, ":]"), fill[[what]], x, fixed = TRUE) : argument 'replacement' has length > 1 and only the first element will be used
), andREADME.md
, potentially pointing badges to a different fork, e.g. [![CRAN status](https://www.r-pkg.org/badges/version/quitte)](https://cran.r-project.org/package=quitte) [![R build status](https://pik.github.com/giannou/quitte/workflows/check/badge.svg)](https://pik.github.com/giannou/quitte/actions) [![codecov](https://codecov.io/gh/giannou/quitte/branch/master/graph/badge.svg)](https://codecov.io/gh/giannou/quitte)
Consider looking for the origin
branch (most likely to exist), or else use only the first remote explicitly:
return(.harmonize(ifelse('origin' %in% names(out), out[['origin']], origin[[1]])))
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.