r-hub / containers Goto Github PK

Maybe.

Sorry for the issue spam (there's a lot of issues I'm running into haha) but the mkl headers aren't included with the mkl (and presumably the intel) containers. Really difficult for me to build against MKL without them!

To be able to reproduce some test failures. Unfortunately it seems that we cannot set more threads than the number of logical CPUs, so Docker must be allowed at least two CPUs for this to work:

ENV OMP_THREAD_LIMIT=2
ENV R_DATATABLE_NUM_THREADS=2

Thanks for these containers — I'm excited to work with the next generation of the r-hub images.

When using the clang17/clang18 images I noticed that I wasn't able to build and install {curl} due to a lack of libcurl4-openssl-dev in the image. I'm happy to send a PR that adds it either to just those images, or to all of the relevant ones.

Note: the setup I'm using doesn't use the rhub repo that comes built in (and I don't think we will want to just yet), but even with that I'm getting a 404 for the URL it's trying. I'm happy to open a second issue for this so we can keep them separate, but wanted to mention here too

trying URL 'https://raw.githubusercontent.com/r-hub/repos/main/ubuntu-22.04/4.4/libc++/src/contrib/curl_5.2.0_b3_R4.4_x86_64-pc-linux-gnu-ubuntu-22.04-libc++.tar.gz'
Error in download.file(url, destfile, method, mode = "wb", ...) : 
  cannot open URL 'https://raw.githubusercontent.com/r-hub/repos/main/ubuntu-22.04/4.4/libc++/src/contrib/curl_5.2.0_b3_R4.4_x86_64-pc-linux-gnu-ubuntu-22.04-libc++.tar.gz'

I don't really want to install it on the images, because it is huge. Maybe I could install a minimum subset and then install the rest on demand.

From https://www.stats.ox.ac.uk/pub/bdr/clang16/README.txt:

Tests as for fedora-clang but using clang 16.0.0git rather than 15.0.7.
This is scheduled for release on Mar 7th.

Currently using C23 as the default C standard (but C23 issues are
reported elsewhare).

Other details as
https://www.stats.ox.ac.uk/pub/bdr/Rconfig/r-devel-linux-x86_64-fedora-clang

References for Rf_length in system headers come from includeing R headers
before system headers rather than after: see 'Writing R Extnesions'.

The [-Wenum-constexpr-conversion] errors seen in

RSQLite

are from the old Boost headers.

std:unary_function was deprecated in C+11 and removed in C++17, and finally
in clang 16 in C++17 mode.  Boost 1.81 was still using it.

Would be nice to have some tools for this. The difficult issue is keeping the library current. An alternative would be us building binary packages and putting them into a repository on GitHub Packages or elsewhere.

It seems that we can use Ubuntu 22.04 for this, with clang-15 from the LLVM repo, and gfortran-12.

We can use R-devel with P3M R-release binaries.

We can compile base R without ASAN support and link it with ASAN support. This way base R is not slow because of the ASAN instrumentation, but packages can be ASAN instrumented (via the user's Makevars).

docker run --rm -ti --platform linux/amd64 ghcr.io/r-hub/containers/gcc13@sha256:841d0ddb72d2a66131e0928c13829cae921e9ab2abc78f50ea584201f4606776 R --vanilla -e TRUE
docker run --rm -ti --platform linux/amd64 ghcr.io/r-hub/containers/gcc13:latest R --vanilla -e TRUE

Both freeze. For comparison, on the same system:

docker run --rm -ti --platform linux/amd64 ghcr.io/cynkra/rig-ubuntu:main R --vanilla -e TRUE

Status: Downloaded newer image for ghcr.io/cynkra/rig-ubuntu:main

R version 4.3.2 (2023-10-31) -- "Eye Holes"
Copyright (C) 2023 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> TRUE
[1] TRUE
>
>

To follow CRAN. For this we need to install pak from source, and since we currently need the devel versions of pak, this is messy. So we wait until pak with the new sysreqs support is on CRAN.

I.e. set env vars and other config for the build, and not for the whole Dockerfile.

This would allow installing multiple R builds in the same system.

This mainly means

• making sure external software is available,
• setting env vars from the system environment file or the system profile,
• setting Makeconf stuff in the system Makeconf file, not in ~/.R/Makeconf.

It seems that when installing packages using pak we get a Execution halted error when using the clang-ubsan container.

For example:

https://github.com/mlverse/tfevents/actions/runs/4252622314/jobs/7396446533

Everything seemed to work fine for oter container types.

ubuntu-clang causes Rcpp to fail upon #include <cmath>. A bit of research showed that clang searches for the libstdc include directory based on the newest installed gcc, which in this case is 12. While (I think) the libstdc++11 headers are installed, the libstdc++12 headers are not. This is easily resolved by adding the libstdc++-12-dev package-or probably by removing gcc-12.

E.g. bsts does not work with binary Boom, on clang-asan.

If the website update fails, then the manifest will be missing some container builds. The same happens for manual container updates without website builds.

A better way to update the manifest would be:

• List the last five container versions via the GH API.
• Check if we know about them in the current manifest file.
• Pull the ones we don't know about, and look up their metadata.
• Merge the old and new metadata.

This one is tricky, because we need to keep the sources around, both for R and the package(s) we want to check.

We also need the wllvm Python package, which is not packaged for Ubuntu, so ideally we would install it together with R.

It does not seem important to use R-devel for this, so we could use R-release and Ubuntu 22.04, and binary packages, except for the packages we want to check.

C code from one package may call C code from another package, but I don't know if rchk handles this situation. If it does then LinkingTo packages should be also installed from source, with wllvm.

In general we don't actually need to install the dependencies of the checked package, only the LinkingTo dependencies, and their dependencies. This should work for most packages, but in case it does not we should fall back to the full (hard) dependency install. This is how it is done in the original rchk container: https://github.com/kalibera/rchk/blob/fbde173a84809e8159e50bdc6a03e12119634594/scripts/utils.r#L76

Many packages with configure use pkg-config to find libraries, and fail in cryptic ways if pkg-config itself is not available.

I created one based on Ubuntu 22.04, it is really easy because Intel has a repo for this:
https://github.com/intel/oneapi-containers/tree/master/images/docker/basekit

But it does not reproduce the errors in the svrep package. :(

So now I need to try to create one based on Fedora, but I am not even sure if we are testing MKL or Fedora at this point....

Current container is here: #40

With R-devel.

Added. Missing the custom APT repo, because the clang16 repo is not compatible with it.

Hi, not sure if this is the correct place to direct this issue to, but hopefully you can point me in the right direction at least?
I'm attempting to debug clang17 issues raised by CRAN on a colleague's package. I can't manage to get a working build image though, because a dependency spam doesn't compile successfully under the clang17 image.
The CRAN page doesn't seem to list any problems with the spam package under clang17 so presumably it compiles and installs properly on their system?

Attached is the error output from running:

docker run -v `pwd`/check:/check ghcr.io/r-hub/containers/clang17:latest r-check

Any help is much appreciated and thank you so much for the work you do in providing these images!

spam_install_errors.txt

Opting out does not work, --as-cran turns it back on.

Autodetect scripts written by Intel use this environment variable to determine where MKL is in the absence of a configuration argument. It would be ideal to set this for containers in which MKL is expected to be used.

edit: probably easiest way to accomplish this is to source /opt/intel/oneapi/mkl/latest/env/vars.sh

Can be a lightweight extension of the gcc13 image. Upstream: https://www.stats.ox.ac.uk/pub/bdr/LTO/README.txt .

I made it work for myself with the following ~/.R/Makevars :

CFLAGS := $(CFLAGS) -flto=10
FFLAGS := $(FFLAGS) -flto=10
CXXFLAGS := $(CXXFLAGS) -flto=10

Unlike the upstream doc, I don't need --use-LTO for R CMD INSTALL . to enable this. But there might be a much nicer way to configure the gcc-13 image to listen to the --use-LTO flag?

This sometimes needs extra packages.

• atlas/
• centos7/
• clang-asan/
• clang16/
• gcc13/
• nold/
• ubuntu-clang/
• ubuntu-gcc12/

At

Once the official snapshot catches up.

AKA r-devel-linux-x86_64-debian-clang:

So all test output is shown.

Or maybe just the manifest is not updated?

We apparently push to the registry here: https://github.com/r-hub/containers/actions/runs/4061491241/jobs/6991563669

This is the current development branch of gcc. Possible ways to get it include:

1. Compile ourselves. Not very hard with an updated version of https://github.com/docker-library/gcc, and we get a Debian 11 container, which is quite big, unfortunately, as the Dockerfile does not clean up. It takes about an hour on my laptop, so probably about 2 hours on GHA. https://github.com/r-hub/docker-library-gcc/tree/gcc-13
2. Use Ubuntu 23.04 and this PPA: https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test?field.series_filter=lunar It is not clear how often they build their packages. The last build was already 10 days ago, so it seems unlikely that they build the weekly snapshots.
3. Use Fedora rawhide (as 38), which has gcc-13 already. Seems like they build it frequently. https://packages.fedoraproject.org/pkgs/gcc/gcc/fedora-rawhide.html
4. Use Fedora 36, and install the gcc-13 packages on it from rawhide. It goes like this:

dnf install fedora-repos-rawhide -y
dnf --disablerepo=* --enablerepo=rawhide  --releasever=38 install gcc g++ gfortran

I am leaning towards 4., as we get updates easily, and the process probably generalizes to newer gcc updates. It does mean that we need system requirements support on Fedora, and that we won't have package binaries.

They'll never get super small, but the ubuntu-clang container probably should not be 2.5GB.

I see messages like:

Error in download.file(url, destfile, method, mode = "wb", ...) : 
  cannot open URL 'https://raw.githubusercontent.com/r-hub/repos/main/fedora-36/4.4/src/contrib/ps_1.7.5_b1_R4.4_x86_64-pc-linux-gnu-fedora-36.tar.gz'
In addition: Warning messages:
1: In download.file(url, destfile, method, mode = "wb", ...) :
  downloaded length 0 != reported length 0
2: In download.file(url, destfile, method, mode = "wb", ...) :
  cannot open URL 'https://raw.githubusercontent.com/r-hub/repos/main/fedora-36/4.4/src/contrib/ps_1.7.5_b1_R4.4_x86_64-pc-linux-gnu-fedora-36.tar.gz': HTTP status was '404 Not Found'

setting options(repos=) to a valid CRAN repository fixed the problem, but this seems unexpected.

Edit: Sorry if this is not yet supposed to work . I just noticed this is actually work in progress :)

E.g. use the check action, possibly, so we can fail on warnings.

This seems simple but I am not getting anywhere in reproducing the failures on CRAN.

1. On Ubuntu/Debian, we can install an alternative BLAS/LAPACK, and then use update-alternatives to use it. Easy to set up and it works with all R builds. But it did not reproduce CRAN's failures: 869f9fa
2. I tried the same after building R with gcc 12.x, to match the compiler on CRAN. Of course this does not mean that ATLAS was built with gcc 12.x, and not surprisingly, this does not reproduce the CRAN failures, either: 0574b17

surf

https://www.stats.ox.ac.uk/pub/bdr/Rblas/ATLAS/surf.out

  > test_check("surf")
  Loading required package: surf
  Loading required package: Matrix
  [ FAIL 5 | WARN 4 | SKIP 0 | PASS 486 ]
  
  ══ Failed tests ════════════════════════════════════════════════════════════════
  ── Failure ('test-zero_counts_ModelB.R:176'): compare influence functions estimates ──
  attr(flow_srs_lin$eta, "influence") not identical to attr(flow_srs_rep$eta, "influence").
  Objects equal but not identical
  ── Failure ('test-zero_counts_ModelB.R:177'): compare influence functions estimates ──
  attr(flow_srs_lin$pij, "influence") not identical to attr(flow_srs_rep$pij, "influence").
  Objects equal but not identical
  ── Failure ('test-zero_counts_ModelB.R:178'): compare influence functions estimates ──
  attr(flow_srs_lin$psi, "influence") not identical to attr(flow_srs_rep$psi, "influence").
  Objects equal but not identical
  ── Failure ('test-zero_counts_ModelB.R:179'): compare influence functions estimates ──
  attr(flow_srs_lin$rho, "influence") not identical to attr(flow_srs_rep$rho, "influence").
  Objects equal but not identical
  ── Failure ('test-zero_counts_ModelB.R:180'): compare influence functions estimates ──
  attr(flow_srs_lin$tau, "influence") not identical to attr(flow_srs_rep$tau, "influence").
  Objects equal but not identical
  
  [ FAIL 5 | WARN 4 | SKIP 0 | PASS 486 ]
  Error: Test failures

layer

https://www.stats.ox.ac.uk/pub/bdr/Rblas/ATLAS/layer.out

* checking examples ... [18s/22s] ERROR
Running examples in ‘layer-Ex.R’ failed
The error most likely occurred in:

> ### Name: plot_tiltedmaps
> ### Title: Tilt raster and sf data
> ### Aliases: plot_tiltedmaps
> 
> ### ** Examples
> 
> # tilt data
> tilt_landscape_1 <- tilt_map(landscape_1)
Loading required package: raster
Loading required package: sp
Error in CPL_geos_is_empty(st_geometry(x)) : 
  Evaluation error: IllegalArgumentException: Points of LinearRing do not form a closed linestring.
Calls: tilt_map ... <Anonymous> -> Ops.sfg -> st_is_empty -> CPL_geos_is_empty
Execution halted

sandwich

This just seems like a test output difference, which also happens for other CRAN checks, but it is not flagged as an error for those:

  Running ‘vcovCL.R’ [74s/104s]
  Comparing ‘vcovCL.Rout’ to ‘vcovCL.Rout.save’ ...199,200c199,200
< (Intercept)     0.139722   -1.754e-03
< institutions   -0.001754    3.822e-05
---
> (Intercept)     0.139697   -1.751e-03
> institutions   -0.001751    3.815e-05
203,204c203,204
< (Intercept)     0.138531   -1.754e-03
< institutions   -0.001754    3.714e-05
---
> (Intercept)     0.138509   -1.752e-03
> institutions   -0.001752    3.708e-05
  Running ‘vcovPC.R’ [10s/13s]

ISwR

This also seems like output difference.

  Comparing ‘allexercises.Rout’ to ‘allexercises.Rout.save’ ...192c192
< In wilcox.test.default(x = DATA[[1L]], y = DATA[[2L]], ...) :
---
> In wilcox.test.default(x = c(4.62, 5.29, 5.52, 3.71, 4.02, 5.09,  :
224c224
< alternative hypothesis: true mean difference is not equal to 0
---
> alternative hypothesis: true difference in means is not equal to 0
228c228
< mean difference 
---
> mean of the differences 
1035c1035
< id     15     51138 70816 252.47  2.4254 0.05287 .
---
> id     15     51137 70816 252.47  2.4254 0.05287 .
1045c1045
< id        15  51138  3409.2  2.4254 0.05287 .
---
> id        15  51137  3409.2  2.4254 0.05287 .
  Running ‘allscripts.R’
  Comparing ‘allscripts.Rout’ to ‘allscripts.Rout.save’ ...487c487
< [1] 189
---
> [1] 185
982c982
< alternative hypothesis: true difference in means between group lean and group obese is not equal to 0
---
> alternative hypothesis: true difference in means is not equal to 0
995c995
< alternative hypothesis: true difference in means between group lean and group obese is not equal to 0
---
> alternative hypothesis: true difference in means is not equal to 0
1024c1024
< In wilcox.test.default(x = DATA[[1L]], y = DATA[[2L]], ...) :
---
> In wilcox.test.default(x = c(7.53, 7.48, 8.08, 8.09, 10.15, 8.4,  :
1049c1049
< alternative hypothesis: true mean difference is not equal to 0
---
> alternative hypothesis: true difference in means is not equal to 0
1053c1053
< mean difference 
---
> mean of the differences 
2749,2751c2749,2751
< gr          1 206698  206698     NaN    NaN
< factor(gr)  4  17405    4351     NaN    NaN
< Residuals   0      0     NaN               
---
> gr          1 206698  206698               
> factor(gr)  4  17405    4351               
> Residuals   0      0                       
3004c3004
< alternative hypothesis: true difference in means between group Yes and group No is not equal to 0
---
> alternative hypothesis: true difference in means is not equal to 0
3620c3620
< [1,] 205     57     NA      NA      NA
---
>     205      57      NA      NA      NA 
 [16s/20s] OK

Would be nice. Or some tools to create them.

Some packages need this without specifying it as a system requirement.

I've been having a consistent issue on all of the ubuntu containers with the exception of release and next. My in-development package grabs its LDFLAGS from R CMD --ldflags which, in this case, contains -ldeflate. This would not be an issue if it were not for debian only packaging libdeflate.so.0 in the installed libdeflate package. For some reason, the libdeflate.so symlink is only installed if libdeflate-dev is installed as well. This consistently fixes the issue.