Mac OS: build fails with out-of-memory error about docker-aut

[ruebot]

I can't recreate this, but I don't use Mac OS.

@ianmilligan1 or @SamFritz can you test with these instructions when you have time?

[sepastian]

Increasing the memory limit for the build inside the container and as a parameter to the build command did not help:

# Dockerfile
:
RUN git clone https://github.com/archivesunleashed/aut.git /aut \
    && cd /aut \
    && export JAVA_OPTS=-Xmx4096m \
    && mvn clean install
:

Then build with:

docker build --memory=4G -t aut .

Same error.

[sepastian]

Setting export JAVA_OPTS="-Xmx512m -XX:MaxPermSize=512m" didn't help either.

[sepastian]

Update: build is now failing for different reasons.

Setting JAVA_TOOL_OPTIONS in Dokerfile solves the out-of-memory-error:

RUN git clone https://github.com/archivesunleashed/aut.git /aut \
    && cd /aut \
    && export JAVA_TOOL_OPTIONS="-Xmx4000m -Xms4000m" \
    && mvn clean install

Also, on MacOS and Win (I believe), build output gets clipped by Docker Desktop, hiding important information on failing builds. To disable, go to Docker Desktop > Preferences > Docker Engine and set features.buildkit to false.

{
  "experimental": false,
  "features": {
    "buildkit": false
  },
  :

The build is now failing with

:
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Picked up JAVA_TOOL_OPTIONS: -Xmx4000m -Xms4000m
Running io.archivesunleashed.ExtractDateDFTest
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/root/.m2/repository/org/apache/spark/spark-unsafe_2.12/3.0.1/spark-unsafe_2.12-3.0.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
22/02/16 08:47:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/02/16 08:47:13 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect.
22/02/16 08:47:42 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.
Killed
The command '/bin/sh -c git clone https://github.com/archivesunleashed/aut.git /aut     && cd /aut     && export JAVA_TOOL_OPTIONS="-Xmx4000m -Xms4000m"     && mvn clean install' returned a non-zero code: 137

Does this have to do with Hadoop behaving differently on Mac/Windows?

I use Mac OS when I have to 🥲

[ianmilligan1]

I am working on a new MacOS laptop (M1 Pro 16GB memory). I can reproduce the Java heap space error above when following the repo instructions, and then can reproduce @sepastian's error when using the fixed memory options.

Are you also on a new Apple chip laptop or on an Intel one @sepastian? I don't know if that matters but I have had some issues with packages here and there since upgrading.

[ruebot]

Could it be a Docker version for y'all? I'm looking at this docker/for-mac#5204 (comment), and that makes sense to me.

You shouldn't need 4G of RAM to build aut, so it's definitely an issue with MacOS and/or Windows Docker.

[ianmilligan1]

I'm on the most recent Docker version. My guess is that there's some stuff in there that's built for intel and the emulation for my M1 laptop isn't working perfectly.

(For what it's worth, I am able to build aut locally although some of the tests cause trouble)

[sepastian]

I'm on an Intel based Mac @ianmilligan1, so doesn't work in Intel either.

True, 4gb aren't required @ruebot. But increasing heap size somewhat is necessary.

[sepastian]

On my Intel Mac I'm running

$ docker --version
Docker version 20.10.11, build dea9396

[ruebot]

@ianmilligan1 can you test this again on main or 1.x.x when you have time? Curious if this is still an issue.

[ianmilligan1]

Tested again and it works!

For context, I first just tried with the version of docker I had running on this laptop and it failed with heap space again. So I re-installed Docker using the version native to Apple chips, and it worked. So I think we are good to go.