Document for EvaP Installation and Set-Up for MacOS Silicon (M1/M2) System Architecture about evap

[richardebeling]

Hi @omrichashper, thanks for opening this issue.

tldr: Could you elaborate on your second point, as this is most likely what we want to fix here:

2. The three Docker commands identified in the readme.md file did not work

What exactly did not work, what errors occurred at what step? I'd guess most likely something in vagrant provision failed, can you see which command caused the issue there?

---

Regarding 1

We're aware of the issues with Apple Silicon hardware and its incompatibility with VirtualBox. VirtualBox announced that they will at some point support Arm64 (M1/M2) hosts, and their download page gives a download link for a developer preview, but we do not know when there will be a stable version available. Until then, Apple Silicon users will have to fall back to docker.

Regarding 2

Since #1726, we aim to allow using docker for hosting the development VM, and afaik we currently have a few developers that use docker for development, both on x86 and ARM hosts. Our main goal is to get a development machine up and running for new developers with as little hassle as possible. As you've seen in the readme, we aim to provide a fully set-up development VM using the three commands listed there:

1. vagrant up --provider docker should start a docker container with the ubuntu focal base image.
2. vagrant provision should then run our setup script inside of that container. This is most likely the part where errors occur, and they might be a bit tricky to spot because there is a lot going on.
3. vagrant ssh then is overridden to just execute docker exec -it bash, giving you an interactive shell.

docker-compose should not be necessary, it should suffice if you have docker installed. The steps that are listed in your installation guide are supposed to be done by the provisioning script:

• python3.8 should be installed and a venv should be created.
• apache2 and a wsgi plugin should be available.
• The apache configuration should be copied from our template
• Our npm dependencies, including typescript, should be installed. For the typescript tests, this should also install a headless browser.

Overall, your guide indicates to me that some of this didn't work on your machine. Could you maybe provide a more detailed error so we can try to fix that?

(Edit: Maybe useful: If you run vagrant destroy, this should completely delete the development VM / docker image, so that the three commands from above can be used to trigger the setup again, hopefully triggering the same error again)

Regarding 3

We tried to keep our onboarding-material as short and simple as possible and automate as much as we can. This way, we intend our provisioning script to be a kind of executable documentation for the setup. I understand that this is not very friendly for new external contributors if stuff is not working as its supposed to be, and that it's not very discoverable. We could add a note to the current wiki entry. The wiki entry currently basically isn't intended for new contributors.

One problem here is that we have limited time, and when we add a new wiki entry, we have to keep it up to date with any changes that we do to our provisioning script. Another problem is that the full setup requires a lot of steps, so even a human-readable documentation would likely be lengthy and confusing to new contributors. (The two-pronged approach that we currently have with docker and virtualbox is not optimal for this, either. We have not yet fully made the leap to docker because it's missing some convenience that virtualbox had, e.g. availability of common system tools inside the container.)

Thanks for sharing your guide with other peers, looking forward to more contributions from Drexel :)

[omrichashper]

Hi @he3lixxx, thanks for the quick response! Some quick notes:

1. Even though I ran vagrant provision several times, I still had to manually create the elements I portray in my document.
2. In addition, I have utilized the vagrant destroy command several times to retrace/redo/test my steps/efforts. When re-entering the vagrant commands, I still experienced the behaviors, that I will explain below, with these vagrant commands.

In regards to the vagrant commands, below is a summary of what I experienced.

First, I ran: vagrant up --provider docker, which resulted in the following error:

The provider 'docker' that was requested to back the machine 'default' is reporting that it isn't usable on this system. The reason is shown below:

The executable 'docker' Vagrant is trying to run was not found in the PATH variable. This is an error. Please verify this software is installed and on the path.

As you mentioned, docker-compose might not be necessary. When I got this error, the first thing I tried was to install docker-compose, and that did not fix the error on its own. This error was not fixed until I installed the Desktop Application for Docker. Once this error was resolved, I continued on with the next command.

Next, I ran vagrant provision, which ran until it fully completed. However, it did spit out an error regarding the chromium browser. Later in my set-up experience, I realized I needed to fix this issue by running the following commands in the venv (after entering vagrant ssh):
sudo apt install chromium-browser and npm install typescript

Finally, when running vagrant ssh, I did enter the venv, but when I went to localhost, I got a "Page Not Found" error and it indicated that localhost was not passing any data. At this point I asked my group mates to try entering python3 manage.py run (because I didn't attempt this), as I immediately kept researching possible solutions and executing them. They communicated to me that it still did not work, and that they needed to go about the steps I included in my document that proceed that step. Unfortunately, I can't provide the error they got from trying this, as I am not sure what it was.

I completely agree with and understand your points on keeping on-boarding material short, simple, and as automated as possible. Regardless of whether you chose to include my document or not, this experience was very rewarding and educational for me, and I hope this response and the information in my document is helpful to you and the rest of the EvaP community. If new, potential contributors are really interested and eager to work on this project, they should still be able to find a solution to make it work with the material you currently provide, just as I did :). Thanks again!

[richardebeling]

The executable 'docker' Vagrant is trying to run was not found in the PATH variable. This is an error. Please verify this software is installed and on the path.

This just says that docker was not properly installed. Installing the "Desktop Application for Docker" should solve this (as it did for you), but brew install docker or any other means should also work. Basically, docker --version must run successfully in the same terminal where you're running vagrant.

It would be nice if you could post the exact error regarding chromium here, I do not currently get such an error. I think we've had cases in the past where the internet connection was flaky during that phase of the setup, so downloading chromium failed, and it gave an error. In any case, this shouldn't be a big issue: You only need chromium to run the typescript tests. The rest of the development VM should work fine without it.

Yes, after running vagrant ssh, you still need to run ./manage.py run to start the development server. Without it, you'd get a "Connection was reset" or "Connection refused" error in the browser at http://localhost:8000. I wouldn't expect "Page not found" here, as it indicates a DNS look failure to me, but localhost shouldn't fail DNS resolving. If this really was the error, I'd double-check that there were no typos in "localhost". The page should load successfully as soon as the command prints "Quit the server with CONTROL-C."

---

So far, I can see two possible improvements to our readme file here:

1. Make it more clear that docker is meant to replace virtualbox -> it needs to be installed (by whatever means, we can link docker's installation wiki page)
2. Make it more clear that the three vagrant commands are meant to replace the two vagrant commands from the virtualbox-workflow, and all the other stuff before and after still needs to be done. (Basically, up and ssh remains, but you manually need to run provision, which is done automatically with virtualbox.)

What do you think?

---

If any of the setup commands from our setup script fail for you, we're also eager to fix that, but we need specific errors to debug it. Both the docker and the virtualbox setup currently work as expected for me on Linux. Maybe @FranzSw or @fidoriel could double-check this on Apple M1.
(I noticed that the docker setup fails to restart apache2, but I think that's not our fault, and apache2 is not required for most developers, so it's probably fine for now and will resolve on its own in the future)

[omrichashper]

1. In regards to installing Docker, I did attempt to install it with homebrew, but then I still got the error I mentioned. Then I installed docker-compose, and finally the Docker desktop App for MacOS Silicon (which resolved my issues regarding vagrant up --provider docker).

2. For the error regarding chromium after running vagrant provision, I believe it simply stated that it couldn't install chromium or chromium-browser, which is why I had to manually install it and typescript in the venv.

3. I'd like to mention once more that I asked my group mates to skip over all my extra steps after entering the venv and just try running python3 manage.py run, and it didn't work for them. As a result, they replicated the steps I took to get around it. However, I will admit that I am not confident that all those steps need to be taken, for example, manually creating the apache.conf file. As you explain, perhaps if I exited the venv and ran vagrant provision once more, it would have taken care of it for me among other issues that I faced throughout the process.

4. Finally, in regards to the modifications to the readme.md that you mentioned, I wouldn't say that it was unclear (at least to me and my group mates) that Docker would replace VirtualBox as the provider, nor was it unclear that the three vagrant commands were replacing the two in the VirtualBox workflow. The issues I experienced in my installation/set-up process were related mostly to the commands not working without external research/efforts, filling in gaps/completing unidentified pre-requisites, and difficulty differentiating between what I do need to do and what I don't.

I hope this feedback is helpful, unfortunately I don't really have anything else to report. Drexel is a fast-paced school, and I am now heading into Finals week. If I find time, perhaps I can try to retrace/retry my installation/set-up experience, see if I experience the same problems, and try some of your suggestions before jumping into my extra steps. Once again, thank you for taking a look at and considering my Wiki document, answering all of my questions and concerns, and for responding to me so quickly :).

[fidoriel]

chromium binaries are not avaliable for arm64:

npm ERR! code 1
npm ERR! path /evap/node_modules/puppeteer
npm ERR! command failed
npm ERR! command sh -c node install.js
npm ERR! The chromium binary is not available for arm64.
npm ERR! If you are on Ubuntu, you can install with:
npm ERR!
npm ERR!  sudo apt install chromium
npm ERR!
npm ERR!
npm ERR!  sudo apt install chromium-browser
npm ERR!
npm ERR! ERROR: Failed to set up Chromium r1011831! Set "PUPPETEER_SKIP_DOWNLOAD" env variable to skip download.
npm ERR! Error
npm ERR!     at handleArm64 (/evap/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:138:11)
npm ERR!     at BrowserFetcher.download (/evap/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:283:13)
npm ERR!     at async downloadBrowser (/evap/node_modules/puppeteer/lib/cjs/puppeteer/node/install.js:65:5)

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/evap/.npm/_logs/2023-03-15T17_49_59_729Z-debug-0.log`

this is the issue on their repo: puppeteer/puppeteer#7740

this worked for me => I am working on a readme PR.

export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
export PUPPETEER_SKIP_DOWNLOAD=true
npm install typescript

It might might be a good idea to go docker only for the docker setup and skip vagrant for simplicity and prone to errors.
EDIT:
I am not sure if both env vars are needed but one did not work for me; and I do not have time to debug.

[richardebeling]

@omrichashper We tried to improve the readme in #1902, and changed the provisioning script to also install chromium to circumvent the puppeteer error. @fidoriel just tested this setup locally on an M1 macbook.

I'm closing the issue for now. If anything still breaks for you or there's additional stuff we can improve, feel free to open another issue.