jambrain / dairybox

Local development VM and tools for Jammer Core

Jammer Core

Jammer Core is the software that runs Ludum Dare game jam events. It is currently live on ldjam.com

NOTE: For better clarity, we are rebranding this software to Jammer Core (formerly ludumdare). The Ludum Dare game jam is still Ludum Dare, but to help reduce confusion with everything else called Ludum Dare, the software powering Ludum Dare has a new name.

Want to help out? Setup Instructions are here:

https://github.com/JammerCore/DairyBox

Development discussion (site-dev only please):

• Web: https://gitter.im/ludumdare/ludumdare (Slack-like)

Structure

Many folder are documented. Just browse the tree to learn about the contents.

Source Code

Source code is found here:

• /src - Source Code (PHP, JavaScript, CSS, etc)

Live sites

These folders contain the live sites that are served. They tend to be simple PHP files that include things from the /src folder. They also contain the output of the toolchain (in the /-/ subfolder).

• /public-ludumdare.com - Ludum Dare focused version of the common site -- https://ldjam.com
  • This will be moving to ludumdare.com eventually.
• /public-url.shortener - A variety of URL shortening services. -- https://ldj.am, https://jam.mr
  • At the moment only https://ldj.am/$id is supported, where $id is the node_id to redirect to. e.g. https://ldj.am/$11

These are not currently in active development.

• /public-jammer.vg - Jammer, the general version of the common site -- https://jammer.vg
• /public-jammer.bio - Jammer Bio's, a variant focused on user pages. i.e. jammer.bio/your-user-name

Other

• /public-api - Restful API calls. https://api.ldjam.com
  • NOTE: /vx/ is experimental and may change. We are working towards a /v1/ launch soon.
• /public-static - Where static files go. Images, etc. -- https://static.jam.vg
• /sandbox - Old code, experiments, and debugging tools. Very chaotic here.

The Makefile

The Makefile is the core build script. It is used to build the project. Invoke it with the make command from inside the VM (or outside with appropriate config).

Usage:

• make - Compile all changed files and build the target
• make clean - Delete all intermediate files. NOTE: When you make again, everything needs to be recompiled
• make lint - Run all code through the linter. NOTE: make only lints files that have changed. This lints everything
• Advanced build options
  • make all - Build all targets (default, until you set TARGET in config.mk)
  • make TARGET=public-ludumdare.com - Make a specific target (in this case, public-ludumdare.com)
  • make mini - Like make, but refreshes the UID (used to bypass caching proxies, etc)
  • make clean-some - Clean, but don't delete the output files (useful when clean building on live)
  • make clean-all - Clean all targets
  • make clean-svg, make clean-css, make clean-js - Clean specifically the SVG, CSS, or JS
  • make clean-all-svg, make clean-all-css, make clean-all-js - Clean specifically the SVG, CSS, or JS of all targets
  • make lint-all - Lint all code for all targets
  • make lint-css, make lint-js, make lint-php - Lint specifically the CSS, JS and PHP code
  • make lint-all-css, make lint-all-js, make lint-all-php - Lint specifically the CSS, JS, or PHP code of all targets
  • make clean-lint - Force re-linting of everything on build
• config.mk - Create this file and you can hardcode makefile settings
  • TARGET=public-ludumdare.com to build just Ludum Dare
  • DEBUG=true to enable debug builds (append ?debug to the URL)
  • SOURCEMAPS=true to enable Source Maps. NOTE: Requires a full rebuild when changed
  • JOBS=2 to compile using 2 threads (parallel build)
    • NOTE: VirtualBox is really bad at threads. Beyond 2, you start seeing serious diminishing returns, to the point where it actually gets worse than 2 the higher you go.
    • If you actually want to try values other than 1 and 2, it you'll need to change the number of CPUs in Vagrantfile, as well as set JOBS.
    • Below is a list of benchmarks, testing different combinations of CPUs and Jobs. Testing on an AMD FX-8350 (8 core, which is not Hyperthreaded, but rather has 2 cores per die that share caches)
  • NOCOLOR=1 to disable color escape codes from output

config.mk JOBS benchmarks

C,J  Sec   Change Diff
-----------------------
1,1: 901 = 100% = 0%
2,2: 521 = 173% = 73%
3,3: 427 = 211% = 38%
4,4: 401 = 225% = 14%
8,8: 824 = 109% = -116%

dairybox's Issues

Move setup instructions to the WIKI

192.168.48.48 is outside the recommended IP range on Linux

I stumbled across this error spinning up the VM on a new machine (Pop! OS/Ubuntu).

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'scotch/box'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'scotch/box' version '3.5' is up to date...
==> default: Setting the name of the VM: ludumdare_default_1643296873812_21212
==> default: Clearing any previously set network interfaces...
The IP address configured for the host-only network is not within the
allowed ranges. Please update the address used to be within the allowed
ranges and run the command again.

  Address: 192.168.48.48
  Ranges: 192.168.56.0/21

Valid ranges can be modified in the /etc/vbox/networks.conf file. For
more information including valid format see:

  https://www.virtualbox.org/manual/ch06.html#network_hostonly

I couldn't find an /etc/vbox/networks.conf file (good job Oracle), so for usability we might want to compromise.

This is easy to fix by editing VagrantFile and changing the IP to say 192.168.56.56, and updating the documentation here and in the ludumdare repository. 48 was a special number though, and I'm sad we need to find an alternative.

A /21 subnet lets us pick any IP in the range of 192.168.56.1 to 192.168.63.254. Mike the Commodore 64 nerd would have settled on 64 (192.168.64.48), but alas this is out of range.

Tentatively considering 192.168.56.48, but again, all documentation will need to be updated.

Based on Scotch/Box, based on Xenial (Ubuntu 16.04), both no longer supported

DairyBox is based on a VM Scotch/Box, which is no longer supported or developed.

Scotch/Box is based on Ubuntu 16.04, whose "5 year support" period ended last year.

The solution here would be find if someone else picked-up the reigns, creating a Scotch/Box like Vagrant box based on a more recent Ubuntu, or to create our own.

At some point we will need to create our own (and I have, but I've run into problems maintaining Vagrant boxes), but in the short term switching to a different 3rd party alternative could get the job done.

404 when running box update

Can't access site?

I've set up according to the walkthough and ran the makefile but I can't access the site though http://ludumdare.org. The error message says 'Server not found.' and happens in IE and Firefox.
Navigating to the VM IP (192.168.48.48) yields only the directory index.
I know I'm missing something, but what is it?

Configure Apache to make ports use API and STATIC URLs

Ludum Dare will have not only a www.ludumdare.com, but an api.ludumdare.com and a static.ludumdare.com (perhaps even a tv.ludumdare.com). We need a way to access these services using DairyBox.

I'm thinking I should configure Apache's 808x ports to access these other services.

192.168.48.48:8080 - api.ludumdare.com
192.168.48.48:8081 - static.ludumdare.com
192.168.48.48:8082 - tv.ludumdare.com
192.168.48.48:8083 - scene.ludumdare.com

Mail catcher doesn't work given the default set up

Following the instructions on the page, I was unable to get mailcatcher to 'catch' any emails. After building the site and running mailcatcher using the script, I clicked the new account button, but the mailcatcher page still didn't show any emails. The user was created in the database, and I was able to activate an account using the activation url from the database, however. I tried this for two separate email addresses, to no success.

Mount 'node_modules' folder by default

• @sgstair

crazy idea: mount node_modules

In our testing this is a significant performance upgrade when compiling.

I got 8m52s --> 6m24s
and @sgstair got 16m59s --> 9m54s

Document "Table Create" procedure

Oh ha. I haven't documented how to update the database tables yet.

The simple way is to vagrant destroy then vagrant up after. But the faster way is this:

vagrant ssh
cd www/src/shrub/tools
sudo ./table-create

It'll prompt you, say yes.

Windows Vagrant issue

I followed the instructions to step 3, but I get the following error.

Which results in the APCu Extension not being installed (or similar)

I'm using a WIndows 8.1 x64 machine

Ubuntu 14 is out of standard support

• packages may not be updated or available anymore.
  • ppa:ondrej/php no longer provides php7 for ubuntu 14

DairyBox Next

Well it's been a few years and much of the software we using is out-of-date or unsupported (see #69).

This issue collects some notes and thoughts on what to do next, and what we need.

Post Scotch/Box

In the absence of a replacement for Scotch/Box, we could make a new setup script that starts from a stock Ubuntu vagrant box, and installs/configures everything as we need it. The downside is setup time would be much longer, but frankly we've added so much to Scotch/Box anyway that it might not be that much worse.

Vagrant works best with images that don't need to do much after they've been instanced. That's the opposite of what we've done.

It would be nice if DairyBox only contained Jammer Core setup specifics, and didn't need all the hacked-in software updates.

JuiceBox vs DairyBoxSource

We could create a monolithic script that not only does the system setup, but all the Jammer Core setup as well. This would solve a lot of our problems, and potentially make upgrading to newer Ubuntu versions far simpler (just change the base-box). The downside is every fresh build would be slow, and it would be proprietary. I call this the "DairyBoxSource" approach.

Alternatively we could create a more work for ourselves, isolate the system setup from the Jammer Core setup, and make a general purpose box. The box would be driven by our needs, but anyone could use it for their own projects (even us). Scotch.io did us a huge favour creating and sharing Scotch/Box, and it would be cool if we did the same.

I've dabbled with this in the past, taking Scotch/Box and upgrading it. I also intended to do this with the original JuiceBox, but in practice Scotch/Box was good enough. I still like the name JuiceBox, so I'll call this the "JuiceBox" approach. 😋

I'm conflicted. I think the JuiceBox approach is better, but it would be much easier to get the DairyBoxSource approach working. That said, gutting DairyBoxSource and splitting it into JuiceBox and DairyBox later could also work. It would be nice to just skip the DairyBoxSource approach, but deadlines are looming.

Needs

Here's a rough list of what the new "box" needs.

Server Needs

• Latest Ubuntu (20.04 today, 22.04 after April)
• Nginx (instead of Apache)
• PHP 8.x with APCu
• Redis (NOTE: only used as a better PHP session cache)
• MariaDB
• ImageMagick, FFMPEG, and other image optimizers

Tooling needs

• GNU Make
• Nodejs
• JavaScript linter and other build tools
• Less and other CSS build tools
• SVG build tools
• Typescript tools (future use)
• Rustup and Cargo (future use)
• [ ] Sass (future use)

Other needs

• Something for database hacking (PHPMyAdmin?)
• Something for outgoing email capturing and review
• Something capable of emulating Amazon S3 object storage (CEPH?)
• Search database (Sphinx?)
• Something to make PHP easier to debug

PHP 7.0 not found during Vagrant up

Looks like there is something wrong with either the php7.0 package or its PPA.

This resulting in the VM serving the PHP files as plain text rather than running them, breaking the toolchain completely.

errors during vagrant up

  default: Fetched 23.2 kB in 2s (10.0 kB/s)
    default: Reading package lists...
    default: W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://packages.blackfire.io any InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 696DBE66A72D76DA
    default: 
    default: W: Failed to fetch http://packages.blackfire.io/debian/dists/any/InRelease  
    default: 
    default: W: Some index files failed to download. They have been ignored, or old ones used instead.
    default: Reading package lists...
    default: Building dependency tree...
    default: 
    default: Reading state information...
    default: E: Unable to locate package php7.0
    default: E: Couldn't find any package by regex 'php7.0'
    default: E: Unable to locate package php7.0-mbstring
    default: E: Couldn't find any package by regex 'php7.0-mbstring'
    default: E: Unable to locate package php7.0-mysql
    default: E: Couldn't find any package by regex 'php7.0-mysql'
    default: E: Unable to locate package php7.0-xml
    default: E: Couldn't find any package by regex 'php7.0-xml'
    default: E: Unable to locate package php7.0-opcache
    default: E: Couldn't find any package by regex 'php7.0-opcache'
    default: E: Unable to locate package php7.0-gd
    default: E: Couldn't find any package by regex 'php7.0-gd'
    default: E: Unable to locate package php7.0-curl
    default: E
    default: : Couldn't find any package by regex 'php7.0-curl'
    default: E: Unable to locate package php7.0-zip
    default: E: Couldn't find any package by regex 'php7.0-zip'
    default: E: Unable to locate package php7.0-redis
    default: E: Couldn't find any package by regex 'php7.0-redis'
    default: E: Unable to locate package php-apcu
    default: Module php5 disabled.
    default: To activate the new configuration, you need to run:
    default:   service apache2 restart
    default: ERROR: Module php7.0 does not exist!
    default: cp: cannot create regular file ‘/etc/php/7.0/apache2/conf.d/’: No such file or directory
    default: cp: cannot create regular file ‘/etc/php/7.0/apache2/conf.d/user.ini’
    default: : No such file or directory

Toolchain fails to build preact 8.1.0

With a fresh Vagrant VM I get the following error during a make:

rollup -c src/config/rollup.config.js .output/public-jammer.vg/main-vg.es6.js > .output/.build/public-jammer.vg/buble.js.tmp
🚨   Could not resolve '../dom' from /vagrant/www/.output/external/preact/vdom/diff.es6.js

I tried tinkering with plugins in the rollup config file, without success.
As a workaround, downgrading preact to v7.2.0 fixed the issue.

MariaDB instead of MySQL in VM

PHP 7.x instead of 5.6 in VM

$NPM_INSTALL_ARGS are only set if using Cygwin, not Git Bash

$NPM_INSTALL_ARGS are only set if using Cygwin, not Git Bash

Issue

npm install fails when doing vagrant up from git bash on windows. Giving the errors below

==> default: npm ERR!
==> default:  Linux 3.13.0-55-generic
==> default: npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
==> default: npm ERR! node v5.0.0
==> default: npm ERR! npm  v3.3.6
==> default: npm ERR! path ../acorn/bin/acorn
==> default: npm ERR! code EPROTO
==> default: npm ERR! errno -71
==> default: npm ERR! syscall symlink
==> default:

Cause

Currently if your on windows and using Cygwin then we use the --no-bin-links flag to avoid issues with windows weird handling of links. But we don't do this if your on windows and using git bash.

if [ `uname -o` = "Cygwin" ]
then
	NPM_INSTALL_ARGS=--no-bin-links
else
	NPM_INSTALL_ARGS=
fi

Possible Solution

So we should probably change it to this

if [ `uname -o` = "Cygwin" ]||[ `uname -o` = "Msys" ]
then
	NPM_INSTALL_ARGS=--no-bin-links
else
	NPM_INSTALL_ARGS=
fi

I haven't tested this yet so it may not fix the issue. If it works when i test it i'll make a pull request. But there a few other issues to deal with.