eleutherai / minetest

Minetest is an open source voxel game engine with easy modding and game creation

EleutherAI Alignment Minetest

This is a fork of the Minetest Voxel Engine, designed to support an OAI gym like environment.

Modes of Operation

The environment will eventually provide 4 modes of operation for players and agents

1. Asynchronous real time interaction for agents between client and server. (WIP)
2. Single agent synchronous interaction between a client and server (Done)
3. Multi agent synchronous interaction between a client and server(WIP)
4. Real time recording of player actions (WIP)

Minetester

Minetester is the Python package that exposes Minetest environments via the gym(nasium) interface. After building the minetest executable you can install it with:

pip install -e .

To verify the installation run

python -m minetester.scripts.test_loop

Quick Build Instructions for Linux

Run these make commands in order to build and install minetester. If anything goes wrong during install, inspect the relevant entry/script in the Makefile to see what it's trying to do.

make linux_deps #install debian dependencies, equivalent commands are nessesary for other distros
make python_build_deps #install build dependencies into the local python environment (we reccomend using a venv)
make repos #init submodules
make sdl2 #build sdl2
make zmqpp #build zmqpp
make proto #create c++ and python protobuf files
make minetest #build minetest binary
make minetester #build minetester python library
make install #install python library into local environment along with nessesary dependencies
make demo #run the demo script
make clean #clean up build artifacts

Additionally the makefile supports a utility to clean only the minetester install

make clean_minetester #remove existing minetester install
make minetester #build minetester python library
make install #install python library into local environment along with nessesary dependencies
make demo #run the demo script

Minetest

Minetest is a free open-source voxel game engine with easy modding and game creation.

Copyright (C) 2010-2022 Perttu Ahola [email protected] and contributors (see source file comments and the version control log)

In case you downloaded the source code

If you downloaded the Minetest Engine source code in which this file is contained, you probably want to download the Minetest Game project too. See its README.txt for more information.

Table of Contents

1. Further Documentation
2. Default Controls
3. Paths
4. Configuration File
5. Command-line Options
6. Compiling
7. Docker
8. Version Scheme

Further documentation

• Website: https://minetest.net/
• Wiki: https://wiki.minetest.net/
• Developer wiki: https://dev.minetest.net/
• Forum: https://forum.minetest.net/
• GitHub: https://github.com/minetest/minetest/
• doc/ directory of source distribution

Default controls

All controls are re-bindable using settings. Some can be changed in the key config dialog in the settings tab.

Button	Action
Move mouse	Look around
W, A, S, D	Move
Space	Jump/move up
Shift	Sneak/move down
Q	Drop itemstack
Shift + Q	Drop single item
Left mouse button	Dig/punch/take item
Right mouse button	Place/use
Shift + right mouse button	Build (without using)
I	Inventory menu
Mouse wheel	Select item
0-9	Select item
Z	Zoom (needs zoom privilege)
T	Chat
/	Command
Esc	Pause menu/abort/exit (pauses only singleplayer game)
R	Enable/disable full range view
+	Increase view range
-	Decrease view range
K	Enable/disable fly mode (needs fly privilege)
P	Enable/disable pitch move mode
J	Enable/disable fast mode (needs fast privilege)
H	Enable/disable noclip mode (needs noclip privilege)
E	Aux1 (Move fast in fast mode. Games may add special features)
C	Cycle through camera modes
V	Cycle through minimap modes
Shift + V	Change minimap orientation
F1	Hide/show HUD
F2	Hide/show chat
F3	Disable/enable fog
F4	Disable/enable camera update (Mapblocks are not updated anymore when disabled, disabled in release builds)
F5	Cycle through debug information screens
F6	Cycle through profiler info screens
F10	Show/hide console
F12	Take screenshot

Paths

Locations:

• bin - Compiled binaries
• share - Distributed read-only data
• user - User-created modifiable data

Where each location is on each platform:

• Windows .zip / RUN_IN_PLACE source:
  • bin = bin
  • share = .
  • user = .
• Windows installed:
  • bin = C:\Program Files\Minetest\bin (Depends on the install location)
  • share = C:\Program Files\Minetest (Depends on the install location)
  • user = %APPDATA%\Minetest or %MINETEST_USER_PATH%
• Linux installed:
  • bin = /usr/bin
  • share = /usr/share/minetest
  • user = ~/.minetest or $MINETEST_USER_PATH
• macOS:
  • bin = Contents/MacOS
  • share = Contents/Resources
  • user = Contents/User or ~/Library/Application Support/minetest or $MINETEST_USER_PATH

Worlds can be found as separate folders in: user/worlds/

Configuration file

• Default location: user/minetest.conf
• This file is created by closing Minetest for the first time.
• A specific file can be specified on the command line: --config <path-to-file>
• A run-in-place build will look for the configuration file in location_of_exe/../minetest.conf and also location_of_exe/../../minetest.conf

Command-line options

• Use --help

Compiling

Compiling on GNU/Linux

Dependencies

Dependency	Version	Commentary
GCC	5.1+	or Clang 3.5+
CMake	3.5+
IrrlichtMt	-	Custom version of Irrlicht, see https://github.com/minetest/irrlicht
Freetype	2.0+
SQLite3	3+
Zstd	1.0+
LuaJIT	2.0+	Bundled Lua 5.1 is used if not present
GMP	5.0.0+	Bundled mini-GMP is used if not present
JsonCPP	1.0.0+	Bundled JsonCPP is used if not present

For Debian/Ubuntu users:

sudo apt install g++ make libc6-dev cmake libpng-dev libjpeg-dev libxi-dev libgl1-mesa-dev libsqlite3-dev libogg-dev libvorbis-dev libopenal-dev libcurl4-gnutls-dev libfreetype6-dev zlib1g-dev libgmp-dev libjsoncpp-dev libzstd-dev libluajit-5.1-dev libzmq3-dev libzmqpp-dev

For Fedora users (unsupported):

sudo dnf install make automake gcc gcc-c++ kernel-devel cmake libcurl-devel openal-soft-devel libvorbis-devel libXi-devel libogg-devel freetype-devel mesa-libGL-devel zlib-devel jsoncpp-devel gmp-devel sqlite-devel luajit-devel leveldb-devel ncurses-devel spatialindex-devel libzstd-devel

For Arch users (unsupported):

sudo pacman -S base-devel libcurl-gnutls cmake libxi libpng sqlite libogg libvorbis openal freetype2 jsoncpp gmp luajit leveldb ncurses zstd

For Alpine users (unsupported):

sudo apk add build-base cmake libpng-dev jpeg-dev libxi-dev mesa-dev sqlite-dev libogg-dev libvorbis-dev openal-soft-dev curl-dev freetype-dev zlib-dev gmp-dev jsoncpp-dev luajit-dev zstd-dev

Note that libzmqpp-dev or it's equivalent dependency may be unavailble, in that case libzmqpp needs to be compiled manually by initializing and building the submodule in lib/zmqpp.

Download

You can install Git for easily keeping your copy up to date. If you don’t want Git, read below on how to get the source without Git. This is an example for installing Git on Debian/Ubuntu:

sudo apt install git

For Fedora users:

sudo dnf install git

Download source (this is the URL to the latest of source repository, which might not work at all times) using Git:

git clone --depth 1 https://github.com/minetest/minetest.git
cd minetest

Download minetest_game (otherwise only the "Development Test" game is available) using Git:

git clone --depth 1 https://github.com/minetest/minetest_game.git games/minetest_game

Download IrrlichtMt to lib/irrlichtmt, it will be used to satisfy the IrrlichtMt dependency that way:

git clone --depth 1 https://github.com/minetest/irrlicht.git lib/irrlichtmt

Download source, without using Git:

wget https://github.com/minetest/minetest/archive/master.tar.gz
tar xf master.tar.gz
cd minetest-master

Download minetest_game, without using Git:

cd games/
wget https://github.com/minetest/minetest_game/archive/master.tar.gz
tar xf master.tar.gz
mv minetest_game-master minetest_game
cd ..

Download IrrlichtMt, without using Git:

cd lib/
wget https://github.com/minetest/irrlicht/archive/master.tar.gz
tar xf master.tar.gz
mv irrlicht-master irrlichtmt
cd ..

Build

Build a version that runs directly from the source directory:

cmake . -DRUN_IN_PLACE=TRUE
make -j$(nproc)

Run it:

./bin/minetest

• Use cmake . -LH to see all CMake options and their current state.

• If you want to install it system-wide (or are making a distribution package), you will want to use -DRUN_IN_PLACE=FALSE.

• You can build a bare server by specifying -DBUILD_SERVER=TRUE.

• You can disable the client build by specifying -DBUILD_CLIENT=FALSE.

• You can select between Release and Debug build by -DCMAKE_BUILD_TYPE=<Debug or Release>.

  • Debug build is slower, but gives much more useful output in a debugger.

• If you build a bare server you don't need to compile IrrlichtMt, just the headers suffice.

  • In that case use -DIRRLICHT_INCLUDE_DIR=/some/where/irrlichtmt/include.

• Minetest will use the IrrlichtMt package that is found first, given by the following order:

  1. Specified IRRLICHTMT_BUILD_DIR CMake variable
  2. ${PROJECT_SOURCE_DIR}/lib/irrlichtmt (if existent)
  3. Installation of IrrlichtMt in the system-specific library paths
  4. For server builds with disabled BUILD_CLIENT variable, the headers from IRRLICHT_INCLUDE_DIR will be used.
  • NOTE: Changing the IrrlichtMt build directory (includes system installs) requires regenerating the CMake cache (rm CMakeCache.txt)

CMake options

General options and their default values:

BUILD_CLIENT=TRUE          - Build Minetest client
BUILD_SERVER=FALSE         - Build Minetest server
BUILD_UNITTESTS=TRUE       - Build unittest sources
BUILD_BENCHMARKS=FALSE     - Build benchmark sources
CMAKE_BUILD_TYPE=Release   - Type of build (Release vs. Debug)
    Release                - Release build
    Debug                  - Debug build
    SemiDebug              - Partially optimized debug build
    RelWithDebInfo         - Release build with debug information
    MinSizeRel             - Release build with -Os passed to compiler to make executable as small as possible
ENABLE_CURL=ON             - Build with cURL; Enables use of online mod repo, public serverlist and remote media fetching via http
ENABLE_CURSES=ON           - Build with (n)curses; Enables a server side terminal (command line option: --terminal)
ENABLE_GETTEXT=ON          - Build with Gettext; Allows using translations
ENABLE_GLES=OFF            - Enable extra support code for OpenGL ES (requires support by IrrlichtMt)
ENABLE_LEVELDB=ON          - Build with LevelDB; Enables use of LevelDB map backend
ENABLE_POSTGRESQL=ON       - Build with libpq; Enables use of PostgreSQL map backend (PostgreSQL 9.5 or greater recommended)
ENABLE_REDIS=ON            - Build with libhiredis; Enables use of Redis map backend
ENABLE_SPATIAL=ON          - Build with LibSpatial; Speeds up AreaStores
ENABLE_SOUND=ON            - Build with OpenAL, libogg & libvorbis; in-game sounds
ENABLE_LUAJIT=ON           - Build with LuaJIT (much faster than non-JIT Lua)
ENABLE_PROMETHEUS=OFF      - Build with Prometheus metrics exporter (listens on tcp/30000 by default)
ENABLE_SYSTEM_GMP=ON       - Use GMP from system (much faster than bundled mini-gmp)
ENABLE_SYSTEM_JSONCPP=ON   - Use JsonCPP from system
RUN_IN_PLACE=FALSE         - Create a portable install (worlds, settings etc. in current directory)
ENABLE_UPDATE_CHECKER=TRUE - Whether to enable update checks by default
USE_GPROF=FALSE            - Enable profiling using GProf
VERSION_EXTRA=             - Text to append to version (e.g. VERSION_EXTRA=foobar -> Minetest 0.4.9-foobar)
ENABLE_TOUCH=FALSE         - Enable Touchscreen support (requires support by IrrlichtMt)

Library specific options:

CURL_DLL                        - Only if building with cURL on Windows; path to libcurl.dll
CURL_INCLUDE_DIR                - Only if building with cURL; directory where curl.h is located
CURL_LIBRARY                    - Only if building with cURL; path to libcurl.a/libcurl.so/libcurl.lib
EGL_INCLUDE_DIR                 - Only if building with GLES; directory that contains egl.h
EGL_LIBRARY                     - Only if building with GLES; path to libEGL.a/libEGL.so
EXTRA_DLL                       - Only on Windows; optional paths to additional DLLs that should be packaged
FREETYPE_INCLUDE_DIR_freetype2  - Directory that contains files such as ftimage.h
FREETYPE_INCLUDE_DIR_ft2build   - Directory that contains ft2build.h
FREETYPE_LIBRARY                - Path to libfreetype.a/libfreetype.so/freetype.lib
FREETYPE_DLL                    - Only on Windows; path to libfreetype-6.dll
GETTEXT_DLL                     - Only when building with gettext on Windows; paths to libintl + libiconv DLLs
GETTEXT_INCLUDE_DIR             - Only when building with gettext; directory that contains libintl.h
GETTEXT_LIBRARY                 - Optional/platform-dependent with gettext; path to libintl.so/libintl.dll.a
GETTEXT_MSGFMT                  - Only when building with gettext; path to msgfmt/msgfmt.exe
ICONV_LIBRARY                   - Optional/platform-dependent; path to libiconv.so/libiconv.dylib
IRRLICHT_DLL                    - Only on Windows; path to IrrlichtMt.dll
IRRLICHT_INCLUDE_DIR            - Directory that contains IrrCompileConfig.h (usable for server build only)
LEVELDB_INCLUDE_DIR             - Only when building with LevelDB; directory that contains db.h
LEVELDB_LIBRARY                 - Only when building with LevelDB; path to libleveldb.a/libleveldb.so/libleveldb.dll.a
LEVELDB_DLL                     - Only when building with LevelDB on Windows; path to libleveldb.dll
PostgreSQL_INCLUDE_DIR          - Only when building with PostgreSQL; directory that contains libpq-fe.h
PostgreSQL_LIBRARY              - Only when building with PostgreSQL; path to libpq.a/libpq.so/libpq.lib
REDIS_INCLUDE_DIR               - Only when building with Redis; directory that contains hiredis.h
REDIS_LIBRARY                   - Only when building with Redis; path to libhiredis.a/libhiredis.so
SPATIAL_INCLUDE_DIR             - Only when building with LibSpatial; directory that contains spatialindex/SpatialIndex.h
SPATIAL_LIBRARY                 - Only when building with LibSpatial; path to libspatialindex_c.so/spatialindex-32.lib
LUA_INCLUDE_DIR                 - Only if you want to use LuaJIT; directory where luajit.h is located
LUA_LIBRARY                     - Only if you want to use LuaJIT; path to libluajit.a/libluajit.so
OGG_DLL                         - Only if building with sound on Windows; path to libogg.dll
OGG_INCLUDE_DIR                 - Only if building with sound; directory that contains an ogg directory which contains ogg.h
OGG_LIBRARY                     - Only if building with sound; path to libogg.a/libogg.so/libogg.dll.a
OPENAL_DLL                      - Only if building with sound on Windows; path to OpenAL32.dll
OPENAL_INCLUDE_DIR              - Only if building with sound; directory where al.h is located
OPENAL_LIBRARY                  - Only if building with sound; path to libopenal.a/libopenal.so/OpenAL32.lib
SQLITE3_INCLUDE_DIR             - Directory that contains sqlite3.h
SQLITE3_LIBRARY                 - Path to libsqlite3.a/libsqlite3.so/sqlite3.lib
VORBISFILE_LIBRARY              - Only if building with sound; path to libvorbisfile.a/libvorbisfile.so/libvorbisfile.dll.a
VORBIS_DLL                      - Only if building with sound on Windows; paths to vorbis DLLs
VORBIS_INCLUDE_DIR              - Only if building with sound; directory that contains a directory vorbis with vorbisenc.h inside
VORBIS_LIBRARY                  - Only if building with sound; path to libvorbis.a/libvorbis.so/libvorbis.dll.a
ZLIB_DLL                        - Only on Windows; path to zlib1.dll
ZLIB_INCLUDE_DIR                - Directory that contains zlib.h
ZLIB_LIBRARY                    - Path to libz.a/libz.so/zlib.lib
ZSTD_DLL                        - Only on Windows; path to libzstd.dll
ZSTD_INCLUDE_DIR                - Directory that contains zstd.h
ZSTD_LIBRARY                    - Path to libzstd.a/libzstd.so/ztd.lib

Compiling on Windows using MSVC

Requirements

• Visual Studio 2015 or newer
• CMake
• vcpkg
• Git

Compiling and installing the dependencies

It is highly recommended to use vcpkg as package manager.

After you successfully built vcpkg you can easily install the required libraries:

vcpkg install zlib zstd curl[winssl] openal-soft libvorbis libogg libjpeg-turbo sqlite3 freetype luajit gmp jsoncpp opengl-registry --triplet x64-windows

• Don't forget about IrrlichtMt. The easiest way is to clone it to lib/irrlichtmt as described in the Linux section.
• curl is optional, but required to read the serverlist, curl[winssl] is required to use the content store.
• openal-soft, libvorbis and libogg are optional, but required to use sound.
• luajit is optional, it replaces the integrated Lua interpreter with a faster just-in-time interpreter.
• gmp and jsoncpp are optional, otherwise the bundled versions will be compiled

There are other optional libraries, but they are not tested if they can build and link correctly.

Use --triplet to specify the target triplet, e.g. x64-windows or x86-windows.

Compile Minetest

a) Using the vcpkg toolchain and CMake GUI

1. Start up the CMake GUI
2. Select Browse Source... and select DIR/minetest
3. Select Browse Build... and select DIR/minetest-build
4. Select Configure
5. Choose the right visual Studio version and target platform. It has to match the version of the installed dependencies
6. Choose Specify toolchain file for cross-compiling
7. Click Next
8. Select the vcpkg toolchain file e.g. D:/vcpkg/scripts/buildsystems/vcpkg.cmake
9. Click Finish
10. Wait until cmake have generated the cash file
11. If there are any errors, solve them and hit Configure
12. Click Generate
13. Click Open Project
14. Compile Minetest inside Visual studio.

b) Using the vcpkg toolchain and the commandline

Run the following script in PowerShell:

cmake . -G"Visual Studio 15 2017 Win64" -DCMAKE_TOOLCHAIN_FILE=D:/vcpkg/scripts/buildsystems/vcpkg.cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_GETTEXT=OFF -DENABLE_CURSES=OFF
cmake --build . --config Release

Make sure that the right compiler is selected and the path to the vcpkg toolchain is correct.

Windows Installer using WiX Toolset

Requirements:

• Visual Studio 2017
• WiX Toolset

In the Visual Studio 2017 Installer select Optional Features -> WiX Toolset.

Build the binaries as described above, but make sure you unselect RUN_IN_PLACE.

Open the generated project file with Visual Studio. Right-click Package and choose Generate. It may take some minutes to generate the installer.

Compiling on MacOS

Requirements

• Homebrew
• Git

Install dependencies with homebrew:

brew install cmake freetype gettext gmp hiredis jpeg jsoncpp leveldb libogg libpng libvorbis luajit zstd

Download

Download source (this is the URL to the latest of source repository, which might not work at all times) using Git:

git clone --depth 1 https://github.com/minetest/minetest.git
cd minetest

Download minetest_game (otherwise only the "Development Test" game is available) using Git:

git clone --depth 1 https://github.com/minetest/minetest_game.git games/minetest_game

Download Minetest's fork of Irrlicht:

git clone --depth 1 https://github.com/minetest/irrlicht.git lib/irrlichtmt

Build

mkdir build
cd build

cmake .. \
    -DCMAKE_OSX_DEPLOYMENT_TARGET=10.14 \
    -DCMAKE_FIND_FRAMEWORK=LAST \
    -DCMAKE_INSTALL_PREFIX=../build/macos/ \
    -DRUN_IN_PLACE=FALSE -DENABLE_GETTEXT=TRUE

make -j$(sysctl -n hw.logicalcpu)
make install

Run

open ./build/macos/minetest.app

Docker

We provide Minetest server Docker images using the GitLab mirror registry.

Images are built on each commit and available using the following tag scheme:

• registry.gitlab.com/minetest/minetest/server:latest (latest build)
• registry.gitlab.com/minetest/minetest/server:<branch/tag> (current branch or current tag)
• registry.gitlab.com/minetest/minetest/server:<commit-id> (current commit id)

If you want to test it on a Docker server you can easily run:

sudo docker run registry.gitlab.com/minetest/minetest/server:<docker tag>

If you want to use it in a production environment you should use volumes bound to the Docker host to persist data and modify the configuration:

sudo docker create -v /home/minetest/data/:/var/lib/minetest/ -v /home/minetest/conf/:/etc/minetest/ registry.gitlab.com/minetest/minetest/server:master

Data will be written to /home/minetest/data on the host, and configuration will be read from /home/minetest/conf/minetest.conf.

Note: If you don't understand the previous commands please read the official Docker documentation before use.

You can also host your Minetest server inside a Kubernetes cluster. See our example implementation in misc/kubernetes.yml.

Version scheme

We use major.minor.patch since 5.0.0-dev. Prior to that we used 0.major.minor.

• Major is incremented when the release contains breaking changes, all other numbers are set to 0.
• Minor is incremented when the release contains new non-breaking features, patch is set to 0.
• Patch is incremented when the release only contains bugfixes and very minor/trivial features considered necessary.

Since 5.0.0-dev and 0.4.17-dev, the dev notation refers to the next release, i.e.: 5.0.0-dev is the development version leading to 5.0.0. Prior to that we used previous_version-dev.

minetest's Issues

Fix relative cmake build process

Currently, if you try to build build the project in a sparate directory from the source root directory, cmake raises errors this is related to new logic that was implemented in the top level CMakeList.txt.

Modify client to boot directly into the game without going through the menu

Supporting Julia for minetest

Problem

It would be great to have something like MineRL for Julia. I have worked on Miner.jl which is like a toy minecraft: https://github.com/ashwani-rathee/Miner.jl which is integratable with ReinforcementLearning.jl which provides reinforcement learning tools with Julia. But I want a richer experience which minetest can provide and I want it to be part of the environment available there. I can make c++ libraries available in Julia through https://github.com/JuliaInterop/CxxWrap.jl.

Solutions

What I want to ask is if there is c++ api to the whole game? and are you guys interested in something like this? I would help with development but I need support on c++ side of things

Any usage docs on Windows?

Seems a good project for insteading MineRL/MineDojo, but lack of docs.

Add hooks in client loop to record frames from the minetest client and save them to disk

Multiplayer environments

Problem

It should be possible to let multiple agents play the game.

Solutions

I want the server to be reusable between different environments, or for there to be a flag that turns off server creation if one already exists.

Alternatives

The multiplayer functionality will be inside one environment and the states will be dispatched to agents in the training code.

Additional context

I tried this, there are bugs.

Add support for gpu rendering through nvidia-xconfig

Currently we rely on xvfb for headless operation, this describes a way to modify the runtime to enable GPU rendering, which could accelerate performance.

BUILD_HEADLESS option with "ON" or "TRUE" value has no effect

Summary

The source code checks for headless mode with #if BUILD_HEADLESS. This define is set by CMake to be equal to the value of the BUILD_HEADLESS CMake option, but this is incorrect. CMake options accept "ON" or "TRUE" as truthy values, but the C++ preprocessor does not. CMake should define the symbol, and it should be checked in the C++ source with #ifdef BUILD_HEADLESS.

Steps to reproduce

cmake -S . -B build_issue_24_repro -DBUILD_HEADLESS=ON -DENABLE_SOUND=FALSE
./hacking_testing/compile_proto.sh
cmake --build build_issue_24_repro
./hacking_testing/test_loop.py

When the preceding steps are executed, you should see that the window is still there with all the normal graphics, indicating that the option had no effect.

Fix directory usage in minetest ENV

Currently, when the minetest binary is installed, it makes assumptions about where it's directories are, and this winds up being inside the package structure, which is not ideal as it leaves remenants in the python directory structure when attempting to uninstall the package.

Segfault when exiting to menu

A segfault occurs if a user joins or starts a game and then tries to exit back to the menu.

Create Docker container setup for the project

Synchronize minetest server and client(s)

By default MT clients and server are running and communicating asynchronously.
To see this, run the minetest env test loop with the mod "reward_v1" and don't end on done.
In this mod the server sends a random reward to all clients on each server global step.
However, if you print the rewards you will see that there are a few gaps, where the reward is 0. This is because the client was running faster at that point and read out the global reward variable which is reset to 0 after each readout before a new random reward has arrived from the server.

In general this async interaction can be problematic when the clients construct their observations based on outdated information from the server.
Besides setting up synchronous communication will be beneficial for training since we can then speed up minetest as much as hardware allows.

Hangs after a few actions

Minetest version

b11d2d83939ee67ac66cfb70797841d2fee42e0c (headless-renderer)

Summary

test_loop.py prints out the following and hangs:

Waiting for obs...
Received obs: (0, 0, 3)
Sending action: OrderedDict([('backward', 1), ('dig', 1), ('drop', 1), ('esc', 0), ('forward', 0), ('hotbar_next', 0), ('hotbar_previous', 1), ('inventory', 0), ('jump', 0), ('left', 0), ('middle', 0), ('mouse', array([-882, -607])), ('place', 1), ('right', 0), ('slot1', 0), ('slot2', 0), ('slot3', 0), ('slot4', 0), ('slot5', 1), ('slot6', 1), ('slot7', 1), ('slot8', 0), ('sneak', 0)])
Waiting for obs...
Received obs: (600, 1024, 3)
Sending action: OrderedDict([('backward', 0), ('dig', 1), ('drop', 0), ('esc', 1), ('forward', 0), ('hotbar_next', 0), ('hotbar_previous', 1), ('inventory', 1), ('jump', 0), ('left', 0), ('middle', 0), ('mouse', array([573, 234])), ('place', 0), ('right', 1), ('slot1', 0), ('slot2', 0), ('slot3', 1), ('slot4', 1), ('slot5', 0), ('slot6', 0), ('slot7', 1), ('slot8', 0), ('sneak', 0)])
Waiting for obs...

It's caused by the aforementioned commit, which introduced the deletion of a buffer that was not being deleted due to a syntax error. Maybe we aren't supposed to delete buffer at all. I haven't looked into it much yet.

Steps to reproduce

Compile the headless renderer and run hacking_testing/test_loop.py.

Move scripts folder into util

Currently the scripts folder was created to include minetester scripts, however there is already a scripts folder called utils, minetester specific scripts should be moved into a minetester folder in utils, and obsolete scripts should in the scripts folder should be deleted.

Add CI/CD integration

Self explanatory, builds/tests should happen automatically when opening a PR.

Migrate from Gym to Gymnasium

Currently, minetester uses the old OAI gym library, which is no longer maintained, we should migrate to the https://github.com/Farama-Foundation/Gymnasium Gymnasium API.

Move scripts out of hacking_testing

The hacking_testing folder contains a bunch of different scripts

• for starting minetest clients and servers
• for generating protobuf code
• for the Minetest gym environment
• for data recording
• for testing

We should probably move them into respective folders or otherwise reorganize to make it more legible.

translator: gpt-neox

Write a gpt-neox and Apertium script to automatically check the relevant strings, translate them and write them to the locale.

local S = minetest.get_translator
local S = minetest.get_translator(minetest.get_current_modname())
description =
description = S

Add installation script for gym environment

Fastforward to minetest 5.7.0 and then freeze the build.

The current develop branch forks from the latest minetest code sometime in December about a month after the release of 5.6.1. When 5.6.2 comes out we should fastforward to that commit so that we have a stable environment to build on.

Add documentation for building and usage

Add meta information to protobuf observation

Information that might be useful to have:

• absolute mouse positions
• mouse wheel state
• pitch and yaw of camera
• whether a menu is open
• time stamp
• any new chat messages
• inventory content

A draft of such a protobuf object is here: https://github.com/EleutherAI/minetest/blob/develop/proto/objects.proto#L107

A more flexible approach would be to outsource this to mods.
For example, define a global table variable INFO that is being read out by the Minetest client (like i the case for REWARD and TERMINAL).
Then we could add any information that is available to mods into this table (although for some things, like inventory, this requires setting up mod channels to the server)

For more people to learn about the EleutherAI/minetest project

Please add the content at the beginning of README.md

# Alignment Environments for Minetest: Exploratory Phase Outline


## What is the project?
The goal of this project is to provide a rich and easily moddable environment that alignment researchers can use to test many aspects of alignment and alignment techniques. From there there are a few potential directions depending on feasibility/interest/etc.
-More specialized Challenge environments that grapple with specific problems in alignment
-Datasets and base models that can be used to train/fine-tune models
-Live servers where humans can interact with agents
-A more involved project to develop alignment techniques for agents

## What is the motivation behind this project?
The motivation of this project is to follow in the footsteps of projects like [AI safety gridworlds](https://www.deepmind.com/open-source/ai-safety-gridworlds), [MineRL BASALT](https://minerl.io/basalt/), and [MineDojo](https://minedojo.org/) and then to build on top of them. While currently alignment research is mostly theoretical, we want to be able to “make the rubber meet the road” in an environment that is both rich in terms of the complexity of tasks that can be done, and easy to extend to test any particular situation. Minetest appears very well suited for this.

Specific examples of why we might want this are:

-Current interpretability research is mostly not all that “grounded” and deals with “fully observable” situations, like image classification and text generation. There are no latent factors present that are not directly fed/output by the NN, but we will want to be able to detect/interpret these in a real system.
-Major “alignment tactics” currently employed assume a hard boundary between the AI and the task/environment/human. We can’t fully test this with a minetest environment, but we can get much closer, and therefore verify that our systems don’t have failure modes that are akin to reward hacking.
-None of the existing rl environments based on minecraft are “live”, and so it’s difficult to study human machine interactions with them, which is likely going to be an important facet of future alignment research.

## Who is part of the project?
The primary contact for the project will be @AI_WAIFU, with @JDP as the secondary contact. There is also interest from @ac @FieryVictoryStar @harfe and @triggerhappygandi 
Do you need a separate channel for discussion?
Yes, it should be called “#alignment-minetest”.

## Do you need compute?
Probably not. The Project should be mostly conventional software development, and minetest runs on a potato, so compute likely won’t be necessary for the exploratory phase. We might need some later to train RL agents or to run a server or multiple concurrent instances. 

## What are the deliverables for the exploratory phase of this project?
-A report on the minetest codebase, how it works, and how to modify it.(This is probably just going to be links to existing docs)
-Candidate “AI alignment test setups” that we can use to study different facets of alignment
-A design and roadmap for releasing a full environment
-Design and roadmap for any additional work we want to do on top of the environment
-Full Project Proposal using the new EAI project proposal template that contains the above
-A prototype oai-gym like environment that can launch a gym client, receive fake keyboard/mouse input, and return frames from the minetest game.

## What is the rough timeline for the exploratory phase?
-The goal is for the exploratory phase to take roughly 2-3 months, ending sometime in December 2022.
# [minetest](https://github.com/minetest/minetest)

Fix return to menu when playing the game normally.

Currently, when compiling the binary for minetest, and opening it up to play. The "exit to menu" functionality is broken.

Steps to reproduce:

-Compile minetest
-Create world
-Join world
-Open menu
-Attempt to "exit to menu"
Expected behaviour is to return to the main menu, actual behaviour is that nothing happens.

Improve data recording pipeline

The current data recorder is very basic. It simply dumps the raw data to disk.
It will make sense to store received observations in more structured and compressed formats, like video.
This is a prerequisite to creating datasets for imitation learning.

Memory leak in render core

Summary

GCC describes the problem well.

../src/client/render/core.cpp: In member function ‘void RenderingCore::draw(irr::video::SColor, bool, bool, bool, bool)’:
../src/client/render/core.cpp:91:20: warning: right operand of comma operator has no effect [-Wunused-value]
   91 |  delete tex, buffer;

Support building on ZorinOS with SDL2 from apt

Problem

My current setup for Minetest development is on ZorinOS. It provides development files for SDL 2.0.10 as libsdl2-dev from the Ubuntu repositories, but that package does not include SDL2's CMake target exports, so it can't be used to build your Minetest fork.

Solutions

I'd like to add support to the build for SDL2 packages with no CMake target exports.

Alternatives

We could have CMake fetch SDL2 automatically from git and configure it, or we could add SDL2 as a submodule. I don't like these solutions because SDL2 is a large download, and the shared system library runs perfectly fine in projects that support it.

Automatic auditwheel version detection

Currently there's a problem where if the host linux version is not exactly right, auditwheel will complain, to get around this, the relevant version should be detected automatically when building the wheel.

Currently, things are hard coded to work with just ubuntu 22

cmake . could not find a package configuration file provided by SDL2.

On my arch machine with sdl2 installed, the cmake . -DRUN_IN_PLACE=TRUE command returns the error

> cmake . -DRUN_IN_PLACE=TRUE
-- The C compiler identification is GNU 13.1.1
-- The CXX compiler identification is GNU 13.1.1
[...]
-- Check size of size_t
-- Check size of size_t - done
CMake Error at src/CMakeLists.txt:40 (find_package):
  Could not find a package configuration file provided by "SDL2" with any of
  the following names:

    SDL2Config.cmake
    sdl2-config.cmake

  Add the installation prefix of "SDL2" to CMAKE_PREFIX_PATH or set
  "SDL2_DIR" to a directory containing one of the above files.  If "SDL2"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!

This is after having run the code from https://github.com/minetest/minetest/blob/master/doc/compiling/linux.md

sudo pacman -S base-devel libcurl-gnutls cmake libxi libpng sqlite libogg libvorbis openal freetype2 jsoncpp gmp luajit leveldb ncurses zstd
git clone --depth 1 https://github.com/EleutherAI/minetest.git
cd minetest
git clone --depth 1 https://github.com/minetest/minetest_game.git games/minetest_game
git clone --depth 1 https://github.com/minetest/irrlicht.git lib/irrlichtmt

The minetest version (https://github.com/minetest/minetest.git)

cmake . -DRUN_IN_PLACE=TRUE
make . -j8

Successfully builds bin/minetest after running the above 2 shell scripts.

Fix SDL2 headless mode requiring XServer

While usage of the SDL2 offscreen driver works with machines that run a XServer instance (which is a type of window display)
it does not seem to work with display-less servers (e.g. in google colab https://colab.research.google.com/drive/1KDvWgmhV445JB73WRqGijkegDVlY7e6p?usp=sharing)
Xvfb can be used to create a virtual XServer to work around that for now.
The question is: is it possible to build a headless version that does not need a XServer?

Modify input handler to record actions and timestamps from the player