worldql / mammoth

Scale a single world horizontally across multiple Minecraft servers.

🪦 This project is no longer maintained. Check out MultiPaper.

Mammoth: Horizontally scalable Minecraft server

A Spigot plugin demonstrating WorldQL's database and message broker.

What is Mammoth?

Mammoth uses WorldQL to scale a single Minecraft world across multiple server processes. Running multiple Minecraft server processes allows for better core utilization and allows for more players to enjoy a single world. A collection of Minecraft servers using this plugin to sync with a WorldQL server is called a Mammoth cluster.

Mammoth has two modes:

• Sliced mode: The world is divided on server boundaries. Areas near server boundaries are synchronized. When a player crosses a server boundary they are transferred alonside their inventory, effects, any ridden mobs, or nearby villagers.
• Seamless mode (experimental): The world is not divided any block changes and player movement are synced.

How to use Mammoth

1. Set up WorldQL. Download the latest release for Linux from https://github.com/WorldQL/worldql_server/actions and extract it into a folder.
   1. Set up a database using the commands below (change the password):

      CREATE DATABASE worldql;
      CREATE USER dbuser_worldql WITH PASSWORD 'example';
      GRANT ALL PRIVILEGES ON DATABASE worldql TO dbuser_worldql;

   3. Create a .env file with the following contents:

    WQL_SUBSCRIPTION_REGION_CUBE_SIZE=16
    WQL_POSTGRES_CONNECTION_STRING="host=localhost dbname=worldql user=dbuser_worldql password=worldql"
   

2. Set up redis-server and run it on localhost.
3. Use https://github.com/WorldQL/mc_provisioner to create your Minecraft cluster. Download the latest version of the Mammoth plugin from https://github.com/WorldQL/mammoth/actions and place it in a plugins folder next to the provisioner executable.
4. Create the servers with ./provisioner init and run them with ./provisioner start.
5. The default config will be copied to all servers in the cluster. You can copy it into your top-level plugins folder and edit it to your desired configuration. Once you've made changes to your plugins folder, you can push it to all servers with ./provisioner stop && ./provisioner sync && ./provisioner start.
6. Create a Velocity proxy server and add all your cluster servers to it with the names mammoth_0, mammoth_1, etc. You can change the server name prefix in config.yml.
7. Start the Velocity proxy and connect to it. It will be running in sliced mode by default with divisions every 100 blocks for demonstration purposes (you should set this to a value over 1000 for actual servers). Place blocks and cross a server border and watch everything sync.

Features

• Cross-server TP.
  • /mtp teleports to another player regardless of their server by looking their position up across the cluster.
  • /mtpa sends an Essentials-style teleport request to a player.
  • /mtpaccept accepts a teleport request.

History of Mammoth

• March 2020: Inspired by large Minecraft events (like Square Garden) and 2b2t, the original goal for horizontally scaling a Minecraft server by splitting the world was realized. I wanted a quarantine project, so I started working on it.
• April 2020: The first version of Mammoth was deployed and played by about 200 people. Game servers would "claim" chunks via a redis server entry. When a player moved between servers, there would be a visible disconnect+reconnect delay. This was overly complicated and is detailed in this Google Doc.
• May 2020: I shut the server down and did some analysis on the world files and how optimally the world was split between nodes (not very). The software had a ton of bugs and could be abused to infinitely duplicate mobs (even the Ender Dragon). I also realized that I didn't need a complex world claiming system, and could simply divide the world via a simple mathematical formula.
• Jun 2020: Released the second version of Mammoth which used a procedural world splitting technique.
• August 2020: SalC1 ran a server crash stream to test the software. I had a small lobby server designed to forward players to the correct cluster server, which immediately crashed. Regardless, over 600 people were able to connect over the course of the night and the project gained a lot of publicity.
• September 2020 - July 2021: Conception and initial development of WorldQL. A complete rewrite of Mammoth to use it.

mammoth's Issues

end chests

When players die, their items should only drop on the server housing the player who dealt the killing blow.

Disable any Minecraft events that don't sync properly by cancelling the events (incl redstone for now)

Also make sure that containers are properly disabled inside the DMZ.

Fix compound and special directional blocks sometimes rendering incorrectly across servers (doors, beds, fences, glass panes, etc)

Stage of development

Hi, I just wondering on which stage the development is right now? Is it still being developed? Can I use it in the current state?

Add wall effect particles to visualize borders

Why? Sometimes players are confused by the world slicing mechanics and don't understand what is happening when they cannot cross server boundaries due to them being on cooldown.

What? Whenever a player bounces off a server boundary because they're on server transfer cooldown, display a wall of particles momentarily that visualizes the server border.

How? Find the dmz line and wall direction (which will be perpendicular to the shove direction) and then use the players current position to know which y value to draw a 4x4 wall of particles at.

bed respawn in right place, and bed respawn doesn't work if bed is destroyed. idk how this will be implemented.

Cross server combat sounds don't work.

[Plugin] Blocks that are broken drop their respective item in creative mode.

Issue

When blocks are broken in creative mode, they drop an item instead of not dropping anything. Leaves will drop sticks or saplings, stone drops cobblestone, etc.

Expected Behavior

Broken blocks in creative mode should not drop any items, besides those intended to, such as chests with items inside of them.

Video of issue

https://streamable.com/4vpani

Create an end crystal using LocalMessage when a player places one.

Move cross-server explosion copies to an ExplosionPrimeEvent listener instead of TNTPrimeEvent.

Note: There is an important difference hetween TNTPrimeEvent (which is Paper only) and ExplosionPrimeEvent. TNTPrimeEvent is called when the fuse of the TNT is ignited, while ExplosionPrimeEvent is actually called just before the actual explosion.

[Plugin] Broken chests duplicate their contents

Issue

When chests are broken, regardless of player gamemode, the items inside of them are duplicated by two.

Expected behavior

When chests are broken, they should only drop the amount of items that they contain.

Video of issue

https://streamable.com/39xl0e

Add command for teleporting one player to another, even if they are on different servers

Why? In slice mode, players on the same world will be on different servers and it can be extremely difficult to teleport to one another as the Minecraft server executing the command may not know the location of the players.

A solution is a new command: /mtp <target-player> <destination-player>. It could work like this on a high level

1. Alice is on server A and Bob is on server B. Alice issues /tp Alice Bob
2. Server A sends a GlobalMessage with replication IncludingSelf with parameter “MinecraftTeleportPositionLookup“ and a flex field including Alice’s in the target field and Bob in the destination.
3. Server B receives this message, looks up Bob’s location, and creates a similar message with parameter “MinecraftTeleport” including Alice’s name and bob’s location.
4. Server A receives this and teleports Alice to the provided coordinates.

Add config file for Postgres connection info and WorldQL listening ports

[Server] memory mismanagement leads to crashing after placing blocks.

Issue

When placing blocks, the WorldQL server crashes immediately.

Video

https://discord.com/channels/701873003679449108/701876824732598414/885880438978138112

GDB Output

GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./WorldQLServerUbuntu...
(No debugging symbols found in ./WorldQLServerUbuntu)
(gdb) set args -server
(gdb) run
Starting program: /home/myokan/worldql/server/WorldQLServerUbuntu -server
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
WorldQL Server 0.0.1 initializing...
[New Thread 0x7ffff6817700 (LWP 11328)]
Connecting to PostgreSQL...
[New Thread 0x7ffff6016700 (LWP 11329)]
[New Thread 0x7ffff5815700 (LWP 11330)]
[New Thread 0x7ffff5014700 (LWP 11331)]
[New Thread 0x7ffff4813700 (LWP 11332)]
[New Thread 0x7fffeffff700 (LWP 11333)]
[HANDSHAKE] Server thread starting...
[HANDSHAKE] Server thread listening...
Connection to database succeeded.
[MAIN] Server thread starting...
[HANDSHAKE] Got handshake from new client 127.0.0.1
[HANDSHAKE] Successfully assigned client tcp://127.0.0.1:29900
[HANDSHAKE] Server thread listening...
[HANDSHAKE] Got handshake from new client 127.0.0.1
[HANDSHAKE] Successfully assigned client tcp://127.0.0.1:29901
[HANDSHAKE] Server thread listening...
Block place
terminate called after throwing an instance of 'std::length_error'
  what():  basic_string::_M_create

Thread 1 "WorldQLServerUb" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7ad0859 in __GI_abort () at abort.c:79
#2  0x00007ffff7d56911 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff7d6238c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff7d623f7 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff7d626a9 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7d59326 in std::__throw_length_error(char const*) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x0000555555565322 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long) ()
#8  0x0000555555567154 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) ()
#9  0x000055555556506a in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long) ()
#10 0x0000555555562bfd in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#11 0x0000555555561bdc in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#12 0x00005555555597c0 in MinecraftBlockPlace::handleMinecraftBlockPlace(WorldQLFB::StandardEvents::Update const*, std::vector<zmq::socket_t, std::allocator<zmq::socket_t> >&, std::vector<int, std::allocator<int> >&, pg_conn*) ()
#13 0x000055555555ad10 in MainServer::runServer(zmq::context_t&, std::vector<zmq::socket_t, std::allocator<zmq::socket_t> >&, std::vector<int, std::allocator<int> >&, pg_conn*) ()
#14 0x000055555555cab9 in main ()

Only listen to block place/break updates for a 50 block area near a region border Also listen for explosion events.

FlatBuffers: object serialization must not be nested.

[03:03:18 WARN]: Exception in thread "Thread-5" java.lang.AssertionError: FlatBuffers: object serialization must not be nested.
[03:03:18 INFO]: Done (6.876s)! For help, type "help"
[03:03:18 WARN]:        at WorldQLClient-1.0-SNAPSHOT.jar//com.google.flatbuffers.FlatBufferBuilder.notNested(FlatBufferBuilder.java:688)
[03:03:18 WARN]:        at WorldQLClient-1.0-SNAPSHOT.jar//com.google.flatbuffers.FlatBufferBuilder.startTable(FlatBufferBuilder.java:744)
[03:03:18 WARN]:        at WorldQLClient-1.0-SNAPSHOT.jar//com.worldql.client.Messages.Message.startMessage(Message.java:48)
[03:03:18 WARN]:        at WorldQLClient-1.0-SNAPSHOT.jar//com.worldql.client.serialization.Codec.encodeMessage(Codec.java:134)
[03:03:18 WARN]:        at WorldQLClient-1.0-SNAPSHOT.jar//com.worldql.client.serialization.Message.encode(Message.java:186)
[03:03:18 INFO]: Timings Reset
[03:03:18 WARN]:        at WorldQLClient-1.0-SNAPSHOT.jar//com.worldql.client.ZeroMQServer.run(ZeroMQServer.java:45)
[03:03:18 WARN]:        at java.base/java.lang.Thread.run(Thread.java:833)

Caused by

mammoth/src/main/java/com/worldql/client/ZeroMQServer.java

Line 45 in 105b663

WorldQLClient.getPluginInstance().getPushSocket().send(message.encode(), ZMQ.DONTWAIT);

Strangely this is intermittent and I can't figure out how to consistently reproduce it. Perhaps improper use of Flatbuffers in the serialization.Message class?

When you respawn after a death, you spawn at the same place you died

Players show up in tab list even after DC. You can see this on the test server currently. This indicates players aren't being cleaned up when they DC, but it only happens sometimes.

Do not create duplicate end crystals when someone places it on another server.

Sometimes players from a different server do not appear. Likely due to logic error in how join packets are sent.

Implement BlockRedstoneEvent and InventoryMoveEvent (hoppers, etc.) locks and nbt sync

Respawn anchor explosions don't sync. Weird because we hook into every entity explosion + end crystals work.

Separate n.m.s packet code by Minecraft version

We should use the approach used by https://github.com/CitizensDev/Citizens2 to manage versions (different folder for each), not reflection.

Send RecordUpdate and LocalMessage block update for changed signs. Should use NBT data setting technique rather than Spigot API. Ideally all blocks and tile entities can be updated through one NBT API.

[Client] Unavailable server causes client to freeze

Expected Behavior

Connection would fail within reasonable time period then log it as an error and disable the plugin or something along these lines

Actual Behavior

Client is stuck in a connection loop and is unresponsive

Could be disruptive if server reboots happen unattended

arrow entities will be copied between server if player is listed as a ghost or ghost recipient.

[Question] Can this be used to just reduce world lag on a single server?

Can WorldQL just be used to move the load of a world on a Spigot instance to a separate sever? I get this is meant to make a single world massively multiplayer, but realistically nobody can render more than 100 players at once without massive lag, and the data transfer rates would be insane. Great option for 2b2t though. But, let's say you wanted to hold 1k people on a single Spigot instance, would using WorldQL make this more possible by moving the world chunk loading to a separate server, or is this meant to be used with multiple servers and bigger setups?

Thanks!

Figure out some way to attribute end crystal pvp deaths to the correct players.

This is a tricky issue even when you're not dealing with cross-server combat.

plan-player-analytics/Plan#1571
https://www.spigotmc.org/threads/ender-crystal-get-killer.474526/

We need this in order to drop blocks for the person who detonated the end crystal that dealt the killing blow.

Add heuristic anti-dupe based on nbt watermarks

Implement NBT lock/sync on container open events for chest/furnace syncing.

player ghost

Bonemeal growth should be cancelled in the DMZ

Add configuration options to enable/disable inventory/vitals sync if users want to use a different plugin.

Contact

Hello, we are a French / EAU company creating minecraft servers for influencers and we would like to get in touch with you to discuss about this project. Do you have a contact ? Here is my discord Hugo - HookWood_#0759

[Server] Provide ARM builds

I run an arm64 (aarch64) based Linux server and being able to run WorldQL natively would be great,

Caused by: java.lang.NullPointerException at io.github.jaxkr.mammoth.PlayerInteractListener.onPlayerJoin(PlayerInteractListener.java:267) ~[?:?]

whatever this is...

The ender dragon is unbeatable because of the 4 quadrants at 0, 0 in the end. This is because there are essentially 4 dragons and 4x the amount of end crystals on top of the obsidian pillars

Sometimes players don't load their data for several seconds and snap back to an old position.

Improve reliability by automatically reconnecting to the WorldQL server if it DCs (no clientbound heartbeats for a while).

Right now the handshake process seems to be very error prone and can cause servers to disconnect. Additionally, if the WorldQL server is restarted the clients have no way to reconnect (without restarting themselves).

Replacing the handshake message with a combined handshake/heartbeat would solve this.
This heartbeat packet would function as an idempotent handshake in which the client says "hey, I'm still here and btw my port and host are x in case you've never seen me before".

Fix exploded blocks (tnt, respawn anchors) not being replaced by air when detonated in the DMZ

Cross server players should die correctly

Fix the death dupe glitches with Mammoth's own redis-based inventory sync.

These can be avoided by switching to MySQLPlayerDataBridge which mitigates this issue by:

• periodically saving players inventories instead of only listening to disconnect events.
• listens to containers being closed as a clue that the inventory needs to be synced.

Disconnecting from the server while in the Nether or the End will save your location to the correct coords, but to the overworld

Noteblock disk play should sync. Noteblocks should be protected like any other chest.

[Client] Players do not immediately broadcast their presence on join.

Expected Behavior

When a player joins on an instance, they should be visible on other instances.

Actual Behavior

Players only appear after rotating their head or moving.

Video

In discord: https://discord.com/channels/701873003679449108/701876824732598414/885879554017755206

Clients do not gracefully handle server faults

When the server runs into halting exceptions, clients do not gracefully handle reconnection.

Video

Check discord: https://discord.com/channels/701873003679449108/701876824732598414/885877307665952809

Expected Behavior

When the server is restarted, existing clients should not experience desync.

Actual Behavior

After server failure, clients remain out-of-sync, leading to strange behavior.

Merge worldguard code into trunk and have it conditionally load if the soft dependencies are installed.

Ensure nether portals work as expected

PortalCreateEventListener is a messy abomination I wrote for the old version of WQL that used parameter arrays to represent Records.

The desired logic is simple, when a portal is created, sync all of the related blocks (obsidian and nether portals) to the other servers. This event is called for the automatically generated exit portal. Exit portal location is random but if a portal is already present it connects to it.

For consistent nether portal mechanics, portals from other servers should be synced regardless of whether they're subscribed to the area. If the PortalCreateEvent sends RecordCreate and GlobalMessage messages for the portal blocks with the "MinecraftBlockUpdate" parameter, it should work nicely.

You'll want to remove the nether portal cancelling code from PlayerTeleportEventListener.

When players reconnect they don't immediately broadcast armor and held items. They have to toggle it for it to appear.