aerospike / aerolab

Quickly spin up Aerospike clusters on docker or in AWS

AeroLab

AeroLab is a tool that creates Aerospike development and testing clusters in Docker or on AWS, streamlining efforts to test cluster configuration options, upgrade procedures, and client applications in a controlled development environment.

NOTE: AeroLab is intended for local development and testing environments. It is not recommended for production operations.

Upgrading

See this document on upgrading from versions 6.0- to 6.1+

NOTE: AeroLab 7.0.0 implements an instance expiry system. By default your instances will terminate after 30 hours. To modify this behaviour, create clusters with --aws-expires TIME or --gcp-expires. For example --aws-expires 50h. To disable expiry, set to 0.

Releases

The releases page contains links to current installer packages for all the supported backends.

Operating System	Package	Notes
macOS	aerolab-macos-*	Native macOS binary, compiled for x86_64 and M series ARM chips
Linux	aerolab-linux-*	Native package for Linux (all x86_64 and ARM64 distros)
Windows	aerolab-windows-*	Native Windows executable. Unzip and run from explorer to install.

Manual build

See this document for manually building AeroLab (not recommended).

Supported backends

• Docker Native and Docker Desktop
  • on macOS
  • on Linux
  • on Windows
• Podman Desktop
  • Install Podman Desktop
  • Enable Docker Compatibility mode in Podman Desktop
  • Use docker backend in aerolab: aerolab config backend -t docker
  • Install full official docker-cli
    • Example MacOS: brew install docker
    • Example Ubuntu: follow the official documentation, but instead of installing the full docker engine, just install docker-ce-cli package
• Podman Native on Linux
  • Follow podman documentation to install podman.
  • To install the full docker cli tool, follow the official documentation, but instead of installing the full docker engine, just install docker-ce-cli package.
  • Enable podman service with: sudo systemctl enable --now podman.service podman.socket && sudo touch /etc/containers/nodocker.
  • Enable the docker repositories: sudo vi /etc/containers/registries.conf and ensure docker is listed on this line: unqualified-search-registries = ["docker.io","localhost"].
  • Use docker backend in aerolab
• AWS
• GCP

Routing to the containers using Docker Desktop

Containers on Docker Desktop cannot be accessed directly by their IPs. For this purpose, AeroLab will automatically attempt to map host ports to container ports.

The containers can then be accessed using IP 127.0.0.1 and the port shown under aerolab inventory list. Aerospike clusters created using this method can be seeded and connected to directly from the desktop computer, using the services-alternate option in either Aerospike tools or in the client libraries.

To disable this functionality and prevent AeroLab from modifying Aerospike configuration files to that effect, create clusters with the --no-autoexpose switch.

An alternative method of access exists on MacOS and Linux, if using --no-autoexpose. See the tunnel container setup instructions for more information about setting up tunneling for direct container access.

Documentation

• Getting started
• Help commands
• GCP
  • Spot instance support
  • Expiry system
  • Extra Volume Support
  • Partitioner
  • Advanced - GCP Firewall Rules
• AWS
  • Spot instance support
  • EFS Support
  • Expiry system
  • Partitioner
  • Advanced - AWS Firewall Rules
  • Advanced - Custom VPC
  • Advanced - Run without PublicIPs
• Docker Multiarch
• Usage examples
  • Basic
  • Advanced
  • Full Stack
• AWS EKS Kubernetes
• DirEnv - different aerolab configuration per directory
• AGI - graphing aerospike statistics from logs
• Deploying clients
  • Basic instance
  • Elastic Search
  • Rest Gateway
  • Trino
  • VSCode
  • Graph
  • Vector
  • AMS monitoring stack
  • Tools and Asbench
• REST API
• Utility scripts
• Volume usage examples
• WebUI Hosted Mode

Changelog

See https://github.com/aerospike/aerolab/tree/master/CHANGELOG/ for version changes

Version

See VERSION.md for latest stable version number

Disabling console colors

Aerolab list commands by default use neutral coloring to compress and present the listing tables of all items. Coloring can be disabled by exporting one of these environment variables:

• export NO_COLOR=1
• export CLICOLOR=0

The following methods work:

1. just for this command: CLICOLOR=0 aerolab cluster list
2. for this terminal session: export CLICOLOR=0
3. forever: add one of the export commands from above your your .zshrc or .bashrc file

aerolab's Issues

Estimated cost of cluster ownership - reporting across many users

We already have estimated running cost in aerolab. Can we have aerolab somehow produce reports in json or CSV for historical data (like for monthly reports).

Make Aerospike Prometheus the default

The latest monitoring stack release eliminates some instance of hardcoding related to the data source. This appears to break dashboard functionality in the UI in some places. This can be fixed by the user by setting Aerospike Prometheus as the default data source. Ideally this can be configured that way when Aerolab configures AMS.

AGI does not aggregate for producing totals

AGI produces graphs based onthe time in the logs. This means, the series may have misaligned data points. This makes it impossible to produce totals on any graph. The request is to allow aggregation-by-time transformation in the plugin. This would take the values in the time interval across multiple series and aggregate by time-range, produce a "total" series.

Add option to monitor agi instance states and react accordingly

aerolab agi-monitor // aws with efs only

• security - at least the request must come from a known AGI instance IP address
• spotMonitor() - if receiving spotTerminate event, wait for spot to be terminated and start a new on-demand
• sizeMonitor() - if receiving a PreProcess COMPLETE notification, estimate RAM and node side requirements; if sizing is required, respond with error code, and terminate instance; start an adequately bigger instance in same instance family

The agi instance itself would need a separate switch to enable this, like --notify-agi-monitor https://... (path to listening aerolab)

This could come with instructions on how to make aerolab deploy itself on a tiny client instance (t3a.micro for example, or maybe nano), and run itself in the agi-monitor mode with a single script.

AGI test different distros on AWS and GCP

Test which OS AGI is faster on on GCP and AWS, and use that:

• ubuntu22.04
• amazonlinux2023
• rocky-linux-9
• debian12

conf generate command for v7 aerospike is missing an argument

Latest rc build test indicates that data-size inside the storage-engine is required, at least for in-memory namespaces. The conf generate command does not currently populate that.

WebUI enhancement ideas

• allow loading a historical command into the form from previous run log or from command-line command (reverse-form-load)
• right-sidebar - quick settings and inventory listing
• make things like security group IDs clickable links to take us to the AWS/GCP console

Need the ability to add specific ports to firewall rules

@hev and I have been testing an application we install on the aerospike node and access via our local machines. The application needs port 5000 exposed. One workaround is to create a firewall rule and use that at cluster creation time but then we lose the ability to lockdown our IP using discover-caller-ip

To be fair we could probably just change the port needed in the application to one that is already exposed.

Clusters can inadvertently join

If IP addresses change on your nodes, and you happen to start cluster B with the same IP that's listed in the mesh-seed-address-port of cluster A, then the cluster will join.
I propose that we add a cluster name of the name you gave aerolab when creating the cluster to avoid this issue.

Add multi-user support to WebUI for hosted aerolab

• jobs list - add "show all users jobs" toggle; add username and email information to each job
  • pendingActionShowAll(): under pending-user-action-list
• implement console for all attach commands
  • also add new installable feature - webconsole - featuring ttyd - enabled by default if installed from webui
  • ttyd local on aerolab, and make that run ssh
• add support for special "owner/user" header to auto-set owner and account the job to a specific user; if header present, do not allow owner edit in webui
• change weblog path: ./weblog/user-owner/items.log
• implement "simple" mode which will have the list of options greatly reduced, present "simple/full" slider
  • definitions should be a list of items, like aerolab config defaults without the values
  • should have a sane default selection
  • user should be able to specify their own list as either replacement, addition or removal
    • eg:
      • +Cluster.Create.NoSetHostname
      • -Cluster.Create.Nodes
      • allow in specification to state:
        • -all - remove all default options and start empty
        • +all - add all options to start (+all without any other will be like a full interface)
  • add option to disable full mode view option toggle altogether
• file choice for upload form type: user to select local path - upload via aerolab or directly (key share?)

net list command does not work when client has same name as server

If a client has same name as server, aerolab's net list command will only read one (overrides the client/cluster name) and result in incomplete results. A major logic rewrite is required for this function to handle clients and clusters with the same name. This might become a non-issue in aerolab once the backend is rewritten, as names may need to be unique then.

Evaluate VictoriaMetrics for AGI replacement

AGI currently uses Aerospike as a backing database for metric storage. While this is really fast, the retrieval is not ordered as this is a NoSQL database, not a timeseries-ordered database. Evaluate using a timeseries database, as long as it can provide all the required (implemented) special features for data selection.

Using custom images with templating is awkward

The cluster node template names are fixed to distroName:distroVersion:aerospikeVersion. As such, if creating a template from a custom image, it is named the same as a template from the standard, discovered image. If one requires to create a cluster using a non-standard image, once, this becomes an issue.

Request: provide an ability to create a cluster as a single flow, skipping template creation, if needed. Or alternative provide an option to custom-name templates and use those custom names. (maybe requiring template create followed by cluster create --template-name xxx type approach). Template names would need to show in template list too.

Implement optional terminal beep

Implement optional terminal beep on all functions, or at least certain long functions (like cluster create, destroy, restart etc).

Maybe a global —beep that can be passed to any command. This would be useful if you are running a large cluster create or a set of commands and want to be notified when that finishes.

Tag expiry systems to allow for version discovery and automatic updating

Tag expiry system somehow with a version, so that automated updating can be performed as required.

Build a Web-UI

• Should have an "easy" mode and "full" mode.
• Should be self-building for the most part, using the structs. The inventory listings obviously are a UI point and cannot self-build per-se.
• Should be using a cache for instance types (already implemented) and for instance lists (due to cloud API limits and possible shared usage).
• Should be allowing multi-backend support (that would be so cool) directly in the UI.
• instance lists should have actions button(s) to interact with cluster.
• Clusters/clients should have option to install ttyd and filebrowser to fully interact via a web browser. Alternatively a proxy web socket?
• Must be multi-user capable and have configuration option to allow to enable/disable functions (like config defaults for multi-user).

AGI: Refactor histograms to be list instead of lots of bins

Combining the 50+ histogram bins into 2 LIST CDT bins would improve histogram graph generation.

What about a Grafana Cloud Integration?

Perhaps, instead of standing up and managing a monitoring stack, we can just hook up logs and metrics to Grafana Cloud? Then you could stand up systems, throw them away, and look at the data (as long as it is retained).

Grafana Cloud pricing:

• 10k metrics billable series, 14 days retention
• 3 active users with Enterprise Plugins
• Logs, Traces, Profiles 50 GBs each, 14 days retention

AGI: consider adding graphs for security.c log message aggregations (like warnings/errors dashboard)

Consider not embedding aerolab binaries in other binaries

Currently to handle arm to amd as well as amd to arm deployments and macos to linux deployments, different binaries need to embed other aerolab versions. Consider not embedding aerolab in aerolab to keep the binary size small. Instead consider adding the different required binaries to the installer deb/tgz/zip/rpm/pkg.

For example MacOS would have:
aerolab
aerolab.linux.amd64
aerolab.linux.arm64

And linux amd64 would have:
aerolab
aerolab.linux.arm64

and linux arm64 would have:
aerolab
aerolab.linux.amd64

Issue: aerolab would need an installer for windows to handle this. We don't currently have one per-se.

Alternative option is that aerolab would not come with alternate versions. It would instead go to the repo releases page and download the versions it needs from the releases API from github (cache for future use on first download). But that feels like an incomplete install for me.

Another alternative is to just leave it as it is.

access-address in aerospike.conf remains with older IP if the cluster is stopped and started in docker steup

The cluster restart fixes the mesh addresses but does not take care of updating the access-address if the container IP had changed. There's no other command option as well to do the same.

$ aerolab cluster start -n srcdc
2023/11/03 20:49:07 Running cluster.start
2023/11/03 20:49:09 Running conf.fixMesh
2023/11/03 20:49:11 Done
2023/11/03 20:49:11 Running aerospike.start
2023/11/03 20:49:11 Done
2023/11/03 20:49:12 Done

Add support for Azure backend

GCP and AWS are already supported, Azure is the next logical step.

Support for concurrent template creation

If launching multiple aerolab instances at the same time (from CI - multiple CI instances), this works as long as the the template has already been created previously, or is being created from aerolab template create prior to the concurrent runs.

If an aerospike version template does not exist, the multiple instances will all fail (maybe except one). Request: make aerolab concurrency-safe for these scenarios.

Consider adding option of exporting inventory in common formats

• ansible or other back-end (gen scripts instead of performing deployments)?
• Connections to other tools - like rsync / ssh / pdsh / pdcp, etc…
• (list -J should include all subcommands we add)

WebUI - add option for specifying multiple regions to use in aws backend in webui

Add option to specify multiple regions in webui when AWS backend is in use. This would translate to polling the inventory from multiple regions, as well as allowing to specify region for certain actions, like cluster/client/agi create/grow commands. WebUI primarily, but could allow this in cli too.

Add asbench UI to aerolab's webui

Add getting-started tooltip walkthrough for first-time users

Make sure it can be turned on manually as well, make sure it can be ignored using a "close" button.

Allow choosing renderer mode for inventory listing to be normal, TSV, CSV, Markdown or HTML

The table drawing library already supports rendering in additional styles (TSV, CSV, Markdown, HTML). Exposing those options would be a good addition for what is essentially THE UI. The TSV would be great for grep and parse (CSV too?) in addition to the already existing json.

Download logs in parallel

Consider support multi-docker installs

Expiry rollover on weekends

It would be nice if AeroLab could detect if expiry that's set falls on a weekend, and, if so, roll over to the next working day.

Consider Tailscale integration

For firewall control, unrestricted access, and potentially login/owner features and cloud access.

Integrate asconfig aerospike configuration and schema validation tool

Ability to add extra NIC's in AWS & GCP deployment

Sometimes more than one NIC is needed, it would be a really useful feature if when deploying into AWS or GCP that you could specify the number of Network Interface Cards to be provisioned.

Monitoring stack should be added by default

This seems potentially more problematic than #99 but I still the trade-offs would be worthwhile. Again, user could opt out but the default should be that when create a cluster, a client is also created called amc (if one doesn't already exist). Again, this will impact scripts created out there and will slow down the standard cluster create but encourages adoption of AMC.

Baremetal support

This can be implemented fairly simply, but allowing a baremetal backend. The backend would take a configuration file which would define the cluster machines available (list of IPs, and ssh keys to access the machines). The baremetal backend would then proceed to support the required functions, except it would keep using the existing machines as defined as an actual cluster, instead of deploying new machines. Aside from cluster start and cluster stop everything else can be easily defined. Stopping and starting physical machines would probably be a bad idea.

add option to bootstrap eksctl container for EKS work

Specific support for aerospike k8s operator

Consider supporting WSL backend

Would be an alternative to the Docker Desktop on Windows.

Cleanup CLI parameters for v8.0

• e.g. --instance, --instance-type
• client config expiry vs. cluster add expiry
• aerolab client destroy --group-name / aerolab client destroy --name
• file upload (-c) doesn’t match the client/cluster
• etc

Consider adding bin convergence in xdr commands

The aerolab xdr connect could have some extra features to allow for bin convergence configuration.

Add support for deploying Aerospike Graph Service using Aerolab

Add option in Aerolab to deploy AGS (Aerospike Graph Service) using Aerolab. Following are the requirements

1. add option to deploy AGS instance(s) locally or in the cloud (GCP and AWS)
2. AGS to be deployed along with a load balancer (like nginx) in front so that user gets one endpoint
3. Add option to specify number of AGS instances to be deployed
4. Add option to deploy AGS instances when spinning up the database OR just add AGS instances with already deployed database cluster

Current product installation instructions
https://aerospike.com/docs/graph/getting-started/installation

UI - Improvement Ideas for Inventory

Some ideas for making the web UI Inventory even more useful:

• copy buttons on fields often needed elsewhere, like (IP addresses, cluster name)
• Deep links to things link the firewall rule, instance, etc... to the console (AWS, GCP, etc...)

Exporter should be added by default

Adding monitoring should not require extra steps, and instead be the default cause for aerolab cluster create. Users could opt out of it, with a flag like '--no-exporter` but the default should be to add it.

Downside, this slows down standard cluster creation and may cause some scripts to throw warnings but seems worth the trade off IMO.

add firefly client deployment

Add firefly client to the list of deployable clients in aerolab client create

Version check logic - handle dev builds

When checking if an upgrade is available, aerolab should consider it's own version lesser than the official version of the same number, if it's own version is a dev build and official version is final release.

Provide a full inventory list for a backend

For AWS and GCP it would be beneficial to have a inventory list --full which would iterate through all enabled regions and list all the inventory items for all regions. This would make exploring what is where easier. It would take a while to iterate, but would allow for a more efficient cleanup.

Cluster and Client list are very wide

Proposal to provide an option to by default shorten the cluster list and client list outputs, with an option of --full for full view. Full view would still have the --no-pager to disable pager and by default would provide a pager. The Shortened list would not use a pager. Proposal: order items in importance and work out what we can display based on screen width. Inventory listing should always show everything.

What is important to users in terms of data they want/need to see?

Rethink client and cluster installation workflow

• Want the supported components and the “best way to extend” aerolab to be basically the same mechanism…
• Establish convention for where to download artifacts (helps with exploring deployed machines)
• node_exporter && Prometheus config (perhaps dynamic config?)
• Can everything be a cluster of node groups (under node-group create/list/etc) and then added capabilities/features to make cluster commands and different client commands?
• Can we add an option to extend features/capabilities with templates aerolab can be fed?
• Introduce Capability concept
• A way for a capability to communicate what OS versions and architectures it needs?
• Remove old client installation procedure (8.0) - or keep client/cluster commands but replace them so they will use the node-group system.

Add truncate graphs to AGI

It would be good to get a truncate graph added to AGI graphs. Relevant log lines are:
Nov 01 2023 13:28:47 GMT: INFO (truncate): (truncate.c:151) {test} got command to truncate to now (436541327298) Nov 01 2023 13:28:47 GMT: INFO (truncate): (truncate.c:355) {test} truncating to 436541327298 on 4 threads Nov 01 2023 13:28:47 GMT: INFO (truncate): (truncate.c:464) {test} done truncate to 436541327298 deleted 5

Add support for slack notifications for AGI instances

Add support for aerolab to send slack notifications on events in AGI