sogno-platform / clonemap

cloud-native Multi-Agent Platform

cloneMAP: cloud-native Multi-Agent Platform

cloneMAP is a multi-agent platform (MAP) that is designed to run in a cloud environment based on Kubernetes. The aim of this project is to combine the advantages of cloud computing and a distributed computing paradigm such as multi-agent systems.

Overview

cloneMAP consists of four modules. Each module is implemented in a scalable and robust way. Where possible, applications are stateless and hence horizontally scalable. State is stored in distributed storages and databases. The single modules can be deployed and used independently. They are coupled by means of REST APIs.

Core

The core components of cloneMAP are the agent management system (AMS), an etcd store and agencies. The AMS (API) is responsible for managing all MASs and the corresponding agents running in cloneMAP. Typical tasks are agent creation, monitoring and termination. To allow the AMS to be horizontally scalable, its state (configuration and status information) is stored in an etcd cluster.

Agencies (API) are the deployment unit for agents. Each agency hosts a specified number of agents. Agencies have the task to aggregate common operations in order to relieve the AMS. In Kubernetes terms an agency is a pod with one container. The agents are single go-routines executed within the agency.

DF

To provide agents with the ability to explore each other the so called directory facilitator (DF) is implemented (API). It consists of the scalable service registry and an etcd store. The same etcd cluster as for the AMS is used. The service registry provides functionalities such as registering, deregistering and searching for services. Hence, agents can us it as yellow pages service.

State and logging

As agents usually have a certain state that changes over time, that state needs to be saved in order to make the platform robust. This is done by module state and logging. A DB service provides agents with a REST API for storing state and logs. Upon restart the agents can query the last state stored in the DB and continue their work from that point. A Cassandra DB is used as backend.

IoT

In order to connect the agents running in cloneMAP to the outside world a message broker is used. Currently only MQTT is available. With the help of this message broker agents can interact with devices outside the cloud (e.g. IoT-devices). This module is called IoT-Interface.

UI

For easy user interaction it is planned to provide a WebUI that interacts with the AMS via its REST API. The WebUI is not yet available.

Getting Started

The following documentation is available for cloneMAP:

• Repository structure
• Developer Guide
• Administration Guide
• User Guide

Documentation of the code follows the godoc specification.

Copyright

2020, Institute for Automation of Complex Power Systems, EONERC

License

This project is licensed under either of

• Apache License, Version 2.0 (LICENSE-Apache or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

Contact

• Stefan Dähling

Institute for Automation of Complex Power Systems (ACS)
EON Energy Research Center (EONERC)
RWTH University Aachen, Germany

clonemap's Issues

Fix outdated docs and typos

Smaller things that have been accumulating

ImagePullErr when trying to run agentlib MAS with k8s

Starting an agentlib MAS works fine with the docker-compose version, but (for me) fails with kubernetes.

    "config":{
        "name":"test",
        "agentsperagency":1,
        "mqtt":{
            "active":true
        },
        "df":{
            "active":false
        },
        "logger":{
            "active":true,
            "msg":true,
            "app":true,
            "status":true,
            "debug":true
        }
    },
    "imagegroups":[
        {
            "config": {
                "image":"agentlibtest"
            },
            "agents":[...]
        }
    ],
    "graph":{
        "node":null,
        "edge":null
    }
}

running the same example with docker-compose, the image gets pulled from registry.git-ce.rwth-aachen.de/ebc/projects/ebc_acs0017_bmwi_agent/agents_python/agentlib:latest just fine. With kubernetes, it results in an ErrImagePull, even if the image is already locally pulled and regardless of whether image-pull-policy is set to "Never" or "Always".

improve error message on invalid image

if an image given in the imagegroups config is nonexistent, clonemap (on the docker version) gives an exit code 125. giving back an "Image not found" message or similar instead would be helpful here.
in some cases instead of the error code, clonemap does falsely report that agencies were created, but I haven't been able to reproduce that problem so far.

Finalize move to Github and extend docu

k8s custom resource definition of etcd is not available anymore

The Custom Resource Definition for the deployment of a etcd cluster on k8s is not available anymore. Moreover, the etcd-operator is deprecated

Finalize WebUI

Finalize the first version of the WebUI and add documentation for its usage

clonemap_local: No meaningful log for `Caught sig terminated` and docker exit code 137

Describe the bug
The clonemap_local kubestub container exits with error code 137, without any meaningful error indicator.
The agencies are not terminated correctly, and all other containers in the docker-compose exit with error code 2.

Expected behavior
Indicate what caused the container to shut down.

Additional content

Logs of kubestub:

Caught sig: terminated
Stop Agency Container mas-0-im-0-agency-0

I have 3 MAS running, from the code I would expect three print statements for agency shutdown.

func (stub *LocalStub) terminate(gracefulStop chan os.Signal) {
	var err error
	sig := <-gracefulStop
	fmt.Printf("Caught sig: %+v\n", sig)
	for i := range stub.agencies {
		agencyName := "mas-" + strconv.Itoa(stub.agencies[i].MASID) + "-im-" +
			strconv.Itoa(stub.agencies[i].ImageGroupID) + "-agency-" +
			strconv.Itoa(stub.agencies[i].AgencyID)
		fmt.Println("Stop Agency Container " + agencyName)
		err = stub.deleteAgency(stub.agencies[i].MASID, stub.agencies[i].ImageGroupID,
			stub.agencies[i].AgencyID)
		if err != nil {
			fmt.Println(err)
			// os.Exit(0)
		}
	}

Container state:

State
Dead false
Error
ExitCode 137
FinishedAt 2023-11-12T14:02:04.413342133Z
OOMKilled false
Paused false
Pid 0
Restarting false
Running false
StartedAt 2023-11-09T17:09:15.202650758Z
Status exited

risk of code injection when creating agencies

The function LocalStub.createAgency creates an agency, using go's exec package. In building the command for starting the agency's container, the name of a configured image is used without being checked first. This could make code injection possible if an image name is chosen containing ;, followed by any command. This command will be executed on the kubestub container.
Depending on how it is set, the same problem may be caused by the log level setting

Agency should notice when Agent dies with an error

Current behavior: When an Agent's task() function ends with a runtime error, it stops running without further consequence

Desired behaviour: Runtime error should be returned to agency and logged properly, and the agency should either come to a graceful stop and restart, or attempt to restart the individual agent

ID of a deleted agent is not removed from agency's list of agents

When an agent is deleted, the list of agents from the corresponding agency is not updated.
This prevents new agents to be scheduled to this agency if it has already reached the maximum number of agents per agency.

curl DELETE does not delete anything

While trying to terminate single agents or full MAS, I found that deleting something does not work.

Two examples:

1. Deleting an agent.

Current behaviour

I put the request requests.delete("localhost:30009/api/clonemap/mas/2/agents/4") and get a response 200.
The corresponding agent received the request: 2022-01-11 07:40:01,106 - [INFO] - Agency: Received Request: DELETE /api/agency/agents/4
Afterwards, I can still access http://localhost:30009/api/clonemap/mas/2/agents/4 and get all data. Also, the docker container is not stopped.

Expected behaviour

The docker container is stopped and the agent with id 4 is deleted.

2. Deleting an MAS.

Current behaviour

I put the request requests.delete("localhost:30009/api/clonemap/mas/0") and get a response 200.
Before this request, running docker ps -a yields

CONTAINER ID   IMAGE                                                                      COMMAND                  CREATED              STATUS              PORTS                                         NAMES
d609977c113c   testdizwionclonemap                                                        "python -u agentlib/…"   9 seconds ago        Up 8 seconds                                                      mas-0-im-0-agency-5.mas0agencies
1357070d0d84   testdizwionclonemap                                                        "python -u agentlib/…"   10 seconds ago       Up 9 seconds                                                      mas-0-im-0-agency-4.mas0agencies
19410bb7b002   testdizwionclonemap                                                        "python -u agentlib/…"   10 seconds ago       Up 10 seconds                                                     mas-0-im-0-agency-3.mas0agencies
2c8b3ebd44d9   testdizwionclonemap                                                        "python -u agentlib/…"   11 seconds ago       Up 11 seconds                                                     mas-0-im-0-agency-2.mas0agencies
147a352a9b38   testdizwionclonemap                                                        "python -u agentlib/…"   14 seconds ago       Up 13 seconds                                                     mas-0-im-0-agency-1.mas0agencies
d08e9659a79c   testdizwionclonemap                                                        "python -u agentlib/…"   15 seconds ago       Up 14 seconds                                                     mas-0-im-0-agency-0.mas0agencies
3335807b57f4   eclipse-mosquitto:1.6.13                                                   "/docker-entrypoint.…"   About a minute ago   Up About a minute   0.0.0.0:30883->1883/tcp, :::30883->1883/tcp   clonemap_mqtt_1
2d25e956e033   registry.git.rwth-aachen.de/acs/public/cloud/mas/clonemap/ams              "./ams"                  About a minute ago   Up About a minute   0.0.0.0:30009->9000/tcp, :::30009->9000/tcp   clonemap_ams_1
9d106a3448d4   registry.git.rwth-aachen.de/acs/public/cloud/mas/clonemap/clonemap_local   "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp     kubestub

The MAS logs the following (I also added two GET requests):

kubestub    | Received Request: DELETE /api/container/0
mqtt_1      | 1641887899: Socket error on client auto-8C05EB97-C8E6-E789-1157-8A4A1424425B, disconnecting.
mqtt_1      | 1641887902: Socket error on client auto-B97DC532-C70A-32B5-5673-5915E5B0AFBB, disconnecting.
mqtt_1      | 1641887904: Socket error on client auto-DFE8AD9A-6AEE-7665-8904-77E3931471C9, disconnecting.
mqtt_1      | 1641887907: Socket error on client auto-3A860F08-4444-28F1-5EF2-A13277D1F795, disconnecting.
ams_1       | [INFO] 2022/01/11 07:59:57 Received Request:  GET   /api/clonemap/mas/1
ams_1       | [ERROR] 2022/01/11 07:59:57 /api/clonemap/mas/1 MAS does not exist
ams_1       | [INFO] 2022/01/11 08:00:00 Received Request:  GET   /api/clonemap/mas/0

docker ps -a yields the correct statement, i.e. all agent containers are removed:

CONTAINER ID   IMAGE                                                                      COMMAND                  CREATED         STATUS         PORTS                                         NAMES
3335807b57f4   eclipse-mosquitto:1.6.13                                                   "/docker-entrypoint.…"   2 minutes ago   Up 2 minutes   0.0.0.0:30883->1883/tcp, :::30883->1883/tcp   clonemap_mqtt_1
2d25e956e033   registry.git.rwth-aachen.de/acs/public/cloud/mas/clonemap/ams              "./ams"                  2 minutes ago   Up 2 minutes   0.0.0.0:30009->9000/tcp, :::30009->9000/tcp   clonemap_ams_1
9d106a3448d4   registry.git.rwth-aachen.de/acs/public/cloud/mas/clonemap/clonemap_local   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp     kubestub

However, the GET request for MAS 0 still gives me all the data.
Also, if I post a new MAS, the ID 0 is not used but instead, ID 1 is created. I think this results from the fact that GET does not return MAS does not exist.

Expected behaviour

The MAS is fully deleted and I can re-use ID 0

@kwe712 @s-daehling :
Could you comment on this error and tell me if it's a bug or actually a feature?

As you can see, I am running it locally and using the following docker-compose:

version: "3.7"

services:
  kubestub:
    container_name: kubestub
    hostname: kubestub
    image: registry.git.rwth-aachen.de/acs/public/cloud/mas/clonemap/clonemap_local
    environment:
      CLONEMAP_LOG_LEVEL: ${CLONEMAP_LOG_LEVEL}
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
    ports:
     - 8000:8000
    networks:
     - clonemap-net
    stop_grace_period: 30s

  ams:
    image: registry.git.rwth-aachen.de/acs/public/cloud/mas/clonemap/ams
    environment:
      CLONEMAP_DEPLOYMENT_TYPE: ${CLONEMAP_DEPLOYMENT_TYPE}
      CLONEMAP_STORAGE_TYPE: ${CLONEMAP_STORAGE_TYPE}
      CLONEMAP_LOG_LEVEL: ${CLONEMAP_LOG_LEVEL}
      CLONEMAP_STUB_HOSTNAME: ${CLONEMAP_STUB_HOSTNAME}
    ports:
     - 30009:9000
    depends_on:
      - kubestub
    networks:
      - clonemap-net

  mqtt:
    image: eclipse-mosquitto:1.6.13
    ports:
      - 30883:1883
    depends_on:
      - kubestub
      - ams
    networks:
      - clonemap-net

networks:
  clonemap-net:
    name: clonemap-net
    attachable: true

with .env

CLONEMAP_STORAGE_TYPE=local
CLONEMAP_DEPLOYMENT_TYPE=local
# mqtt

CLONEMAP_LOG_LEVEL=info
# fe kubestub

CLONEMAP_STUB_HOSTNAME=kubestub

CLONEMAP_MODULE_MQTT=true
CLONEMAP_MODULE_FRONTEND=true