eclipse-iofog / agent

The ioFog Agent product

Imagine a world where you can choose self-contained pieces of code (called microservices) and make them run anywhere you want at the push of a button. Where you can remotely control the code that is running on twenty iPhones in workers' pockets, thirty servers running in a factory building, and ten computers running in the trucks that ship your products... and you can do it all with the same single technology. Imagine a world where you move the processing close to where the data is happening, and where you can finally separate streams of information from the end applications that use them. This world will be brought to life by iofog, and a big part of the vision is the ioFog product that runs on the various computers.

This repository is the production code base for the x86 Linux version of the ioFog product.

ioFog is a service that runs constantly in the background on a Linux machine. It is the agent that turns a Linux computer into a piece of the iofog I/O compute fog.

There should be an ioFog code base for every processing platform that becomes part of the I/O compute fog. Network connectivity, process invocation, thread management, and other details of an ioFog will vary from platform to platform. The same ioFog principles apply to every version, but the implementation of the principles should match the native languages and structures best suited for the platform.

Status

Principles of an ioFog Agent:

• Never go down
• Respond immediately to the fog controller
• Operate flawlessly when offline
• Report status frequently and reliably
• Execute instructions with no understanding of the bigger picture
• Provide a high-performance message bus and local API
• Enforce the configured resource consumption constraints strictly
• Allow the most flexible and powerful processing element model possible
• Be able to instantiate processing elements from any available source
• Be able to communicate with any reachable fog controller
• Allow processing elements to implement security and connectivity as they would natively
• Ensure that complying with the local API is the only requirement placed on a processing element
• Only shutdown or restart processing elements when requested or when absolutely necessary
• Run only processing elements with verified source and integrity
• Never allow a message to reach unauthorized processing elements
• Only allow messages of the proper registered type to reach processing elements
• Guarantee message source and order

See the docs folder in this repository for architecture, project microculture, engineering philosophy, functional specifications, and more.

ioFog Agent Setup

The entire ioFog platform is best deployed through the unified CLI: iofogctl.

Go to iofog.org to learn how to deploy the ioFog Control Plane and Agents.

Usage

1. To view help menu

    sudo iofog-agent help

2. To view current status

    sudo iofog-agent status   

3. To view version and license

    sudo iofog-agent version

4. To view current configuration

    sudo iofog-agent info

5. Provision iofog for use

    sudo iofog-agent provision ABCDWXYZ

Logs

• Log files are located at '/var/log/iofog-agent'

System Requirements (Recommended)

• Processor: 64 bit Dual Core or better
• RAM: 1 GB minimum
• Hard Disk: 5 GB minimum
• Java Runtime (JRE) 8 or higher
• Docker 1.10 or higher
• Linux kernel 3.10 or higher

Platforms Supported (Ubuntu Linux)

• 14.04 - Trusty Tahr
• 16.04 - Xenial Xerus

 - ioFog Agent Update:

    sudo service iofog-agent stop       
    sudo apt-get install --only-upgrade iofog-agent
    sudo service iofog-agent start
    or
    sudo service iofog-agent stop
    sudo apt-get install --only-upgrade iofog-agent-dev (developer's version)
    sudo service iofog-agent stop        

agent's Issues

Agent always starts in debug mode

The agent always starts in Java debug mode, with a debug port open.

The `iofog-agent` shell script is missing the execute bit

pi@raspberrypi:~/deploy $ /usr/bin/iofog-agent
-bash: /usr/bin/iofog-agent: Permission denied

Failed to delete microservices running on an agent with coredns containers running

I have a ioFog agent running with coredns/coredns:latest(Image id: ead0a4a53df8) containers on it. I deployed a microservice on that agent and then I wanted to delete the microservice and failed. The iofog-agent log gave

[08/12/2023 09:52:04.997] [SEVERE] [PMCM] [Process Manager] : com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.dockerjava.api.model.Capability` from String "all": value not one of declared Enum instance names: [ALL, SYS_BOOT, DAC_OVERRIDE, NET_RAW, BLOCK_SUSPEND, FOWNER, IPC_LOCK, IPC_OWNER, SYS_PACCT, NET_BIND_SERVICE, WAKE_ALARM, FSETID, DAC_READ_SEARCH, SYS_CHROOT, SYS_RAWIO, SYS_ADMIN, KILL, MAC_ADMIN, SYS_RESOURCE, CHOWN, SETPCAP, SYS_PTRACE, NET_ADMIN, SETFCAP, SYS_NICE, LINUX_IMMUTABLE, AUDIT_CONTROL, LEASE, AUDIT_WRITE, SYS_MODULE, MKNOD, SYSLOG, MAC_OVERRIDE, SYS_TIME, SETGID, SETUID, SYS_TTY_CONFIG, NET_BROADCAST]
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 2054] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["HostConfig"]->com.github.dockerjava.api.model.HostConfig["CapDrop"]->java.lang.Object[][0]) - Exception: Error monitoring container - Stack trace: org.eclipse.iofog.exception.AgentSystemException: Error monitoring container
        at org.eclipse.iofog.process_manager.ProcessManager.lambda$new$1(ProcessManager.java:117)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.ws.rs.client.ResponseProcessingException: com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `com.github.dockerjava.api.model.Capability` from String "all": value not one of declared Enum instance names: [ALL, SYS_BOOT, DAC_OVERRIDE, NET_RAW, BLOCK_SUSPEND, FOWNER, IPC_LOCK, IPC_OWNER, SYS_PACCT, NET_BIND_SERVICE, WAKE_ALARM, FSETID, DAC_READ_SEARCH, SYS_CHROOT, SYS_RAWIO, SYS_ADMIN, KILL, MAC_ADMIN, SYS_RESOURCE, CHOWN, SETPCAP, SYS_PTRACE, NET_ADMIN, SETFCAP, SYS_NICE, LINUX_IMMUTABLE, AUDIT_CONTROL, LEASE, AUDIT_WRITE, SYS_MODULE, MKNOD, SYSLOG, MAC_OVERRIDE, SYS_TIME, SETGID, SETUID, SYS_TTY_CONFIG, NET_BROADCAST]
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 2054] (through reference chain: com.github.dockerjava.api.command.InspectContainerResponse["HostConfig"]->com.github.dockerjava.api.model.HostConfig["CapDrop"]->java.lang.Object[][0])
        at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:873)
        at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:767)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:229)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
        at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:428)
        at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:324)
        at com.github.dockerjava.jaxrs.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:30)
        at com.github.dockerjava.jaxrs.InspectContainerCmdExec.execute(InspectContainerCmdExec.java:13)
        at com.github.dockerjava.jaxrs.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:23)
        at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
        at com.github.dockerjava.core.command.InspectContainerCmdImpl.exec(InspectContainerCmdImpl.java:52)
        at org.eclipse.iofog.process_manager.DockerUtil.lambda$getRunningContainers$2(DockerUtil.java:363)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.eclipse.iofog.process_manager.DockerUtil.getRunningContainers(DockerUtil.java:367)
        at org.eclipse.iofog.process_manager.ProcessManager.deleteRemainingMicroservices(ProcessManager.java:199)
        at org.eclipse.iofog.process_manager.ProcessManager.lambda$new$1(ProcessManager.java:114)
        ... 1 more

docker inspect with the coredns container gave this

    "HostConfig": {
        "CapAdd": [
            "NET_BIND_SERVICE"
        ],
        "CapDrop": [
            "all"
        ],

and my docker version on that agent is

Client:
 Version:           20.10.25
 API version:       1.40
 Go version:        go1.18.1
 Git commit:        20.10.25-0ubuntu1~20.04.1
 Built:             Fri Jul 14 22:00:45 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.25
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.1
  Git commit:       20.10.25-0ubuntu1~20.04.1
  Built:            Thu Jun 29 21:55:06 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~20.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

I do not know how to fix it. My work depends on ioFog deploying and deleting microservices properly and hope it can be fixed.

Container with command is not launched with the arguments in the specified order (alphabettically sorted)

The Application template contains:

          commands:
            - '/usr/bin/contextBroker'
            - '-fg'
            - '-multiservice'
            - '-ngsiv1Autocast'
            - '-dbhost'
            - 'localhost:2017'
            - '-corsOrigin'
            - '__ALL'

After instanciation and deployement

 iofogctl describe microservice

shows:

    commands:
    - /usr/bin/contextBroker
    - -fg
    - -multiservice
    - -ngsiv1Autocast
    - -dbhost
    - localhost:2017
    - -corsOrigin
    - __ALL

But running container get:

/usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -corsOrigin -dbhost -fg -mult
iservice -ngsiv1Autocast /usr/bin/contextBroker __ALL localhost:2017

Provide systemd service

I when installing the agent I noticed that is still uses an init.d script. Event on a Raspberry Pi, that seems rather outdated.

Declare dependency on Java

The agent requires Java to run. Installing the deb package does not install a JRE. So that running the Agent will fail:

pi@raspberrypi:~/deploy $ sudo /etc/init.d/iofog-agent start
Starting iofog-agent service...
Using Java version  found at ''
/etc/init.d/iofog-agent: 30: /etc/init.d/iofog-agent: java: not found

"controller certificate verification failed" occurs with ioFog agent

I attempted to connect ioFog agent and ioFog controller.
However, the ioFog controller does not recognize the ioFog agent.

I checked the log of ioFog agent(/var/log/iofog-agent/iofog-agent.0.log).
In the log, "controller certificate verification failed" was described as follows.

[12/04/2018 11:16:19.022] [WARN] [Field Agent] : not provisioned
[12/04/2018 11:16:19.022] [WARN] [Field Agent] : not provisioned
[12/04/2018 11:16:19.024] [INFO] [Message Bus] : STARTING MESSAGE BUS SERVER
[12/04/2018 11:16:19.024] [INFO] [Message Bus Server] : starting...
[12/04/2018 11:16:19.553] [INFO] [Message Bus] : MESSAGE BUS SERVER STARTED
[12/04/2018 11:16:19.586] [INFO] [LOCAL API ] : Local Api Instantiated
[12/04/2018 11:16:19.592] [INFO] [Local API] : Container configuration retrieved
[12/04/2018 11:16:19.601] [INFO] [Supervisor] : started
[12/04/2018 11:16:19.950] [INFO] [Local API] : Local api server started at port: 54321

[12/04/2018 11:16:23.097] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:24.587] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:29.014] [INFO] [Process Manager] : monitoring containers
[12/04/2018 11:16:29.101] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:29.591] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:29.951] [INFO] [Local API] : Initiating control signals for unacknowledged signals
[12/04/2018 11:16:29.953] [INFO] [Local API] : Initiating message sending for the unacknowledged messages
[12/04/2018 11:16:34.592] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:35.104] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:39.128] [INFO] [Process Manager] : monitoring containers
[12/04/2018 11:16:39.593] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:39.951] [INFO] [Local API] : Initiating control signals for unacknowledged signals
[12/04/2018 11:16:39.953] [INFO] [Local API] : Initiating message sending for the unacknowledged messages
[12/04/2018 11:16:41.107] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:44.594] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:47.110] [INFO] [Resource Consumption Manager] : get usage data
[12/04/2018 11:16:49.139] [INFO] [Process Manager] : monitoring containers
[12/04/2018 11:16:49.595] [INFO] [Message Bus] : check message bus server status
[12/04/2018 11:16:49.952] [INFO] [Local API] : Initiating control signals for unacknowledged signals
[12/04/2018 11:16:49.954] [INFO] [Local API] : Initiating message sending for the unacknowledged messages
[12/04/2018 11:16:52.139] [INFO] [Field Agent] : post IOFog status
[12/04/2018 11:16:52.140] [WARN] [Field Agent] : not provisioned
[12/04/2018 11:16:52.141] [WARN] [Field Agent] : controller certificate verification failed
[12/04/2018 11:16:52.141] [WARN] [Field Agent] : not provisioned
[12/04/2018 11:16:52.141] [INFO] [Field Agent] : start posting IOFog status

My ioFog agent setting is below.

$ sudo iofog-agent info
[sudo] password for test:
Iofog UUID : not provisioned
IP Address : 192.168.11.19
Network Interface : ens33
Developer's Mode : on
ioFog Controller : http://192.168.11.10:51121/api/v3/
ioFog Certificate : /etc/iofog-agent/cert.crt
Docker URL : unix:///var/run/docker.sock
Disk Usage Limit : 50.00 GiB
Message Storage Directory : /var/lib/iofog-agent/
Memory RAM Limit : 4096.00 MiB
CPU Usage Limit : 80.00%
Log Disk Limit : 10.00 GiB
Log File Directory : /var/log/iofog-agent/
Log Rolling File Count : 10
Status Update Frequency : 30
Get Changes Frequency : 60
Scan Devices Frequency : 60
Post Diagnostics Frequency : 10
Isolated Docker Containers Mode : off
GPS mode : auto
GPS coordinates(lat,lon) : 35.6906,139.77
Fog type : intel_amd

My ioFog controller setting is below.

$iofog-controller config list
Port: 51121
Home url: https://google.com
Email activation: false
Log files directory: /var/log/iofog-controller
Log files size: 1048576
Dev mode: true

※I am deploying iofog agent and iofog controller in separate virtual machines.

Why is ioFog controller unable to recognize ioFog agent?

Agent fails to parse CPU usage on systems using European locale

The awk command used for determining the agent's CPU utilization assumes the output to use a . as the decimal limiter. However, on a system using European (here: German) locale, the value output uses a , as decimal limiter which leads to a failure to parse the value using Java's Float.parseFloat() method.
I can see two options to fix this:

1. Prepend the awk command with a LC_NUMERIC=en_US.UTF-8 && ` in order to use a decimal point instead of a comma when formatting floats, or
2. Prevent the problem by not outputting a float in the first place. I doubt the value of knowing that the CPU utilization is 56.345 instead of 56 percent. Thus, the awk command could be changed to format the output like this:

   grep 'cpu' /proc/stat | awk '{usage=($2+$3+$4)*100/($2+$3+$4+$5+$6+$7+$8+$9)} END {printf (%d, usage)}'

   And then parse the value using Integer.parseInt() ...

Deploying a local agent fails to make it reboot-safe

Once you managed to fire up a local agent, and reboot the system. The container is gone and not restarted automatically.

Potential Null Dereference during the execution of command

Potential Null Dereference Path

In file: CommandShellExecutor.java there is the following code segment

public static <V, E> CommandShellResultSet<V, E> executeCommand(String command, Function<CommandShellResultSet<List<String>, List<String>>, CommandShellResultSet<V, E>> mapper) {
        return executeCommand(command).map(mapper);
}

which invokes another method executeCommand

public static CommandShellResultSet<List<String>, List<String>> executeCommand(String command) {
        String[] fullCommand = computeCommand(command);
        return execute(fullCommand);
}

which invokes execute

private static CommandShellResultSet<List<String>, List<String>> execute(String[] fullCommand) {
        CommandShellResultSet<List<String>, List<String>> resultSet = null;
        try {
            Process process = Runtime.getRuntime().exec(fullCommand);
            List<String> value = readOutput(process, Process::getInputStream);
            List<String> errors = readOutput(process, Process::getErrorStream);
            resultSet = new CommandShellResultSet<>(value, errors);
        } catch (IOException e) {
            LoggingService.logError(MODULE_NAME, e.getMessage(), e);
        }
        return resultSet;
}

this method initialize the resultSet variable with null value and only assign any value to it inside ta try-catch block. But before assigning any value to resultSet if IOException occurs due to readOutput or Runtime.getRuntime().exec() then resultSet will be returned as null which can later lead to NullPointerException in the very first method due to null.map(mapper)

Possible Workaround

One possible workaround is always check the returned value of execute() like

public static <V, E> CommandShellResultSet<V, E> executeCommand(String command, Function<CommandShellResultSet<List<String>, List<String>>, CommandShellResultSet<V, E>> mapper) {
        CommandShellResultSet<List<String>, List<String>> resultSet = executeCommand(command);
        if(resultSet != null){
            return resultSet.map(mapper);
        }else{
            // return something appropriate
        }
}

another workaround can be use of try-catch which is generally not recommended as a best practice

public static <V, E> CommandShellResultSet<V, E> executeCommand(String command, Function<CommandShellResultSet<List<String>, List<String>>, CommandShellResultSet<V, E>> mapper) {
        try{
            return executeCommand(command).map(mapper);
        }catch(NullPointerException e){
            // do something appropriate
        }finally{
            // if needed
        }
}

as the executeCommand method was used in the several places in the codebase so, if any of the places this scenario arises which can lead to an unexpected behavior or crash the program.

Fix from triage team!!!

We have mentioned to return something appropriate because of lack of knowledge of the codebase, it is not possible to know which value should be the appropriate value to return in case of any null vlaue or which fix type is more appropriate for this codebase.

Sponsorship and Support:

This work is done by the security researchers from OpenRefactory and is supported by the Open Source Security Foundation (OpenSSF): Project Alpha-Omega. Alpha-Omega is a project partnering with open source software project maintainers to systematically find new, as-yet-undiscovered vulnerabilities in open source code - and get them fixed - to improve global software supply chain security.

The bug is found by running the iCR tool by OpenRefactory, Inc. and then manually triaging the results.