aws-greengrass / aws-greengrass-nucleus Goto Github PK

The Greengrass nucleus component provides functionality for device side orchestration of deployments and lifecycle management for execution of Greengrass components and applications. This includes features such as starting, stopping, and monitoring execution of components and apps, interprocess communication server for communication between components, component installation and configuration management.

License: Apache License 2.0

Java 99.79% Shell 0.08% Batchfile 0.11% Gherkin 0.02%

aws java greengrass iot edge-computing

aws-greengrass-nucleus's Introduction

Greengrass Nucleus

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

SPDX-License-Identifier: Apache-2.0

Configuration
Dependencies on other services
A set of lifecycle phases in the form of a finite state machine.

A service may have processes, threads, code, network connections, ... But not necessarily. Some have all of these, some have only one.

You can think of the nucleus as a mash-up of make, a super-lightweight publish/subscribe system, and a small hierarchic key-value data store. The various services have continuously varying states that the nucleus monitors and manages. A dependent service is not started until its dependencies are started, and if they become unstable, the dependent service is notified. The internal interconnections are handled via dependency injection. Restarts are managed automatically.

When configuration changes, all users of them are notified. Everything adapts continuously.

A quick tour through com.aws.greengrass

config Manages the system configuration (model). It's fundamentally a hierarchic key-value store with timestamps. It can be serialized to/from yaml, json, or a transaction log. The transaction log can be replayed to reconstruct the config, or streamed live to another process to maintain a mirror. The terminology is borrowed from the world of publish/subscribe systems. Config values can have validators and watcher.
dependency The dependency injection framework. The meat is in context.java which contains a Map of known objects, and the ability to get (and magically create) objects from the Context. When an object is created by the framework, it does dependency injection. If the created object participates in the Lifecycle framework, its lifecycle is initiated. This feeds the Lifecycle dependency graph.
lifecyclemanager Ties the model to Lifecycle objects in the dependency graph. The primary class is GreengrassService, which contains most of the state transition logic. GenericExternalService is a subclass that implements a service whose behavior is defined by commands and scripts. Either of these classes may be subclassed to provide services whose behavior is defined by code running within Greengrass.
util A grab-bag of useful utilities.

Learn more

aws-greengrass-nucleus's People

Contributors

Stargazers

Watchers

aws-greengrass-nucleus's Issues

addUserToGroup does not support Busybox

The function addUserToGroup uses usermod under the hood to add a user to a group

runCmd("usermod -a -G " + group + " " + user, o -> {

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/util/platforms/unix/UnixPlatform.java

Line 367 in 87d89a6

runCmd("usermod -a -G " + group + " " + user, o -> {

Busybox based Linux use the command equivalent

"addgroup " + user + " " + group

https://busybox.net/downloads/BusyBox.html

This option also works on Debian based systems if both the user and the group are pre-existing cf. man 8 adduser

(DockerApplicationManager): The input device is not a TTY

Describe the bug
I try to deploy a docker image component with the run script to be docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld:latest.
However, the deployment always fail and in the log file /greengrass/v2/logs/hello_world.log it shows that stderr. the input device is not a TTY.

To Reproduce

Create a hello_world-1.0.0.json file in the folder of ./greengrassv2/recipes.
Paste the recipe info into the hello_world-1.0.0.json provided below.

{
  "RecipeFormatVersion": "2020-01-25",
  "ComponentName": "hello_world",
  "ComponentVersion": "1.0.0",
  "ComponentDescription": "A component that runs a public Docker container from Amazon ECR.",
  "ComponentPublisher": "Greengrass Tester. Inc",
  "ComponentDependencies": {
    "aws.greengrass.DockerApplicationManager": {
      "VersionRequirement": "~2.0.0"
    },
    "aws.greengrass.TokenExchangeService": {
      "VersionRequirement": "~2.0.0"
    }
  },
  "Manifests": [
    {
      "Platform": {
        "os": "linux"
      },
      "Lifecycle": {
        "Run": {
          "Script": "docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld:latest"
        }
      },
      "Artifacts": [
        {
          "URI": "docker:public.ecr.aws/d7u3w7w4/helloworld:latest"
        }
      ]
    }
  ]
}

Run sudo /greengrass/v2/bin/greengrass-cli deployment create --recipeDir ./greengrassv2/recipes --merge hello_world=1.0.0

Expected behavior
I expect that after the deployment, there will be a terminal shows the same stdout from the command docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld as shown below.

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Actual behavior
The deployment will fail. And the log file of /greengrass/v2/logs/hello_world.log will show that the input device is not a TTY and Run script exited as shown below.

test@test:~/Desktop/greengrass$ sudo cat /greengrass/v2/logs/hello_world.log
2021-05-17T02:40:12.836Z [INFO] (pool-2-thread-117) hello_world: shell-runner-start. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=STARTING, command=["docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld:latest"]}
2021-05-17T02:40:12.864Z [WARN] (Copier) hello_world: stderr. the input device is not a TTY. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:40:12.867Z [INFO] (Copier) hello_world: Run script exited. {exitCode=1, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:40:12.868Z [INFO] (pool-2-thread-117) hello_world: shell-runner-start. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=STARTING, command=["docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld:latest"]}
2021-05-17T02:40:12.892Z [WARN] (Copier) hello_world: stderr. the input device is not a TTY. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:40:12.895Z [INFO] (Copier) hello_world: Run script exited. {exitCode=1, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:40:12.896Z [INFO] (pool-2-thread-114) hello_world: shell-runner-start. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=STARTING, command=["docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld:latest"]}
2021-05-17T02:40:12.920Z [WARN] (Copier) hello_world: stderr. the input device is not a TTY. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:40:12.923Z [INFO] (Copier) hello_world: Run script exited. {exitCode=1, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:43:12.993Z [INFO] (pool-2-thread-120) hello_world: shell-runner-start. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=STARTING, command=["docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld:latest"]}
2021-05-17T02:43:13.017Z [WARN] (Copier) hello_world: stderr. the input device is not a TTY. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:43:13.020Z [INFO] (Copier) hello_world: Run script exited. {exitCode=1, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:43:13.021Z [INFO] (pool-2-thread-121) hello_world: shell-runner-start. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=STARTING, command=["docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld:latest"]}
2021-05-17T02:43:13.047Z [WARN] (Copier) hello_world: stderr. the input device is not a TTY. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:43:13.050Z [INFO] (Copier) hello_world: Run script exited. {exitCode=1, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:43:13.052Z [INFO] (pool-2-thread-121) hello_world: shell-runner-start. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=STARTING, command=["docker run -it --rm public.ecr.aws/d7u3w7w4/helloworld:latest"]}
2021-05-17T02:43:13.076Z [WARN] (Copier) hello_world: stderr. the input device is not a TTY. {scriptName=services.hello_world.lifecycle.Run.Script, serviceName=hello_world, currentState=RUNNING}
2021-05-17T02:43:13.079Z [INFO] (Copier) hello_world: Run script exited. {exitCode=1, serviceName=hello_world, currentState=RUNNING}

Environment

OS: Ubuntu 18.04
JDK version: Java 8
Nucleus version: 2.1.0

Additional context
I was trying to deploy a Docker image from our private repository and hoping that after the deployment, it will automatically docker run our image and show the UI we designed for users to interact with. But I am not sure how to attach a pseudo terminal for it to show the stdout properly and even let a UI pops up. I then tested the hello_world docker image and try to see if I can attach a terminal for it after the deployment.

Greengrass is not shutting down components properly upon shutdown and restarts them on restart

After noticing that components were still running after GG failed to shut them down upon shutdown, they were reparented to init. Then when greengrass restarted, duplicate processes were started. After seeing this I created a few test programs and saw the same behaviors.

For example, I have two components d and g.

root@raspberrypi:/home/pi/forkbug# pstree -p
systemd(1)─┬─alsactl(339)
           ├─avahi-daemon(326)───avahi-daemon(340)
           ├─bluetoothd(522)
           ├─cron(328)
           ├─dbus-daemon(325)
           ├─dhcpcd(504)
           ├─hciattach(516)
           ├─login(524)───bash(635)
           ├─rngd(384)─┬─{rngd}(385)
           │           ├─{rngd}(386)
           │           └─{rngd}(387)
           ├─rsyslogd(329)─┬─{rsyslogd}(381)
           │               ├─{rsyslogd}(382)
           │               └─{rsyslogd}(383)
           ├─sshd(525)─┬─sshd(7691)───sshd(7697)───bash(7698)───sudo(9892)───bash(9893)───pstree(10625)
           │           └─sshd(9091)───sshd(9097)───bash(9098)───sudo(10367)───java(10368)─┬─sudo(10462)───sh(10465)───g(10466)─┬─g(10467)───g(10473)
           │                                                                              │                                    ├─g(10468)───g(10475)
           │                                                                              │                                    ├─g(10469)───g(10474)
           │                                                                              │                                    ├─g(10470)───g(10472)
           │                                                                              │                                    └─g(10471)───g(10476)
           │                                                                              ├─sudo(10521)───sh(10525)───d(10526)─┬─d(10527)───d(10532)
           │                                                                              │                                    ├─d(10528)───d(10535)
           │                                                                              │                                    ├─d(10529)───d(10533)
           │                                                                              │                                    ├─d(10530)───d(10534)
           │                                                                              │                                    └─d(10531)───d(10536)
           │                                                                              ├─{java}(10369)
           │                                                                              ├─{java}(10370)
           │                                                                              ├─{java}(10371)
           │                                                                              ├─{java}(10372)
           │                                                                              ├─{java}(10373)
           │                                                                              ├─{java}(10374)
           │                                                                              ├─{java}(10375)
           │                                                                              ├─{java}(10376)
           │                                                                              ├─{java}(10377)
           │                                                                              ├─{java}(10378)
           │                                                                              ├─{java}(10380)
           │                                                                              ├─{java}(10384)
           │                                                                              ├─{java}(10426)
           │                                                                              ├─{java}(10427)
           │                                                                              ├─{java}(10428)
           │                                                                              ├─{java}(10429)
           │                                                                              ├─{java}(10430)
           │                                                                              ├─{java}(10431)
           │                                                                              ├─{java}(10432)
           │                                                                              ├─{java}(10433)
           │                                                                              ├─{java}(10434)
           │                                                                              ├─{java}(10435)
           │                                                                              ├─{java}(10436)
           │                                                                              ├─{java}(10437)
           │                                                                              ├─{java}(10438)
           │                                                                              ├─{java}(10439)
           │                                                                              ├─{java}(10440)
           │                                                                              ├─{java}(10441)
           │                                                                              ├─{java}(10442)
           │                                                                              ├─{java}(10443)
           │                                                                              ├─{java}(10444)
           │                                                                              ├─{java}(10445)
           │                                                                              ├─{java}(10447)
           │                                                                              ├─{java}(10450)
           │                                                                              ├─{java}(10451)
           │                                                                              ├─{java}(10452)
           │                                                                              ├─{java}(10453)
           │                                                                              ├─{java}(10454)
           │                                                                              ├─{java}(10457)
           │                                                                              ├─{java}(10459)
           │                                                                              ├─{java}(10463)
           │                                                                              ├─{java}(10464)
           │                                                                              ├─{java}(10522)
           │                                                                              ├─{java}(10523)
           │                                                                              └─{java}(10524)
           ├─systemd(624)───(sd-pam)(625)
           ├─systemd-journal(123)
           ├─systemd-logind(335)
           ├─systemd-timesyn(286)───{systemd-timesyn}(323)
           ├─systemd-udevd(145)
           ├─thd(354)
           ├─wpa_supplicant(338)
           └─wpa_supplicant(431)

d has a top level supervisor and spawns 5 workers, restarting any workers that fail. In d each worker that manages a socket, also has a child process that it manages individual requests. The worker spawns a new child for each request. The g component does the same thing, but also creates separate process groups per worker, but has the top level supervisor act as child subreaper via prctl(PR_SET_CHILD_SUBREAPER). so that individual worker groups can be killed, but should a worker die it's children get reaped by the supervisor. In both cases the top level d and g components don't demonsize and send signals to their child processes to terminate.

When greengrass shuts down and attempts to shutdown the processes it enters into an errored state:

2021-04-08T12:22:59.471Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=d, currentState=STOPPING}
2021-04-08T12:22:59.473Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutting down process ["/greengrass/packages/artifacts/d/0.0.1/d >/tmp/d.log 2>&1\n"]. {serviceName=d, currentState=STOPPING}
2021-04-08T12:22:59.478Z [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-report-state. {serviceName=UpdateSystemPolicyService, currentState=STOPPING, newState=FINISHED}
2021-04-08T12:22:59.478Z [INFO] (pool-2-thread-23) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=g, currentState=STOPPING}
2021-04-08T12:22:59.479Z [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: Stopping backingTask start. {serviceName=UpdateSystemPolicyService, currentState=STOPPING}
2021-04-08T12:22:59.480Z [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=STOPPING, newState=FINISHED}
2021-04-08T12:22:59.481Z [INFO] (pool-2-thread-23) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutting down process ["/greengrass/packages/artifacts/g/0.0.1/g >/tmp/g.log 2>&1\n"]. {serviceName=g, currentState=STOPPING}
2021-04-08T12:22:59.482Z [WARN] (pool-2-thread-13) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-run-interrupted. Service interrupted while running startup. {serviceName=UpdateSystemPolicyService, currentState=FINISHED}
2021-04-08T12:22:59.526Z [INFO] (pool-2-thread-23) com.aws.greengrass.util.platforms.Platform: Killing child processes of pid 10632. {}
2021-04-08T12:22:59.527Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.util.platforms.Platform: Killing child processes of pid 10634. {}
2021-04-08T12:22:59.593Z [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-report-state. {serviceName=DeploymentService, currentState=STOPPING, newState=FINISHED}
2021-04-08T12:22:59.595Z [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: Stopping backingTask start. {serviceName=DeploymentService, currentState=STOPPING}
2021-04-08T12:22:59.596Z [WARN] (pool-2-thread-15) com.aws.greengrass.deployment.DeploymentService: service-run-interrupted. Service interrupted while running startup. {serviceName=DeploymentService, currentState=STOPPING}
2021-04-08T12:22:59.596Z [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=STOPPING, newState=FINISHED}
2021-04-08T12:23:14.471Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.util.platforms.Platform: Killing child processes of pid 10634. {}
2021-04-08T12:23:14.471Z [ERROR] (d-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-errored. {reason=Timeout in shutdown, serviceName=d, currentState=STOPPING}
2021-04-08T12:23:14.474Z [INFO] (d-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=d, currentState=STOPPING, newState=ERRORED}
2021-04-08T12:23:14.476Z [INFO] (d-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=d, currentState=STOPPING, newState=ERRORED}
2021-04-08T12:23:14.479Z [INFO] (pool-2-thread-23) com.aws.greengrass.util.platforms.Platform: Killing child processes of pid 10632. {}
2021-04-08T12:23:14.478Z [ERROR] (g-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-errored. {reason=Timeout in shutdown, serviceName=g, currentState=STOPPING}
2021-04-08T12:23:14.480Z [INFO] (g-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-report-state. {serviceName=g, currentState=STOPPING, newState=ERRORED}
2021-04-08T12:23:14.482Z [INFO] (g-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=g, currentState=STOPPING, newState=ERRORED}
2021-04-08T12:23:14.492Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.KernelLifecycle: executor-service-shutdown-initiated. {}
2021-04-08T12:23:14.492Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown completed for process ["/greengrass/packages/artifacts/d/0.0.1/d >/tmp/d.log 2>&1\n"]. {serviceName=d, currentState=ERRORED}
2021-04-08T12:23:14.493Z [INFO] (aws.greengrass.Cli-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=d, currentState=ERRORED}
2021-04-08T12:23:14.492Z [INFO] (pool-2-thread-23) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown completed for process ["/greengrass/packages/artifacts/g/0.0.1/g >/tmp/g.log 2>&1\n"]. {serviceName=g, currentState=ERRORED}
2021-04-08T12:23:14.497Z [INFO] (pool-2-thread-23) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=g, currentState=ERRORED}
2021-04-08T12:23:14.495Z [INFO] (Thread-1) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Waiting for executors to shutdown. {}
2021-04-08T12:23:14.495Z [WARN] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-state-transition-interrupted. Service lifecycle thread interrupted. Thread will exit now. {serviceName=FleetStatusService, currentState=FINISHED}
2021-04-08T12:23:14.497Z [INFO] (Copier) com.aws.greengrass.lifecyclemanager.GenericExternalService: Run script exited. {exitCode=143, serviceName=d, currentState=ERRORED}
2021-04-08T12:23:14.501Z [INFO] (Copier) d: Run script exited. {exitCode=143, serviceName=d, currentState=ERRORED}
2021-04-08T12:23:14.504Z [INFO] (Thread-1) com.aws.greengrass.lifecyclemanager.KernelLifecycle: executor-service-shutdown-complete. {}
2021-04-08T12:23:14.499Z [INFO] (Copier) com.aws.greengrass.lifecyclemanager.GenericExternalService: Run script exited. {exitCode=143, serviceName=g, currentState=ERRORED}
2021-04-08T12:23:14.510Z [INFO] (Copier) g: Run script exited. {exitCode=143, serviceName=g, currentState=ERRORED}
2021-04-08T12:23:14.529Z [INFO] (Thread-1) com.aws.greengrass.lifecyclemanager.KernelLifecycle: context-shutdown-initiated. {}
2021-04-08T12:23:14.572Z [INFO] (Serialized listener processor) com.aws.greengrass.dependency.Context: Interrupted while running tasks. Publish thread will exit now.. {}
2021-04-08T12:23:14.572Z [INFO] (Thread-1) com.aws.greengrass.lifecyclemanager.KernelLifecycle: context-shutdown-complete. {}

The killProcessAndChildren call successfully terminates the sh wrapper, but because the other worker respawn their children the methodology for reading the pids does not work. And since it tries to terminate each process listed (from a static list) rather than sending a signal to the entire process group, it fails to terminate the components.

systemd(1)─┬─alsactl(339)
           ├─avahi-daemon(326)───avahi-daemon(340)
           ├─bluetoothd(522)
           ├─cron(328)
           ├─d(10647)─┬─d(10652)───d(10657)
           │          ├─d(10653)───d(10658)
           │          ├─d(10654)───d(10659)
           │          ├─d(10655)───d(10660)
           │          └─d(10656)───d(10661)
           ├─dbus-daemon(325)
           ├─dhcpcd(504)
           ├─g(10467)───g(10473)
           ├─g(10468)───g(10475)
           ├─g(10469)───g(10474)
           ├─g(10470)───g(10472)
           ├─g(10471)───g(10476)
           ├─g(10638)─┬─g(10641)───g(10646)
           │          ├─g(10642)───g(10648)
           │          ├─g(10643)───g(10650)
           │          ├─g(10644)───g(10651)
           │          └─g(10645)───g(10649)
           ├─hciattach(516)
           ├─login(524)───bash(635)
           ├─rngd(384)─┬─{rngd}(385)
           │           ├─{rngd}(386)
           │           └─{rngd}(387)
           ├─rsyslogd(329)─┬─{rsyslogd}(381)
           │               ├─{rsyslogd}(382)
           │               └─{rsyslogd}(383)
           ├─sshd(525)─┬─sshd(7691)───sshd(7697)───bash(7698)───sudo(9892)───bash(9893)───pstree(10733)
           │           └─sshd(9091)───sshd(9097)───bash(9098)
           ├─systemd(624)───(sd-pam)(625)
           ├─systemd-journal(123)
           ├─systemd-logind(335)
           ├─systemd-timesyn(286)───{systemd-timesyn}(323)
           ├─systemd-udevd(145)
           ├─thd(354)
           ├─wpa_supplicant(338)
           └─wpa_supplicant(431)

As you can see d and g get reparented to init, and some of the prior processes were killed and new ones were spawned in their place. The net result is they are still mostly running. Then if you startup greengrass again:

     ├─g(10638)─┬─g(10641)───g(10646)
       │          ├─g(10642)───g(10648)
       │          ├─g(10643)───g(10650)
       │          ├─g(10644)───g(10651)
       │          └─g(10645)───g(10649)
       ├─hciattach(516)
       ├─login(524)───bash(635)
       ├─rngd(384)─┬─{rngd}(385)
       │           ├─{rngd}(386)
       │           └─{rngd}(387)
       ├─rsyslogd(329)─┬─{rsyslogd}(381)
       │               ├─{rsyslogd}(382)
       │               └─{rsyslogd}(383)
       ├─sshd(525)─┬─sshd(7691)───sshd(7697)───bash(7698)───sudo(9892)───bash(9893)───pstree(10860)
       │           └─sshd(9091)───sshd(9097)───bash(9098)───sudo(10734)───java(10735)─┬─sudo(10828)───sh(10835)───g(10837)─┬─g(10839)───g(10848)
       │                                                                              │                                    ├─g(10840)───g(10847)
       │                                                                              │                                    ├─g(10841)───g(10845)
       │                                                                              │                                    ├─g(10842)───g(10844)
       │                                                                              │                                    └─g(10843)───g(10846)
       │                                                                              ├─sudo(10829)───sh(10836)───d(10838)─┬─d(10849)───d(10854)
       │                                                                              │                                    ├─d(10850)───d(10855)
       │                                                                              │                                    ├─d(10851)───d(10856)
       │                                                                              │                                    ├─d(10852)───d(10858)
       │                                                                              │                                    └─d(10853)───d(10857)
       │                                                                              ├─{java}(10736)
       │                                                                              ├─{java}(10737)
       │                                                                              ├─{java}(10738)
       │                                                                              ├─{java}(10739)
       │                                                                              ├─{java}(10740)
       │                                                                              ├─{java}(10741)
       │                                                                              ├─{java}(10742)
       │                                                                              ├─{java}(10743)
       │                                                                              ├─{java}(10744)
       │                                                                              ├─{java}(10745)
       │                                                                              ├─{java}(10746)
       │                                                                              ├─{java}(10750)
       │                                                                              ├─{java}(10790)
       │                                                                              ├─{java}(10791)
       │                                                                              ├─{java}(10792)
       │                                                                              ├─{java}(10793)
       │                                                                              ├─{java}(10794)
       │                                                                              ├─{java}(10796)
       │                                                                              ├─{java}(10797)
       │                                                                              ├─{java}(10798)
       │                                                                              ├─{java}(10799)
       │                                                                              ├─{java}(10800)
       │                                                                              ├─{java}(10801)
       │                                                                              ├─{java}(10802)
       │                                                                              ├─{java}(10803)
       │                                                                              ├─{java}(10804)
       │                                                                              ├─{java}(10805)
       │                                                                              ├─{java}(10806)
       │                                                                              ├─{java}(10807)
       │                                                                              ├─{java}(10808)
       │                                                                              ├─{java}(10809)
       │                                                                              ├─{java}(10810)
       │                                                                              ├─{java}(10811)
       │                                                                              ├─{java}(10812)
       │                                                                              ├─{java}(10813)
       │                                                                              ├─{java}(10814)
       │                                                                              ├─{java}(10816)
       │                                                                              ├─{java}(10817)
       │                                                                              ├─{java}(10818)
       │                                                                              ├─{java}(10820)
       │                                                                              ├─{java}(10822)
       │                                                                              ├─{java}(10823)
       │                                                                              ├─{java}(10824)
       │                                                                              ├─{java}(10825)
       │                                                                              ├─{java}(10826)
       │                                                                              ├─{java}(10827)
       │                                                                              ├─{java}(10830)
       │                                                                              ├─{java}(10831)
       │                                                                              ├─{java}(10832)
       │                                                                              ├─{java}(10833)
       │                                                                              └─{java}(10834)
       ├─systemd(624)───(sd-pam)(625)
       ├─systemd-journal(123)
       ├─systemd-logind(335)
       ├─systemd-timesyn(286)───{systemd-timesyn}(323)
       ├─systemd-udevd(145)
       ├─thd(354)
       ├─wpa_supplicant(338)
       └─wpa_supplicant(431)

Now there are multiple new ones running in addition to the originals. This can have very unexpected behaviors, especially when deployments restart greengrass.

To fix this issue, the killProcess code should be rewritten to use the child subreaper functionality of Linux 3.4, and the per-process kill command should be sent to the entire process group for the component (either using killpg, or kill with the negative of the component's lead pid. (cf man 2 kill)

(deployment): Deployment cancellation does not go through if deployment action has not yet been registered in UpdateSystemPolicyService pendingActions queue

Describe the bug
When a deployment gets cancelled, it can be in progress and in any of the possible stages, i.e
Stage 1. Preparation - dependency resolution/artifact download/config generation
Stage 2. Done with prep but waiting until components stop deferring deployment via IPC
Stage 3. Executing the change i.e. merging config, lifecycle changes, monitoring, (rollback if something fails) etc

In Stage 3, we don't allow cancellation because it can leave the system broken/ nondeterministic.
In Stage 2 and 1, cancellation should go through because it is the only way to get out of situations like long retry loop/components forever deferring deployments via IPC.

Right now, deployment cancellation in Stage 1 does not go through because the check to determine if cancellation is allowed based on the stage is incorrect

To Reproduce
We discovered this accidentally, when there was some misconfiguration in region and endpoint configs, the deployment is not able to reach cloud component service during dependency resolution, but the MQTT connection works fine so it can poll deployments and cancellation signals. If you try the same thing and try to cancel the deployment, this issue can be reproduced.

Expected behavior
When deployment is in Stage 1 and it gets cancelled, the cancellation and further deployments should go through

Actual behavior
When deployment was stuck in Stage 1 and it got cancelled, the cancellation or any further deployments did not go through and the original deployment stayed stuck in Stage 1

Environment

OS: [e.g. Ubuntu 20.04]
JDK version:
Nucleus version: 2.0.3

Additional context
Add any other context about the problem here.

E.g. what is the impact of the bug? - Deployment cancellation does not take effect when deployment is still in preparation mode(download recipes/resolve components/download artifacts etc)

(greengrasscoreipc): iot_core_publish memory leak?

Describe the bug
I'm sending thousands of messages per minute into IoT Core using IPC.new_publish_to_iot_core(), and there seems to be a tiny memory leak in the IPC client, which stacks up over time. Over the course of a few days, my script consumes hundreds of megabytes.

To Reproduce

import os
from awscrt.io import (
    ClientBootstrap,
    DefaultHostResolver,
    EventLoopGroup,
    SocketDomain,
    SocketOptions,
)
from awsiot.eventstreamrpc import Connection, LifecycleHandler, MessageAmendment
import awsiot.greengrasscoreipc.client as client
from awsiot.greengrasscoreipc.model import (
    QOS,
    PublishToIoTCoreRequest
)

class IPCUtils:
    def connect(self):
        elg = EventLoopGroup()
        resolver = DefaultHostResolver(elg)
        bootstrap = ClientBootstrap(elg, resolver)
        socket_options = SocketOptions()
        socket_options.domain = SocketDomain.Local
        amender = MessageAmendment.create_static_authtoken_amender(os.getenv("SVCUID"))
        hostname = os.getenv("AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT")
        connection = Connection(
            host_name=hostname,
            port=8033,
            bootstrap=bootstrap,
            socket_options=socket_options,
            connect_message_amender=amender,
        )
        self.lifecycle_handler = LifecycleHandler()
        connect_future = connection.connect(self.lifecycle_handler)
        TIMEOUT = 10
        connect_future.result(timeout=TIMEOUT)
        return connection

class IPCService():
    def __init__(self, log):
        self.log = log
        ipc_utils = IPCUtils()
        connection = ipc_utils.connect()
        self.ipc_client = client.GreengrassCoreIPCClient(connection)

    def sendMessage(self, topic, message):
        request = PublishToIoTCoreRequest()
        request.topic_name = topic
        request.payload = message
        request.qos = QOS.AT_LEAST_ONCE
        operation = self.ipc_client.new_publish_to_iot_core()
        try:
            operation.activate(request)
        except Exception as err:
            self.log.error(f'IPC error {err}')

if __name__ == '__main__':
    import time

    serv = IPCService({})
    while True: 
        for x in range (0, 20): 
            serv.sendMessage("TestThing/test/topic/{}".format(x), b'{"somejson":"value", "largeval": "0e 85 f0 a0 f3 55 39 a9 44 df fa 30 57 3b 14 8d 88 23 a5 04 80 de 18 b2 c9 cc 59 65 45 f6 89 bb 92 8f 12 b0 85 a8 1a 31 a2 38 9e 1d a1 d2 f8 16 8b 4f 31 8e b9 3b 7c 5d 93 71 e9 5b 0a c8 80 57 1e 1d f9 3b 38 07 47 15 f7 c7 15 63 66 27 ba e3 62 a5 60 dd e4 16 b4 c7 0c 69 ff d1 e9 79 b3 59 42 6d c9 0e 44 a5 9d 7d 0e bb ef e6 fc 55 ee 40 d0 ec 04 26 17 09 66 63 28 aa 57 0c f4 34 82 37 9c 94 80 62 64 1c e3 cc 78 04 11 7d 0b 40 ea b9 e6 92 b8 bd e2 56 7d 92 21 71 12 e1 01 f4 a7 e5 64 a5 e7 98 31 d1 f8 31 c3 5b 53 9f b5 41 2a e5 51 09 a9 4a 71 dd d7 46 d1 f1 55 4e 3e 8d 63 17 7e f5 4e 32 94 f8 a3 02 38 e7 82 30 a2 ae ea d5 09 12 12 d7 68 7c 5d 43 7b f5 e2 a3 42 77 90 13 5c e6 b8 92 b0 df 07 93 e1 27 42 80 75 06 bc 22 8d 01 02 ff 31 e2 66 fe 3c 42 d6 61 f4 ab 1e b9 73 8d db 3d 46 19 64 3a c8 71 d2 dc 56 9e d4 c8 8a 6a 6a d8 c0 1c fa 00 5e f2 68 3e 2f 17 8e 44 6d f7 55 19 ab 8f 43 35 e5 8c 1e 3d 7a 5e 1b 63 0d 83 66 3b 40 c2 a1 06 0c 1b f3 bc fc 49 1c 02 4a 6b 27 bb 24 11 04 9f 30 18 4c d5 4f f1 e3 12 df c9 6c 16 d2 35 ff 5a 60 dc d4 2a 3d fa 8b 70 51 b6 ca 4f 89 50 b9 e2 c0 c3 40 94 46 eb 59 43 31 38 6b ae 47 1a ee 56 f6 36 e9 1d f3 03 49 ec 9a 16 2e 7f 63 45 00 03 67 d1 2d bc 9f 69 21 7d 8f ac cb 6f b3 89 d1 3f ce 89 f7 17 73 d3 af 45 b7 3d 40 f8 be c8 b5 47 60 01 6e 62 5a f1 87 e5 f4 53 52 86 66 08 b6 80 ec 5a 88 cc 13 8c 45 7f e3 d4 8b 97 03 7b 03 ec f5 5d 95 30 4b d6 86 64 08 a8 ef 3c d6 d8 fb 58 61 b2 78 86 f3 a3 22 70 63 a2 ea 89 52 f3 4a 6d 7a 85 7e b6 02 3c e6 9e 6b 31 4a b9 9f 2f a8 ef 9e 48 03 da 21 a4 a8 e6 ea d7 cb ca 3b c9 03 e2 71 ac f3 9f 0a 22 e2 0e 6c 5f 47 6f ad b2 a8 a8 7c 58 05 97 22 35 e3 5d 32 a8 0c 26 98 63 40 03 90 65 6e db 9c b8 3b 56 2f a6 1b b2 9d f0 25 59 93 a7 ac 3c f6 f3 54 b1 b3 52 dd cd 30 8a ab e2 b8 fe 6f ec 14 9f 28 be 96 dc 75 90 ef 19 22 7e b1 fc 7f 0f a0 fe 61 aa 6e de 57 c1 24 74 22 ed e4 96 74 64 b6 5d 4c 98 3f 9c 6a e7 59 ec f8 5e 83 62 9c f0 fc 3e 3f d1 e7 28 52 92 fd ae 9d 4c d4 6c f0 97 04 fb 33 9a 69 68 64 94 3f 5a 7a 86 20 09 d8 5a dd 10 e2 d4 f5 59 48 52 45 8c 3c fc 32 0f bc 09 98 e8 2c a2 40 10 09 a9 8b b6 e1 58 e6 f0 cf f0 1a c5 d1 1b be f1 fc c5 30 a6 cf 41 56 51 ab 2e 32 3f 79 68 8a 53 a2 fc a3 69 79 1c cd 34 4b 55 46 e6 cb cb dd fd 59 e6 b5 7c 55 44 98 6b f9 21 89 cb b8 8a "}')
        time.sleep(1)

Expected behavior
Python script should run without growing memory resource usage.

Actual behavior
Memory usage increases as quantity of messages increases.

Environment

OS: Raspbian GNU/Linux 10 (buster)
JDK version: openjdk version "1.8.0_212"
Nucleus version: 2.0.4

Additional context
The amount of memory leaked does not appear to be proportional to the request payload size.
My attempts to use tracemalloc to further diagnose this leak haven't turned up anything I can immediately see wrong (but i'm not an expert in python)

(nucleus): issues with init-config flag when installing greengrass

Describe the bug
I am using terraform to create the AWS resources required by greengrass beforehand (The ones outlined in the manual installation page: https://docs.aws.amazon.com/greengrass/v2/developerguide/manual-installation.html).
After that I use ansible to

copy over the cert files to the root directory (/greengrass/v2),
copy over the installer zip to the GreengrassCore directory and create a config file (~/GreengrassCore/config.yml)
and run the command:

sudo -E java -Droot="{{greengrass_root_dir}}" -Dlog.store=FILE \
      -jar "{{greengrass_installer_dir}}"/lib/Greengrass.jar \
      --init-config "{{greengrass_installer_dir}}"/config.yaml \
      --component-default-user {{ggc_user}}:{{ggc_group}} \
      --setup-system-service true

I get a message that:

"Added ggc_user to ggc_group ",
"Successfully set up Nucleus as a system service"

However, when I check the effectiveConfig.yaml file, I see that the thing name, iot end point etc. are all
empty. Checking the greengrass logs, I see the warning:

2021-02-26T15:08:34.687Z [WARN] (main) com.aws.greengrass.deployment.IotJobsHelper: Device not configured to talk to AWS Iot cloud. IOT job deployment is offline: [thingName cannot be empty, certificateFilePath cannot be empty, privateKeyPath cannot be empty, rootCaPath cannot be empty, iotDataEndpoint cannot be empty, iotCredEndpoint cannot be empty]. {}

To Reproduce
See above.

Expected behavior
Installation is successful and core device is created in the greengrass console

Actual behavior
Core device is not created

Environment

OS: Ubuntu 20.04
JDK version: openjdk version "1.8.0_282"
Nucleus version: 2.0.4

Additional context
If I use the provision flag, and specify the thing name and other parameters as outlined in the quick install, it does work. But creates a new certificate, and attaches it to the thing. But there are 2 issues with it:

I have to provide AWS credentials to the installer
Certficates are created on the fly and not in terraform state, which is a problem.

For quick install (which was successful), I see that the greengrass client was successfully created:

com.aws.greengrass.componentmanager.GreengrassComponentServiceClientFactory: initialize-greengrass-client. {service-region=us-east-2, service-endpoint=https://greengrass-ats.iot.us-east-2.amazonaws.com:8443}

So may be there are some other arguments that need to be passed for the manual step to work?

SubscribeToComponentUpdate should accept component names as an argument

While the current implementation allows the component to listen for update messages for itself, and can defer or update status for itself, this makes it difficult to coordinate updating components when there are dependencies or communication pathways between them and the external system. It would be better to have the ability to create supervisor components, which manage the orchestration between components (similar to gen_supervisor in the OTP platform (https://erlang.org/doc/man/supervisor.html) which allows a 3rd party component to embody the logic required to handle those dependencies and requirements.

For example, suppose I have a set of sensors which are each being read by individual components that run calibration curves on the per sensor data. Each of these curves is uniquely calibrated to the specific sensor, and may be updated independently with new configuration values. These components publish their normalized data to topics which are then consumed by multiple consumers which manage specific tasks (ML inference, servo control, etc), As coordination between components is vital to the safe operation of the device, we would like to have a supervisor module which can monitor each of the update states, and be able to negotiate the proper sequence of component updates during a deployment. The supervisor also needs to be update-able too, and be able to hand over it's observation functions to another instance of itself without necessarily requiring restarting any of the components it observes.

By adding this feature it would be possible to implement that supervisor pattern, and enable a supervisor to supervise a supervisor in a supervisor tree.

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit 2a57377.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.integrationtests.ipc.IPCServicesTest.registerResourcePermissionTest failed 1 times over 10 iterations with 1 unique failures.

com.aws.greengrass.easysetup.GreengrassSetup: Invalid region causes NullPointerException

Describe the bug
Running easysetup with invalid --aws-region causes NullPointerException.

I'm not a Java expert, I just looked at. I see that some controls around awsRegion are already implemented on #883 but I faced this issue on v2.0.5. I'm not sure that why this isn't covered by these controls.

To Reproduce

Follow instructions on the Quick installation

Expected behavior
To return meaningful error message instead of NullPointerException.

Actual behavior

$ sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE \
>   -jar ./GreengrassCore/lib/Greengrass.jar \
>   --aws-region region \
>   --thing-name MyGreengrassCore \
>   --thing-group-name MyGreengrassCoreGroup \
>   --tes-role-name MyGreengrassV2TokenExchangeRole \
>   --tes-role-alias-name MyGreengrassCoreTokenExchangeRoleAlias \
>   --component-default-user ggc_user:ggc_group \
>   --provision true \
>   --setup-system-service true
Added ggc_user to ggc_group 
Error while trying to setup Greengrass Nucleus
java.lang.NullPointerException
	at com.aws.greengrass.util.RegionUtils.getGlobalRegion(RegionUtils.java:86)
	at com.aws.greengrass.util.IamSdkClientFactory.getIamClient(IamSdkClientFactory.java:47)
	at com.aws.greengrass.easysetup.DeviceProvisioningHelper.<init>(DeviceProvisioningHelper.java:155)
	at com.aws.greengrass.easysetup.GreengrassSetup.performSetup(GreengrassSetup.java:291)
	at com.aws.greengrass.easysetup.GreengrassSetup.main(GreengrassSetup.java:242)

Replacing --aws-region region with --aws-region us-east-1 allows to jump from the NPE.

Environment

OS: Raspbian GNU/Linux 10 (buster)
JDK version: 1.8.0_212
Nucleus version: 2.0.5

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit fa0fa20.
See the uploaded artifacts from the action for details.

com.aws.greengrass.easysetup.DeviceProvisioningHelperTest.GIVEN_test_update_device_config_WHEN_thing_info_provided_THEN_add_config_to_config_store failed 1 times over 10 iterations with 1 unique failures.
com.aws.greengrass.lifecyclemanager.KernelTest.GIVEN_kernel_WHEN_locate_finds_class_definition_in_config_THEN_create_service(ExtensionContext) failed 1 times over 10 iterations with 1 unique failures.
com.aws.greengrass.lifecyclemanager.KernelTest.GIVEN_kernel_with_disk_spooler_config_WHEN_locate_spooler_impl_THEN_create_test_spooler_service failed 1 times over 10 iterations with 1 unique failures.

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit 22611a7.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.integrationtests.kernel.GenericExternalServiceTest.GIVEN_service_with_timeout_WHEN_timeout_expires_THEN_move_service_to_errored failed 1 times over 10 iterations with 1 unique failures.

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit c06ef18.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.kernel.EvergreenServiceTest.GIVEN_service_WHEN_dependencies_change_THEN_service_restarts failed 1 times over 10 iterations with 1 unique failures.

(com.aws.greengrass.util.orchestration.InitUtils): Greengrass Nucleus Fails to start in Docker Container

Describe the bug
When executing the Greengrass jar inside a Docker container having an Ubuntu OS it fails with the following error:
Unable to set up Nucleus as a system service.

To Reproduce

Pulled a Docker Image having Ubuntu(18.04).
Entered into the container.
Installed Java 11 on it.
Configured AWS Credentials.
Downloaded and installed GreenGrass Nucleus as specified here.
sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassCore/lib/Greengrass.jar \ --aws-region us-east-1 --thing-name SECore \ --component-default-user ggc_user:ggc_group --provision true --setup-system-service true --deploy-dev-tools true
Install Fails with this error Unable to set up Nucleus as a system service.

Expected behavior
Install should be successful with the following message:
Successfully set up Nucleus as a system service

Actual behavior
Install Fails with this error Unable to set up Nucleus as a system service.

Environment

OS: Ubuntu 18.04 inside a docker container.
JDK version: 11
Nucleus version: AWS IoT Greengrass Core software v2.0, latest nucleus.

Additional context
This is an experiment. I was successful in running the nucleus inside an ubuntu VM but running the same inside a docker fails. I have attached the runtime logs and Greengrass error logs present under /greengrass/v2.
greengrass_v2_error_logs.txt
runtime_logs.txt

(platform resolver): Incorrectly identifies MacOS system if hostname contains "darwin" or "macos"

Describe the bug

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/config/PlatformResolver.java

Line 103 in 8a9bd02

String sysver = Exec.sh("uname -a").toLowerCase();

Uses uname -a which includes the hostname. If the hostname includes "darwin" the PlatformResolver will incorrectly identify the OS as Mac OSX.

To Reproduce
Change hostname to 'darwin' and launch Greengrass

Expected behavior
PlatformResolver shouldn't be resolving Linux machines with a hostname containing "darwin" to MacOSX

Actual behavior
Tell us what actually happened.

Environment

OS: Jetson Nano, running Ubuntu.
JDK version:
Nucleus version:

Additional context
Add any other context about the problem here.

E.g. what is the impact of the bug?

PubSubIPCEventStreamAgent.java: NullPointerException if event stream connect sends empty string for the auth token

If you are using the IPC layer and send an empty auth token in the connect message:

{ "authToken":""}

and then immediately send an aws.greengrass#SubscribeToTopic message

{"topic":"test/topic"}

the service will throw a null pointer exception. I think it is probably to do with unsafely dereferencing the auth data here:

        serviceName = context.getAuthenticationData().getIdentityLabel();

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/builtin/services/pubsub/PubSubIPCEventStreamAgent.java

Line 216 in 64fe556

serviceName = context.getAuthenticationData().getIdentityLabel();

Here's a dump of the error message:

2021-03-03T10:55:46.375Z [INFO] (Thread-4) software.amazon.awssdk.eventstreamrpc.RpcServer: New connection code [AWS_ERROR_SUCCESS] for Id 27, Class ServerConnection, Refs 1 - . {}
2021-03-03T10:55:46.440Z [ERROR] (Thread-4) com.aws.greengrass.ipc.IPCEventStreamService: Received empty auth token to authenticate IPC client. {}
2021-03-03T10:55:46.441Z [ERROR] (Thread-4) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: class java.lang.RuntimeException occurred while attempting to authN/authZ connect: Received empty auth token to authenticate IPC client. {}
java.lang.RuntimeException: Received empty auth token to authenticate IPC client
at com.aws.greengrass.ipc.IPCEventStreamService.ipcAuthenticationHandler(IPCEventStreamService.java:215)
at com.aws.greengrass.ipc.IPCEventStreamService.lambda$startup$2(IPCEventStreamService.java:104)
at software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler.onConnectRequest(ServiceOperationMappingContinuationHandler.java:85)
at software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler.onProtocolMessage(ServiceOperationMappingContinuationHandler.java:39)
at software.amazon.awssdk.crt.eventstream.ServerConnectionHandler.onProtocolMessage(ServerConnectionHandler.java:49)

2021-03-03T10:55:46.446Z [INFO] (Thread-4) software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler: Sending connect response for null. {}
Exception in thread "Thread-4" java.lang.NullPointerException
at com.aws.greengrass.builtin.services.pubsub.PubSubIPCEventStreamAgent$SubscribeToTopicOperationHandler.(PubSubIPCEventStreamAgent.java:210)
at com.aws.greengrass.builtin.services.pubsub.PubSubIPCEventStreamAgent.getSubscribeToTopicHandler(PubSubIPCEventStreamAgent.java:62)
at com.aws.greengrass.ipc.modules.PubSubIPCService.lambda$startup$0(PubSubIPCService.java:50)
at software.amazon.awssdk.eventstreamrpc.ServiceOperationMappingContinuationHandler.onIncomingStream(ServiceOperationMappingContinuationHandler.java:139)
at software.amazon.awssdk.crt.eventstream.ServerConnectionHandler.onIncomingStream(ServerConnectionHandler.java:63)

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000000, pid=6770, tid=6829
#
# JRE version: OpenJDK Runtime Environment (11.0.9.1+1) (build 11.0.9.1+1-post-Raspbian-1deb10u2)
# Java VM: OpenJDK Server VM (11.0.9.1+1-post-Raspbian-1deb10u2, mixed mode, serial gc, linux-)
# Problematic frame:
# C  0x00000000
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/pi/GreengrassCore/hs_err_pid6770.log
#
# If you would like to submit a bug report, please visit:
#   Unknown
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted

LambdaLauncher: New Lambda versions are Broken until restart

Describe the bug
After deploying a new version of an existing Lambda component, instances of that Lambda will fail to run until Greengrass restarts.

To Reproduce

Deploy a Lambda Component to a device.
Run it: it works.
Deploy a new version of the same component
Run it: it gets "Permission Denied" when trying to access some file in the "work" folder

Expected behavior
The new version should be in a working state immediately after being re-deployed.

Actual behavior
New Lambda versions are not playing well with the existing environment, and fail.

Environment

OS: Ubuntu 20.04
JDK version: 11
Nucleus version: 2.0.4
This happens both when running Lambdas without Isolation and in Greengrass containers.

Additional context
This is the same question as posed in the forum here by another user. I get exactly the same behavior.

But his solution is to manually restart the GG service. This is not doable when you have a whole fleet.

With other components, you can trigger a restart through the bootstrap script, but you cannot configure that for Lambda components. That would be a workaround, in general, you don't want this to be a need, but if there is a way to do it is fine.

(com.aws.greengrass.util.orchestration.SystemdUtils): Greengrass Nucleus Fails to start in WSL

Describe the bug
When executing Greengrass configuration inside Windows Linux Subsystem (WSL) it fails to start with the following error: Unable to setup Nucleus as system service

To Reproduce

Install WSL (Ubuntu 20.04) on Windows Server 2019 link
Install Java environment on WSL
Configure AWS credentials (as described here )
Download and install GeenGrass Nucleus (as described here )

sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE \
  -jar ./GreengrassCore/lib/Greengrass.jar \
  --aws-region us-east-1 \
  --thing-name GreengrassQuickStartCore-[id] \
  --thing-group-name GreengrassQuickStartGroup \
  --tes-role-name GGProvisioningRole \
  --tes-role-alias-name GGProvisioningRoleAlias \
  --component-default-user ggc_user:ggc_group \
  --provision true \
  --setup-system-service true

Expected behavior
It should start the service successfully

Actual behavior
Fail to start with error: Unable to setup Nucleus as system service

Environment

OS: Windows Linux Subsystem (Ubuntu 20.04) on Windows Server 2019
JDK version: 11
Nucleus version: 2.0.5

Additional context
GreenGrass error logs:

2021-03-12T03:45:16.804Z [ERROR] (Copier) com.aws.greengrass.util.orchestration.SystemdUtils: systemd-setup. {stderr=System has not been booted with systemd as init system (PID 1). Can't operate., command=systemctl daemon-reload}
2021-03-12T03:45:16.805Z [ERROR] (Copier) com.aws.greengrass.util.orchestration.SystemdUtils: systemd-setup. {stderr=Failed to connect to bus: Host is down, command=systemctl daemon-reload}
2021-03-12T03:45:16.807Z [ERROR] (main) com.aws.greengrass.util.orchestration.SystemdUtils: systemd-setup. Failed to set up systemd service. {}
java.io.IOException: Command systemctl daemon-reload failed
        at com.aws.greengrass.util.orchestration.SystemdUtils.runCommand(SystemdUtils.java:87)
        at com.aws.greengrass.util.orchestration.SystemdUtils.setupSystemService(SystemdUtils.java:46)
        at com.aws.greengrass.easysetup.GreengrassSetup.performSetup(GreengrassSetup.java:299)
        at com.aws.greengrass.easysetup.GreengrassSetup.main(GreengrassSetup.java:242)

createUser and createGroup do not support Busybox

The createUser function uses a nonportable useradd to create the user which is not available in busybox by default:

runCmd("useradd -r -m " + user, o -> {

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/util/platforms/unix/UnixPlatform.java

Line 355 in 87d89a6

runCmd("useradd -r -m " + user, o -> {

Busyboxy supports a adduser command the equivalent of which would be "adduser -S " + user (the -S is for system).

Similarly, the createGroup function uses

runCmd("groupadd -r " + group, o -> {

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/util/platforms/unix/UnixPlatform.java

Line 361 in 87d89a6

runCmd("groupadd -r " + group, o -> {

the equivalent Busybox command is

"addgroup -S " + group

It should be noted that on Debian based systems the adduser and addgroup commands are deemed more "user friendly" though they both use --system instead of -S for their runtime flags.

PublishToTopicRequest: jsonMessage does not fully support JSON

The description of the aws.greengrass#PublishToTopic documentation states that:

jsonMessage

(Optional) A JSON message. This object, JsonMessage, contains the following information:

message

    The JSON message as an object.

The underlying model does not support anything other than objects which deserialize into a Map<String,Object>.

https://github.com/aws/aws-iot-device-sdk-java-v2/blob/71fab934d591a4a3b9d1f70e66dfedadca019d70/sdk/greengrass/greengrass-client/src/event-stream-rpc-java/model/software/amazon/awssdk/aws/greengrass/model/JsonMessage.java#L30

The underlying protocol message is of the form:

{ "topic": "test/topic", "publishMessage" : { "jsonMessage": { "message" : {"foo": "bar" }}}}

While using the current Java SDK forces you to supply a Map<String,Object>, this means components would be best served by sending other data structures such as arrays, strings, numbers, booleans, or null values need to wrap them in a JSON object, or send a binary message.

If you send the message:

{"topic": "test/topic", "publishMessage": { "jsonMessage": { "message" : [ 1, 2, 3 ] }}}

Greengrass will return a result of "InternalServerError" as a raw value (unlike the typical empty json object response {}). The reason for this is that the serviceModel.fromJSON() call
throws an unexpected deserilization error when parsing the incoming message:

aws-greengrass-nucleus/src/main/java/software/amazon/awssdk/eventstreamrpc/OperationContinuationHandler.java

Line 269 in d65d433

initialRequest = serviceModel.fromJson(getRequestClass(), bytes);

This behavior both violates "the principle of least surprise" by having a "jsonMessage" IPC message that doesn't support valid JSON, and also by returning back an error message that does not conform to the same encoding as a successful request, as InternalServerError is not valid JSON.

It should also be noted that as the current design parses the full JSON payload prior to passing it to the IPC agent, this means that each payload is de-serialized and re-serialized when passing through the RPC layer. This is rather wasteful from a resource utilization perspective, and can substantially increase the runtime memory requirements for high message volume applications.

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit 070f7b6.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.integrationtests.kernel.GenericExternalServiceTest.GIVEN_service_with_timeout_WHEN_timeout_expires_THEN_move_service_to_errored failed 1 times over 10 iterations with 1 unique failures.

(recipe): Recipe variables from configuration does not work for setenv

Describe the bug
We have a recipe with a Lifecycle.Run.Setenv.site: {configuration:/site} that gets replaced with empty curly brackets, even though the configuration says configuration.site: "labpml".

To Reproduce
Create a recipe with recipe variables in the Setenv section.

Expected behavior
Lifecycle.Run.Setenv.site: {configuration:/site} should be Lifecycle.Run.Setenv.site: "labpml" when configuration.site: "labpml"

Actual behavior
Output is Lifecycle.Run.Setenv.site: {}

Environment

OS: [e.g. Ubuntu 20.04]
JDK version:
Nucleus version: newest?

Additional context
Add any other context about the problem here.

E.g. what is the impact of the bug?
We cannot have deployment specific env vars

Lambda Python logger: traceback prints are cluttered with additional text

Describe the bug
When a user calls any traceback library function, the GG logger heavily pollutes the output.

To Reproduce
call traceback.print_stack() from a lambda component.

Expected behavior
It should log a clear, readable stacktrace.

Actual behavior
It looks like this when you call traceback.print_exc(); traceback.print_stack()

2021-04-27T21:54:07.372Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,Traceback (most recent call last):. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.372Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.377Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.378Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,  File "/lambda/lambda_function.py", line 32, in on_stream_event. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.378Z [ERROR] (pool-2-thread-31) TestGPSLambda:     self.send_to_iot_core(message_string). {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.379Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.380Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.383Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,  File "/lambda/lambda_function.py", line 59, in send_to_iot_core. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.383Z [ERROR] (pool-2-thread-31) TestGPSLambda:     future.result(TIMEOUT). {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.384Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.385Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.389Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 434, in result. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.389Z [ERROR] (pool-2-thread-31) TestGPSLambda:     raise TimeoutError(). {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.389Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.391Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.394Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,concurrent.futures._base.TimeoutError. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.394Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.397Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:105,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.400Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,  File "/usr/local/lib/python3.7/dist-packages/awscrt/eventstream/rpc.py", line 627, in _on_continuation_message. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.400Z [ERROR] (pool-2-thread-31) TestGPSLambda:     flags=flags). {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.401Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.402Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.405Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,  File "/usr/local/lib/python3.7/dist-packages/awsiot/eventstreamrpc.py", line 804, in on_continuation_message. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.405Z [ERROR] (pool-2-thread-31) TestGPSLambda:     self.operation._on_continuation_message(*args, **kwargs). {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.405Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.406Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.408Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,  File "/usr/local/lib/python3.7/dist-packages/awsiot/eventstreamrpc.py", line 711, in _on_continuation_message. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.408Z [ERROR] (pool-2-thread-31) TestGPSLambda:     self._handle_data(model_name, payload). {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.409Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.410Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.412Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,  File "/usr/local/lib/python3.7/dist-packages/awsiot/eventstreamrpc.py", line 753, in _handle_data. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.412Z [ERROR] (pool-2-thread-31) TestGPSLambda:     self._stream_handler.on_stream_event(shape). {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.412Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.414Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.416Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,  File "/lambda/lambda_function.py", line 35, in on_stream_event. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.416Z [ERROR] (pool-2-thread-31) TestGPSLambda:     traceback.print_stack(). {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.416Z [ERROR] (pool-2-thread-31) TestGPSLambda: {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}
2021-04-27T21:54:07.418Z [ERROR] (pool-2-thread-31) TestGPSLambda: traceback.py:25,. {serviceInstance=0, serviceName=TestGPSLambda, currentState=RUNNING}

Environment

OS: Raspbian
JDK version: irrelevant
Python version: 3.7
Nucleus version: 2.1.0

Additional context
You can't even begin to decipher exception tracebacks. This is very bad.

Greengrass stop working after a while

Describe the bug
After some hours all components results as running, but they don't work. They don't publish/receive messages and don't print any log.
In greengrass.log continously print:
2021-05-05T08:23:35.688Z [INFO] (Thread-4) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection resumed. {clientId=84fd277845a3, sessionPresent=true}
2021-05-05T08:23:35.745Z [INFO] (pool-2-thread-36) com.aws.greengrass.deployment.ShadowDeploymentListener: Published to get named shadow topic. {}
2021-05-05T08:23:35.755Z [WARN] (Thread-4) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection interrupted. {clientId=84fd277845a3, error=The connection was closed unexpectedly.}
2021-05-05T08:23:36.191Z [INFO] (Thread-4) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection resumed. {clientId=84fd277845a3, sessionPresent=true}
2021-05-05T08:23:36.220Z [INFO] (pool-2-thread-36) com.aws.greengrass.deployment.ShadowDeploymentListener: Published to get named shadow topic. {}
...

To Reproduce
This is our configuration.
Component recipe:

{
        "RecipeFormatVersion": "2020-01-25",
        "ComponentName": "component_name",
        "ComponentVersion": "0.0.1",
        "ComponentDescription": "a component",
        "ComponentPublisher": "me",
        "ComponentConfiguration": {
            "DefaultConfiguration": {
                "containerParams": {
                    "memorySize": 16384,
                    "mountROSysfs": False,
                    "volumes": {},
                    "devices": {}
                },
                "containerMode": "NoContainer",
                "timeoutInSeconds": 3,
                "maxInstancesCount": 100,
                "inputPayloadEncodingType": "json",
                "maxQueueSize": 1000,
                "pinned": True,
                "maxIdleTimeInSeconds": 60,
                "statusTimeoutInSeconds": 60,
                "pubsubTopics": {}
            }
        },
        "Manifests": [
            {
                "Name": "Linux",
                "Platform": {
                    "os": "linux"
                },
                "Lifecycle": {
                    "Bootstrap": f"echo 'bootstrap' > /log/{component_name}.status",
                    "Install": f"echo 'install' > /log/{component_name}.status",
                    "Run": f"python3 {{artifacts:decompressedPath}}/component/main.py",
                    "Shutdown": f"echo 'shutdown' > /log/{component_name}.status",
                    "Recover": f"echo 'recover' > /log/{component_name}.status",
                },
                "Artifacts": [
                    {
                        "Uri": f"s3://{bucket_name}/component.zip",
                        "Unarchive": "ZIP",
                    }
                ]
            }
        ]
    }

Policy attached to certificate:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Connect"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Receive",
        "iot:Publish"
      ],
      "Resource": [
        "arn:aws:iot:eu-west-1:account_id:topic/$aws/things/84fd277845a3*/greengrassv2/health/json",
        "arn:aws:iot:eu-west-1:account_id:topic/$aws/things/84fd277845a3*/jobs/*",
        "arn:aws:iot:eu-west-1:account_id:topic/$aws/things/84fd277845a3*/shadow/*",
        "arn:aws:iot:eu-west-1:account_id:topic/84fd277845a3/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:Subscribe"
      ],
      "Resource": [
        "arn:aws:iot:eu-west-1:account_id:topicfilter/$aws/things/84fd277845a3*/jobs/*",
        "arn:aws:iot:eu-west-1:account_id:topicfilter/$aws/things/84fd277845a3*/shadow/*",
        "arn:aws:iot:eu-west-1:account_id:topicfilter/84fd277845a3/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "greengrass:GetComponentVersionArtifact",
        "greengrass:ResolveComponentCandidates"
      ],
      "Resource": "*"
    }
  ]
}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:AssumeRoleWithCertificate",
      "Resource": "arn:aws:iot:eu-west-1:account_id:rolealias/GreengrassCoreTokenExchangeRoleAlias"
    }
  ]
}

GreengrassCoreTokenExchangeRoleAlias Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iot:DescribeCertificate",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams",
                "iot:Connect",
                "iot:Publish",
                "iot:Subscribe",
                "iot:Receive",
                "s3:GetBucketLocation",
                
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
            ],
            "Resource": f"arn:aws:s3:::greengrass-components-artifacts-{region}/*"
        }
    ]
}

Expected behavior
Components continue to work

Actual behavior
Component stop working after a while

Environment

OS: Raspbian
JDK version: 1.8
Nucleus version: 2.0.5

Additional context
with "systemctl restart greengrass.service" everything works correctly again

accessControl configurations from older deployments are not overridden

Publishing to a topic causes UnauthorizedError because of invalid configuration of a previous version of a component

To Reproduce
Step 1:
Create a custom component that published to AWS IoT Core with incorrect accessControl configurations such as:

ComponentConfiguration:
DefaultConfiguration:
accessControl:
aws.greengrass.ipc.mqttproxy:
com.example.bugreport.mqttproxy.1:
policyDescription: "Bug reporting"
opertaions:
- "*"
resources:
- "*"

Operations is mispelled in the above configuration

Step 2
Create newer version of the custom component that published to AWS IoT Core with correct accessControl configurations such as:
ComponentConfiguration:
DefaultConfiguration:
accessControl:
aws.greengrass.ipc.mqttproxy:
com.example.bugreport.mqttproxy.1:
policyDescription: "Bug reporting"
operations:
- "*"
resources:
- "*"
Spelling of Operations is corrected

Expected behavior
Deployment of newer version of a component should remove the accessControl configuration of previously deployed version

Actual behavior
Deployment of newer version of a component causes accessControl config to be appended to previous configurations. This can be verified by the new effectiveConfig.yaml file in {GREENGRASS_PATH}/v2/config directory

Environment

OS: Ubuntu 18.04.5 LTS
JDK version: 1.8.0_275
Nucleus version: 2.0.3

Additional context
Bug impact - If a component with incorrect accessControl configuration is deployed. In future deployments, IPC will cause UnauthorizedError despite updating the component configurations.
Example:

(easysetup): iam:CreatePolicy required, even though policy exists

Describe the bug
When adopting a new Greengrass Core, we need temp credentials to do a lot of things. I don't want those credentials to be able to create new IAM policies and roles, so I created them myself, since they are reused across all adoptions. But, I cannot do the adoption since I don't have iam:CreatePolicy rights.

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/easysetup/DeviceProvisioningHelper.java

Line 399 in 8a9bd02

CreatePolicyResponse createPolicyResponse = iamClient.createPolicy(

Here you run createPolicy directly, and it will never answer that it already exists since we don't have permission to create.
I would propose running the getPolicy first for checking if it exists instead of relying on the error message.

To Reproduce
Run easysetup with minimal permissions

Expected behavior
It should say it exist and move on.

Actual behavior

Attaching TES role policy to IoT thing...
Error while trying to setup Greengrass Nucleus
software.amazon.awssdk.services.iam.model.IamException: User: arn:aws:sts::xxx:assumed-role/GreengrassCommissioningRole/tempcore-30293-setup is not authorized to perform: iam:CreatePolicy on resource: policy GreengrassV2TokenExchangeRoleAccess

Environment

OS: Raspbian Stretch
JDK version: OpenJDK8
Nucleus version: latest?

Additional context
I need to add very permissive permissions to a role that will be running in the first-run scripts of my embedded device to add it as a temp core device in greengrass.

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit e8b597a.
See the uploaded artifacts from the action for details.

com.aws.greengrass.easysetup.DeviceProvisioningHelperTest.GIVEN_test_update_device_config_WHEN_thing_info_provided_THEN_add_config_to_config_store failed 1 times over 10 iterations with 1 unique failures.
com.aws.greengrass.lifecyclemanager.KernelTest.GIVEN_kernel_WHEN_locate_finds_class_definition_in_config_THEN_create_service(ExtensionContext) failed 1 times over 10 iterations with 1 unique failures.

(EasySetup): provide an option to store certificates in a separate folder from Greengrass Root

Feature Description
Greengrass Easy Setup should provide an option to store certificates and the initial config.yaml in a separate folder from Greengrass Root. This will simplify the setup of the system by separating RO data from RW data and enable a simple factory reset of the device.

Use Case
In order to setup my device in way that allows for en easy factory reset I currently need to first install Greengrass with the following procedure

install greengrass using the Console provided command but with --deploy-dev-tools false --start false
Copy /greengrass/v2/config/effectiveConfig.yaml in a separate folder
Copy the certificates and private key in a separate folder
Edit effectiveConfig.yaml and replace the location of the certificates and private key with the paths to where you copied them

After this is done I can easily reset the device by removing the Greengrass root folder and then launch Greengrass with:

sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassCore/lib/Greengrass.jar --provision false --setup-system-service true --config <path to effectiveConfig.yaml

Proposed Solution

Add a flag to Easy Setup to specify a separate folder from the Greengrass root folder where to store the certificates and the initial config.yaml.
If this flag is provided, Greengrass stores there the device certificates, RootCA and the initial effectiveConfig.yaml.
The configuration refers to the certificates stored in this separate folder, and no certificates are stored in the greengrass root folder.

Other
Add detailed explanation, stacktraces, related issues, links for us to have context, etc

👋 I may be able to implement this feature request
⚠️ This feature might incur a breaking change

com.aws.greengrass.mqttclient.spool: New persistent storage spooler(s) : sqlite spool / filespool

Feature Description
Add new spoolers with persistent storage - either sqlite spool or a filesystem spool. These would be drop-in replacements for the InMemorySpool.

Use Case
Where greengrass devices operate in conditions where they may be offline without network access for long periods of time (days, weeks) the InMemorySpool may not be sufficient for storing proxied IoT MQTT messages. Additionally, large MQTT messages will quickly fill up the default size of the InMemoryPool even when offline for short periods of time.

Proposed Solution
A basic filesystem spool would be the simplest to write, with each message stored as a file. But persisting many small messages could potentially fill up available inodes and have poor throughput performance.
An sqlite database spool would allow persistent storage to a single datastore.sqlite file, but with more CPU overhead.

The sqlite spool would be my preferred implementation.

Some limits should still be enforced on the persistent spoolers - potentially

maxNumberOfMessages
maxSizeOnDisk (harder to evalute for sqlite?)

Other
Would this be something that the AWS team would work on internally / would you accept a PR if we implemented the feature?

👋 I may be able to implement this feature request

not seeing core device in console after install

Describe the bug
A clear and concise description of what the bug is.

After running the installation nothing is shown in the console. The config.yaml is correct, I can confirm that.

To Reproduce
Steps to reproduce the behavior. If possible, provide a minimal amount of code that causes the bug.

I followed the steps in documentation manually install the GGC, but nothing shows up in console
Added ggc_user to ggc_group Successfully set up Nucleus as a system service

Expected behavior
A clear and concise description of what you expected to happen.

Actual behavior
Tell us what actually happened.

Environment

OS: Raspbian OS 32-bit
- JDK version: 8
Nucleus version: 2.0.4

Additional context
Add any other context about the problem here.

logs:

2021-03-05T15:09:32.023Z [INFO] (main) com.aws.greengrass.util.platforms.Platform: Getting platform instance com.aws.greengrass.util.platforms.unix.UnixPlatform.. {} 2021-03-05T15:09:32.133Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: No ongoing deployment detected. Proceed as default. {} 2021-03-05T15:09:32.134Z [INFO] (main) com.aws.greengrass.config.Configuration: config-loading. Read configuration from a file path. {path=./config.yaml} 2021-03-05T15:09:32.446Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/greengrass/v2/config/effectiveConfig.yaml} 2021-03-05T15:09:34.490Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/greengrass/v2/config/effectiveConfig.yaml} 2021-03-05T15:09:34.497Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Waiting for services to shutdown. {} 2021-03-05T15:09:35.571Z [ERROR] (Copier) com.aws.greengrass.util.orchestration.SystemdUtils: systemd-setup. {stderr=Created symlink /etc/systemd/system/multi-user.target.wants/greengrass.service → /etc/systemd/system/greengrass.service., command=systemctl enable greengrass.service} 2021-03-05T15:09:36.046Z [INFO] (main) com.aws.greengrass.util.orchestration.SystemdUtils: systemd-setup. Successfully set up systemd service. {} 2021-03-05T15:09:36.046Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: system-shutdown. {main=null} 2021-03-05T15:09:36.060Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/greengrass/v2/config/effectiveConfig.yaml} 2021-03-05T15:09:36.060Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Waiting for services to shutdown. {} 2021-03-05T15:09:36.062Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.KernelLifecycle: executor-service-shutdown-initiated. {} 2021-03-05T15:09:36.062Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Waiting for executors to shutdown. {} 2021-03-05T15:09:36.062Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: executor-service-shutdown-complete. {} 2021-03-05T15:09:36.064Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: context-shutdown-initiated. {} 2021-03-05T15:09:36.066Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: context-shutdown-complete. {} 2021-03-05T15:09:36.066Z [INFO] (Serialized listener processor) com.aws.greengrass.dependency.Context: Interrupted while running tasks. Publish thread will exit now.. {}

E.g. what is the impact of the bug?

Deployment stuck in progress

Describe the bug

When I list deployments, one is stuck in progress and is blocking any other deployments:

622276f4-ea54-4ae6-a122-4d2a839f34e3: FAILED
acd4723a-1e22-4ef6-ad43-6448ff901dfb: SUCCEEDED
7f44e898-d408-4c1b-94ae-6f86b7b04409: FAILED
d539cffa-14a1-4326-a05d-9d669d0c87f0: IN_PROGRESS
c9d1e4b9-6c80-40f1-8b30-88dd5c419159: SUCCEEDED
20c9f98f-f166-49a1-9969-d9c0fe6aa8b7: FAILED
458e1c61-d303-470a-8e04-03d038586b57: SUCCEEDED
6959ea05-9ec3-4ee2-b28f-7644992e1d1b: SUCCEEDED
9fdf4359-7483-467d-a255-fddd7aace873: FAILED
3bc7dec3-68cf-4f0b-9f09-c583d1f94ab1: QUEUED

To Reproduce

I was trying to create a docker component based on these instructions: https://docs.aws.amazon.com/greengrass/v2/developerguide/run-docker-container.html

I noticed related permission errors in the component logs, which might have helped cause the issue:

2021-01-01T16:00:30.106Z [WARN] (Copier) com.example.MyDockerComponent: stderr. Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/load?quiet=1: dial unix /var/run/docker.sock: connect: permission denied. {scriptName=services.com.example.MyDockerComponent.lifecycle.Install.Script, serviceName=com.example.MyDockerComponent, currentState=NEW}

Also in an attempt to fix it, I ran systemctl restart greengrass and restarted the device.

Expected behavior

The deployment should fail rather than getting stuck in progress.

Actual behavior

According to the logs, the deployment failed, but it stayed in the IN_PROGRESS state.

2021-01-01T16:28:44.484Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.KernelLifecycle: executor-service-shutdown-initiated. {}
2021-01-01T16:28:44.483Z [WARN] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-state-transition-interrupted. Service lifecycle thread interrupted. Thread will exit now. {serviceName=FleetStatusService, currentState=FINISHED}
2021-01-01T16:28:44.485Z [INFO] (Thread-1) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Waiting for executors to shutdown. {}
2021-01-01T16:28:44.486Z [ERROR] (pool-2-thread-21) com.aws.greengrass.deployment.activator.DeploymentActivator: merge-config. Deployment failed. {deploymentId=d539cffa-14a1-4326-a05d-9d669d0c87f0}
java.util.concurrent.ExecutionException: java.lang.InterruptedException
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
	at com.aws.greengrass.deployment.DeploymentConfigMerger$AggregateServicesChangeManager.removeObsoleteServices(DeploymentConfigMerger.java:286)
	at com.aws.greengrass.deployment.activator.DefaultActivator.activate(DefaultActivator.java:85)
	at com.aws.greengrass.deployment.DeploymentConfigMerger.updateActionForDeployment(DeploymentConfigMerger.java:128)
	at com.aws.greengrass.deployment.DeploymentConfigMerger.mergeInNewConfig(DeploymentConfigMerger.java:96)
	at com.aws.greengrass.deployment.DefaultDeploymentTask.call(DefaultDeploymentTask.java:112)
	at com.aws.greengrass.deployment.DefaultDeploymentTask.call(DefaultDeploymentTask.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at com.aws.greengrass.lifecyclemanager.GreengrassService.lambda$close$2(GreengrassService.java:355)
	... 3 more

2021-01-01T16:28:44.496Z [INFO] (pool-2-thread-21) com.aws.greengrass.deployment.DeploymentService: deployment-task-execution. Finished deployment task. {deploymentId=d539cffa-14a1-4326-a05d-9d669d0c87f0, serviceName=DeploymentService, currentState=FINISHED}
2021-01-01T16:28:44.501Z [INFO] (pool-2-thread-21) com.aws.greengrass.componentmanager.ComponentStore: delete-component-start. {componentIdentifier=com.example.MyDockerComponent-v1.0.0}
2021-01-01T16:28:44.531Z [INFO] (pool-2-thread-21) com.aws.greengrass.componentmanager.ComponentStore: delete-component-finish. {componentIdentifier=com.example.MyDockerComponent-v1.0.0}

Environment

OS: Raspbian Buster
JDK version: JDK 1.8
Nucleus version: v2.0.3 (this is the latest release, and I installed it yesterday -- not sure how to confirm this)

General: Building instructions

Feature Description
General building instructions for Greengrass Nucleus is missing.

Use Case
Building the project manually on my own device would be helpful for understanding the code better.

com.aws.greengrass.lifecyclemanager: New Lifecycle stage "Uninstall"

Feature Description
We already have 'install', 'run', 'shutdown' etc as Lifecycle stages where we can configure scripts, but how about about an 'Uninstall' stage where we can perform any cleanup operations when a component is upgraded or removed.

Use Case
If my component puts temporary data in the CWD, or needs to notify another component that it's being removed, it would be good to use this 'uninstall' stage which operates like the inverse of 'install', allowing us to call scripts when the current artifact is removed.

My particular use-case was to cleanup the filesystem after an install script installs dependencies - my dependencies ended up in /home/ggc_user/.local/share/virtualenvs/{instance_id}, and weren't removed when the component artifacts were removed. I'm aware that this isn't "The Right Way" to do things, but nonetheless I feel an Uninstall licecycle stage would be useful to others.

Note: Shutdown !== Uninstall, as components can be stopped/started without being uninstalled.

Proposed Solution
Uninstall scripts should be executed before any artifact cleanup is performed.
Uninstall scripts should have a default timeout value, so that uninstalls don't hang the system indefinitely.

👋 I may be able to implement this feature request
⚠️ This feature might incur a breaking change

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit 182c17f.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.integrationtests.ipc.IPCServicesTest.registerResourcePermissionTest failed 1 times over 10 iterations with 1 unique failures.

GreenGrassV2 Core: Deviceless Provisioning

Feature Description

We should be able to provision a Greengrass core device without running the installer.

The output of this step is:

A GreengrassV2 core device in AWS.
Certificates+other artifacts to later on put on a device if we want.

Use Case

We distribute "vanilla" devices. These come with all the needed software but without any credentials.
When our clients activates our devices, they go through a setup wizard which results in downloading the certificates/credentials for their own AWS IoT thing, thus becoming an authenticated device.

With GreengrassV2, it seems that you can only provision a device when you install greengrass core??

We don't want the devices to have the ability to provision devices.

I see that there is a config-init flag in the installer, but it does not seem to be possible to provision the device outside of the installation workflow.

Proposed Solution
Please include prototype/workaround/sketch/reference implementation

The installer with --provision false seems to make sense.
New API methods should be added so we can provision the device.
The installer can then have additional input configurations where you can point it to the certificates/existing configuration.

Other
Add detailed explanation, stacktraces, related issues, links for us to have context, etc

👋 I may be able to implement this feature request
⚠️ This feature might incur a breaking change

(spooler / awsiotmqttclient): handle CLIENT_ERROR on message publish

Describe the bug
If you send an MQTT message that results in CLIENT_ERROR , the connection gets reset (from the AWS IoT platform end), then the client reconnects and re-tries sending the message. This will then cause a CLIENT_ERROR, which causes a connection reset, and then the whole process happens again.

To Reproduce
Send an MQTT message using IPC with more than 7 slashes in the topic.

Expected behavior
The message should be discarded from the spooler on error.

Actual behavior
The message stays (in the spooler?) and is retried, causing an infinite loop.

Environment
OS: Raspbian GNU/Linux 10 (buster)
JDK version: openjdk version "1.8.0_212"
Nucleus version: 2.0.4

Additional context
As the mqttclient continually retries sending the broken message, no other MQTT messages can be received as the connection to the platform gets reset very quickly - this includes shadow device subscriptions, so it could potentially cause the greengrass device to become unresponsive.

Log:

2021-02-09T17:15:00.060Z [WARN] (Thread-5) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection interrupted. {clientId=RaspiCore-1, error=The connection was closed unexpectedly.}
2021-02-09T17:15:00.264Z [INFO] (Thread-5) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection resumed. {clientId=RaspiCore-1, sessionPresent=true}
2021-02-09T17:15:00.265Z [INFO] (pool-2-thread-24) com.aws.greengrass.deployment.ShadowDeploymentListener: Published to get named shadow topic. {}
2021-02-09T17:15:00.265Z [DEBUG] (pool-2-thread-23) com.aws.greengrass.deployment.IotJobsHelper: Requesting the next deployment. {ThingName=RaspiCore-1}
2021-02-09T17:15:00.266Z [DEBUG] (pool-2-thread-24) com.aws.greengrass.deployment.DeploymentStatusKeeper: Updating status of persisted deployments to subscribers. {deploymentType=SHADOW, numberOfSubscribers=2}
2021-02-09T17:15:00.266Z [DEBUG] (pool-2-thread-23) com.aws.greengrass.deployment.DeploymentStatusKeeper: Updating status of persisted deployments to subscribers. {deploymentType=IOT_JOBS, numberOfSubscribers=2}
2021-02-09T17:15:00.288Z [WARN] (Thread-5) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection interrupted. {clientId=RaspiCore-1, error=The connection was closed unexpectedly.}
2021-02-09T17:15:01.486Z [INFO] (Thread-5) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection resumed. {clientId=RaspiCore-1, sessionPresent=true}
2021-02-09T17:15:01.488Z [INFO] (pool-2-thread-24) com.aws.greengrass.deployment.ShadowDeploymentListener: Published to get named shadow topic. {}
2021-02-09T17:15:01.488Z [DEBUG] (pool-2-thread-23) com.aws.greengrass.deployment.IotJobsHelper: Requesting the next deployment. {ThingName=RaspiCore-1}
2021-02-09T17:15:01.489Z [DEBUG] (pool-2-thread-24) com.aws.greengrass.deployment.DeploymentStatusKeeper: Updating status of persisted deployments to subscribers. {deploymentType=SHADOW, numberOfSubscribers=2}
2021-02-09T17:15:01.489Z [DEBUG] (pool-2-thread-23) com.aws.greengrass.deployment.DeploymentStatusKeeper: Updating status of persisted deployments to subscribers. {deploymentType=IOT_JOBS, numberOfSubscribers=2}
2021-02-09T17:15:01.508Z [WARN] (Thread-5) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection interrupted. {clientId=RaspiCore-1, error=The connection was closed unexpectedly.}

Unquoted characters in JSON formatted error message

I’ve been trying to deploy 2.0.4 to a device already running 2.0.3. It’s failing and I’m still trying to get to the bottom of it. Along the way I noticed this line in the JSON formatted log:
{"error": "No serializer found for class com.fasterxml.jackson.core.util.DefaultPrettyPrinter and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: com.aws.greengrass.logging.impl.GreengrassLogMessage["cause"]->java.io.IOException["cause"]->com.fasterxml.jackson.databind.JsonMappingException["cause"]->com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException["processor"]->com.fasterxml.jackson.dataformat.yaml.YAMLParser["codec"]->com.fasterxml.jackson.databind.ObjectMapper["serializationConfig"]->com.fasterxml.jackson.databind.SerializationConfig["defaultPrettyPrinter"])"}
It’s a simple key-value pair, where the value is a very long string. But the string contains ” characters that are not \ quoted — the JSON parser, when parsing this message, says:
Unexpected character ('c' (code 99)): was expecting comma to separate Object entries

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit 00ed279.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.config.ConfigurationTest.hmm failed 10 times over 10 iterations with 1 unique failures.

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit dad1569.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.integrationtests.ipc.IPCServicesTest.lifecycleTest failed 1 times over 10 iterations with 1 unique failures.

(com.aws.greengrass.lifecyclemanager): Separate directory for all write operations

Feature Description
Greengrass v2 is deployed under a single root directory. This feature add the possibility to configure to use a separate directory for all write operations, including creating directories and files.

Use Case
We have a use case where our device has limited storage to store the FW at manufacturing but additional space on a different device (e.g. SD card). We also have a requirement to configure in read-only mode the FW partition.

We would like to keep a minimal version of Greengrass (just the nucleus component) bundled with the FW and use the additional storage for everything else where Greengrass needs write-permission (package, log, etc).

Proposed Solution
This feature is available in Greengrass v1.
In the nucleus configuration we should have a parameter to define the write-directory path.

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit 3be32e3.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.integrationtests.deployment.DeploymentConfigMergingTest.GIVEN_kernel_running_single_service_WHEN_merge_change_adding_nested_dependency_THEN_dependent_services_start_and_service_restarts failed 1 times over 10 iterations with 1 unique failures.

(nucleus): Infinite loop broken pipe in logs

Describe the bug
Tried to create a new version to deploy to one of my cores now that has been running for a couple of weeks, and the deployment just stands as in progress. Logged in to the core and checked the greengrass.log and it says something about failing to fetch credentials due to timeouts and broken pipe.

This issue appears in the logs today at 2021-04-09T03:01:00.218Z, and the previous log statement then was at
2021-03-29T23:50:00.700Z [INFO] (pool-2-thread-38) com.aws.greengrass.tes.CredentialRequestHandler: Received IAM credentials that will be cached until 2021-03-30T00:45:00Z. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}

To Reproduce
Keep greengrass running for an extended period, then try to deploy.

Expected behavior
The application to recover, and deployment to succeed without manual intervention.

Actual behavior
Nothing happens part from logs full of errors.

Environment

OS: Raspbian Lite Buster
JDK version: openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-post-Raspbian-1deb10u2)
OpenJDK Server VM (build 11.0.9.1+1-post-Raspbian-1deb10u2, mixed mode)
Nucleus version: 2.0.5

Additional context
The issue makes it impossible to deploy new versions to the core.

2021-04-09T13:01:00.355Z [ERROR] (pool-2-thread-5390) com.aws.greengrass.iot.IotConnectionManager: Getting connection failed for endpoint https://c31e8o2fzpui1i.credentials.iot.eu-central-1.amazonaws.com with error {} . {}
java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
	at com.aws.greengrass.iot.IotConnectionManager.getConnection(IotConnectionManager.java:111)
	at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:76)
	at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:168)
	at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:254)
	at com.aws.greengrass.tes.CredentialRequestHandler.handle(CredentialRequestHandler.java:133)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:692)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:664)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

2021-04-09T13:01:00.358Z [WARN] (pool-2-thread-5390) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
com.aws.greengrass.deployment.exceptions.AWSIotException: java.util.concurrent.TimeoutException
	at com.aws.greengrass.iot.IotConnectionManager.getConnection(IotConnectionManager.java:114)
	at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:76)
	at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:168)
	at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:254)
	at com.aws.greengrass.tes.CredentialRequestHandler.handle(CredentialRequestHandler.java:133)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:692)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:664)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
	at com.aws.greengrass.iot.IotConnectionManager.getConnection(IotConnectionManager.java:111)
	... 13 more

2021-04-09T13:01:00.361Z [INFO] (pool-2-thread-5390) com.aws.greengrass.tes.CredentialRequestHandler: Request failed. {}
java.io.IOException: Broken pipe
	at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
	at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:463)
	at jdk.httpserver/sun.net.httpserver.Request$WriteStream.write(Request.java:391)
	at jdk.httpserver/sun.net.httpserver.FixedLengthOutputStream.write(FixedLengthOutputStream.java:78)
	at java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108)
	at jdk.httpserver/sun.net.httpserver.PlaceholderOutputStream.write(ExchangeImpl.java:444)
	at com.aws.greengrass.tes.CredentialRequestHandler.handle(CredentialRequestHandler.java:135)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:692)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:664)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

[Bot] Flaky Test(s) Identified

Flaky test(s) found for commit 1c56b5d.
See the uploaded artifacts from the action for details.

com.aws.iot.evergreen.integrationtests.ipc.IPCServicesTest.lifecycleTest failed 2 times over 10 iterations with 2 unique failures.
com.aws.iot.evergreen.integrationtests.kernel.KernelShutdownTest.WHEN_kernel_shutdown_THEN_services_are_shutdown_in_reverse_dependecy_order failed 1 times over 10 iterations with 1 unique failures.

SubscribeToTopic: blobs are mutated and not RFC4648 compliant

When one subscribes to a topic using the SubscribeToTopic and sends the binary message 'hello world' as base64 encoded binary message as:

{ "topic": "test/topic", "publishMessage": "binaryMessage" : { "message" : "aGVsbG8gd29ybGQ=" }}}

The subscriber receives the message:

{ "binaryMessage": {"message":""aGVsbG8gd29ybGQ\u003d"}}

And while this is a valid UTF-8 string, it is not a RFC4648 compliant encoding. This then requires the subscriber to decode the UTF-8 string to get a valid base64 encoded string to then decode. This is a very non-obvious behavior. As the '=' character is already a regular UTF-8 code point well supported by 7 bit channels for decades, this double encoding is detrimental.

It should also be noted that this also demonstrates that the message is getting deserialized during processing (producing unnecessary processing overhead), and then being mutated by the IPC system. This violates most users expectations of how a IPC transit should function as the blob should be treated as an opaque value.

Additionally, the base64 encoding already produces a 33% overhead, and this double encoding only add more unnecessary overhead to each binaryMessage.

Null pointer exception when no version supplied via an --init-config

If you omit the version of the Nucleus component in your --init-config file

{
  "system": {
    "certificateFilePath": "/greengrass/gg-config/certs/test123.pem",
    "privateKeyPath": "/greengrass/gg-config/certs/test123.pem",
    "rootCaPath": "/greengrass/gg-config/certs/AmazonRootCA1.pem",
    "thingName": "test123",
    "rootpath": "/greengrass/"
  },
  "services": {
    "aws.greengrass.Nucleus": {
      "componentType": "NUCLEUS",
      "configuration": {
        "awsRegion": "eu-west-1",
        "iotRoleAlias": "GreengrassV2TokenExchangeRoleAlias",
        "iotDataEndpoint": "a3a00z24dknd3b-ats.iot.eu-west-1.amazonaws.com",
        "iotCredEndpoint": "c1w19xtvvf60nl.credentials.iot.eu-west-1.amazonaws.com",
        "mqtt": {
          "port": 443
        },
   ...
}

And then you will get a null pointer exception during a deployment:

2021-04-06T13:32:23.607Z [INFO] (pool-2-thread-10) com.aws.greengrass.deployment.DeploymentService: Current deployment finished. {DeploymentId=f633f88d-1f4d-4d72-9fe5-24d84624d541, serviceName=DeploymentService, currentState=RUNNING}
2021-04-06T13:32:23.609Z [ERROR] (pool-2-thread-10) com.aws.greengrass.deployment.DeploymentService: Deployment task throws unknown exception. {DeploymentId=f633f88d-1f4d-4d72-9fe5-24d84624d541, serviceName=DeploymentService, currentState=RUNNING}
java.lang.NullPointerException
        at com.aws.greengrass.lifecyclemanager.GenericExternalService.isBootstrapRequired(GenericExternalService.java:235)
        at com.aws.greengrass.deployment.bootstrap.BootstrapManager.serviceBootstrapRequired(BootstrapManager.java:332)
        at com.aws.greengrass.deployment.bootstrap.BootstrapManager.lambda$isBootstrapRequired$0(BootstrapManager.java:131)
        at java.base/java.util.HashMap.forEach(HashMap.java:1336)
        at com.aws.greengrass.deployment.bootstrap.BootstrapManager.isBootstrapRequired(BootstrapManager.java:130)
        at com.aws.greengrass.deployment.activator.DeploymentActivatorFactory.getDeploymentActivator(DeploymentActivatorFactory.java:34)
        at com.aws.greengrass.deployment.DeploymentConfigMerger.mergeInNewConfig(DeploymentConfigMerger.java:69)
        at com.aws.greengrass.deployment.DefaultDeploymentTask.call(DefaultDeploymentTask.java:112)
        at com.aws.greengrass.deployment.DefaultDeploymentTask.call(DefaultDeploymentTask.java:36)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

And the deployment will enter a fail state. Nucleus knows its version however, and shouldn't depend on this input for the configuration.

Unable to get Tunneling working on GGv2 on Ubuntu 20.04

Describe the bug
Hello AWS team, I might be missing something with the public component, at first sight everything is OK, however I'm seeing may errors in the aws.greengrass.SecureTunneling.log and I constantly seeing Proxy server rejected web socket upgrade request: (HTTP/1.1 403 Forbidden) "Invalid access-token"

To Reproduce
I did have ggv2 in ubuntu 16.04LTS, due to potential security requirements, I also tried in Ubuntu 20.04 with the same results.
Steps to reproduce the behavior. If possible, provide a minimal amount of code that causes the bug.

Installed ggv2 from scratch in Ubuntu 20.04, all normal

I deployed the component (just this one)

all seems normal
I did create a tunneling and to make sure, I did subscribe to the topic. All looked great

With my tunneling open, now I do try to activate my localproxy

I did checked the content of the token, all seems to match. I also compared the destination token and matched with the MQTT message.

I also force my laptop to accept outbound connection iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT

Expected behavior
I was expecting an smooth tunneling, just like the component installation.

Actual behavior
alway I got the Proxy server rejected web socket upgrade request: (HTTP/1.1 403 Forbidden) "Invalid access-token"
Checking ggv2 logs i see may errors.

Environment

OS: Ubuntu 20.04.2
JDK version: openjdk version "1.8.0_282"
Nucleus version: aws.greengrass.Cli-v2.0.4

Additional context

I wondering if there is a missing priviledge somewhere. I suspected on the localproxy I had, so I decided to compile it, took a while, however the same error. I'm including the logs aws.greengrass.SecureTunneling.log
Appreciate your support.
thanks
Keiner

2021-02-28T02:41:51.338Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2021-02-27 20:41:51.338 [Thread-1] SecureTunnelingTask - Successfully subscribed to topic: $aws/things/keiner_laptop/tunnels/notify. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceNa
me=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.631Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [INFO ] 2021-02-27 20:44:05.630 [Thread-1] SubscribeResponseHandler - Received new tunnel notification message.. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunn
eling, currentState=RUNNING}
2021-02-28T02:44:05.644Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.643 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2021-02-27T20:44:05.640Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/tmp/' is not set to recommended va
lue... {Permissions: {desired: 745, actual: 777}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.646Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.646 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2021-02-27T20:44:05.640Z [WARN]  {FileUtils.cpp}: Permissions to given file/dir path '/tmp/device-client-settings.json131
31856035475339241614480110832' is not set to recommended value... {Permissions: {desired: 644, actual: 664}}. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.648Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.647 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2021-02-27T20:44:05.640Z [INFO]  {Config.cpp}: Successfully fetched JSON config file: {. {scriptName=services.aws.greengr
ass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.649Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.647 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "endpoint": "replace_with_endpoint_value",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.649Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.647 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "cert": "replace_with_certificate_file_location",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.649Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.647 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "key": "replace_with_private_key_file_location",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.649Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.647 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "root-ca": "replace_with_root_ca_file_location",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.649Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.648 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "thing-name": "replace_with_thing_name",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.650Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.648 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "logging": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.650Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.648 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "level": "ERROR",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.650Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.648 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "type": "STDOUT",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.650Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.649 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "file": "/var/log/aws-iot-device-client/aws-iot-device-client.log". {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.650Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.649 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   },. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.651Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.649 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "jobs": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.651Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.649 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "enabled": false,. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.651Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.649 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "handler-directory": "replace_with_path_to_handler_dir". {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.652Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.649 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   },. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.652Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.650 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "tunneling": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.652Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.650 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "enabled": true. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.652Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.650 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   },. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.654Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.650 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "device-defender":    {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.654Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.650 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "enabled":  false,. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.650 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "interval": 300. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.650 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   },. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.650 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   "fleet-provisioning": {. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.651 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "enabled": false,. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.652 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "template-name": "replace_with_template_name",. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.652 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:     "csr-file": "replace_with_csr-file-path". {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.652 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process:   }. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.652 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: }. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}
2021-02-28T02:44:05.655Z [INFO] (Copier) aws.greengrass.SecureTunneling: stdout. [ERROR] 2021-02-27 20:44:05.652 [pool-3-thread-1] SubscribeResponseHandler - Secure Tunneling Process: 2021-02-27T20:44:05.640Z [DEBUG] {Config.cpp}: Did not find a runtime configuration file, assuming Fleet Provisioning has not run for this device. {scriptName=services.aws.greengrass.SecureTunneling.lifecycle.run.script, serviceName=aws.greengrass.SecureTunneling, currentState=RUNNING}

E.g. what is the impact of the bug?

Recipe to include a Default Configuration schema for the component

Feature Description
Components that accept a Default Configuration should also provide a schema for the properties, their type, usage description, allowed values for such configuration.

Use Case
As a user, when deploying a component, I would like to know which is the configuration I can provide for the component. This information can also be used to generate a richer UX in the console.

Proposed Solution
Provide an additional Key as part of the Recipe to define a schema for the Default Configuration of the compoenent.

Other

👋 I may be able to implement this feature request
⚠️ This feature might incur a breaking change

getChildPids does not work on Busybox based Linuxes

The function getChildPids uses the ps command and flags not supported by Busybox

    Process proc = Runtime.getRuntime().exec(new String[]{"ps", "-ax", "-o", "pid,ppid"});

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/util/platforms/unix/UnixPlatform.java

Line 489 in 87d89a6

 Process proc = Runtime.getRuntime().exec(new String[]{"ps", "-ax", "-o", "pid,ppid"}); 

Busyboxy supports -o and -T and that is it,

cf. https://busybox.net/downloads/BusyBox.html
cf. https://git.busybox.net/busybox/tree/procps/ps.posix

One way to get the list of all the child processes for a given process that works on on Linux systems is to readdir on
/proc/{pid}/task/ to get he list of child processes. You can check the parent process for each by reading the 4th field
of the stat file /proc/{pid}/task/{tid}/stat.

The process group id is also available in the 5th field. (In the case of my running core there are 37 child processes in this group for example).

com.aws.greengrass.util.Permissions: Needs to ignore symlinks

Describe the bug
When deploying code components to greengrass that setup symlinks (e.g. pipenv creating a virtual environment for python dependencies), the package installer executes Permissions.setArtifactPermissions on that archive directory. This causes errors the second time the code is executed when files on the filesystem already exist and include symlinks to (for example) system libraries like python:

com.aws.greengrass.componentmanager.exceptions.PackageDownloadException: Failed to change permissions of component com.switchsystems.REDACTED1.0.14 artifact ComponentArtifact(artifactUri=s3://REDACTED1/artifacts_1.0.2.zip, checksum=PhlQGRzARtlZc4Y/RMZA67cadpo0nwzEFr8QoOjz64o=, algorithm=SHA-256, unarchive=ZIP, permission=Permission(read=OWNER, execute=NONE))
        at com.aws.greengrass.componentmanager.ComponentManager.prepareArtifacts(ComponentManager.java:412)
        at com.aws.greengrass.componentmanager.ComponentManager.preparePackage(ComponentManager.java:333)
        at com.aws.greengrass.componentmanager.ComponentManager.lambda$preparePackages$1(ComponentManager.java:316)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.nio.file.FileSystemException: /greengrass/v2/packages/artifacts-unarchived/REDACTED/artifacts_1.0.2/.venv/include/python3.7m: Too many levels of symbolic links or unable to access attributes of symbolic link
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:96)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:254)
        at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:276)
        at com.aws.greengrass.util.platforms.unix.UnixPlatform.lambda$setPermissions$10(UnixPlatform.java:382)
        at com.aws.greengrass.util.platforms.unix.UnixPlatform.setPermissions(UnixPlatform.java:411)
        at com.aws.greengrass.util.platforms.Platform.setPermissions(Platform.java:114)
        at com.aws.greengrass.util.Permissions.setArtifactPermission(Permissions.java:46)
        at com.aws.greengrass.util.Permissions.setArtifactPermission(Permissions.java:49)
        at com.aws.greengrass.util.Permissions.setArtifactPermission(Permissions.java:49)
        at com.aws.greengrass.util.Permissions.setArtifactPermission(Permissions.java:49)
        at com.aws.greengrass.componentmanager.ComponentManager.prepareArtifacts(ComponentManager.java:408)

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/componentmanager/ComponentManager.java

Line 408 in 95e68fa

Permissions.setArtifactPermission(unarchivePath,

The first time this runs, everything is fine.
My component then has a 'setup' lifecycle step which creates symlinks in unarchivePath/.venv. The next time a deployment comes along (e.g. a different component is modified) the exsting package is re-prepared, and the code executes on the existing files in unarchivePath , which then causes the error above.

To Reproduce

Deploy a component to the core device
On the core device's filesystem, create a symlink inside the /packages/artifacts/your-component which links to something external, e.g. ln -s /usr/include/python3.7m ./include/python3.7m
Re-run a deploy to the device

Expected behavior
Permissions should be set

Actual behavior
FileSystemException is thrown as the script cannot set permissions on files outside of the artifacts directory when following symlinks

Environment

OS: Debian GNU/Linux 10 (buster)
JDK version: openjdk 11.0.9.1
Nucleus version: 2.0.3

Proposed Fix
This could be fixed by ignoring symlinks inside the function

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/util/Permissions.java

Line 40 in 95e68fa

 public static void setArtifactPermission(Path p, FileSystemPermission permission) throws IOException { 

(FleetStatusService): Healthy state interval

Feature Description
Being able to set short intervals/threshold for healthy checks, like 10 minutes. IoT core devices being down for 24 hours could could be critical.

Use Case
Instead of implementing own service for health checks, using the output from aws greengrassv2 list-core-devices would be great. We would want a warning after 10-30 minutes. Customers will call complaining before the 24 hour mark if we don't want to build our own ping service.

Proposed Solution
Be able to set the threshold of what is healthy and not ourselves.

Other

👋 I may be able to implement this feature request
⚠️ This feature might incur a breaking change

Installation doesn't work from behind a proxy

Hi,

I'm trying to run the installation from behind a corporate proxy.

From this box, I am only allowed to reach the internet through an http proxy.
Also name resolution happens through the proxy.
I managed to make SQS work though the proxy

Using a custom init-config and all the operations that are using the aws sdk are working, but it fails when it's trying to download the rootCA. That part of the install script is using the java.net and java.nio and apparently they don't use the proxy:

aws-greengrass-nucleus/src/main/java/com/aws/greengrass/easysetup/DeviceProvisioningHelper.java

Line 285 in 4da4e9b

private void downloadFileFromURL(String url, File f) throws IOException {

I've also tried to specify -Djava.net.useSystemProxies=true but still it doesn't work. Here is the log:

sudo -E java -Djava.net.useSystemProxies=true  -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassCore/lib/Greengrass.jar --aws-region eu-central-1 --thing-name GreengrassQuickStartCore-177876a9adf --thing-group-name GreengrassQuickStartGroup --component-default-user ggc_user:ggc_group --provision true --setup-system-service true --deploy-dev-tools true --init-config /tmp/effectiveConfig.yaml
Added ggc_user to ggc_group
Provisioning AWS IoT resources for the device with IoT Thing Name: [GreengrassQuickStartCore-177876a9adf]...
Found IoT policy "GreengrassV2IoTThingPolicy", reusing it
Creating keys and certificate...
Attaching policy to certificate...
Creating IoT Thing "GreengrassQuickStartCore-177876a9adf"...
Attaching certificate to IoT thing...
Successfully provisioned AWS IoT resources for the device with IoT Thing Name: [GreengrassQuickStartCore-177876a9adf]!
Adding IoT Thing [GreengrassQuickStartCore-177876a9adf] into Thing Group: [GreengrassQuickStartGroup]...
IoT Thing Group "GreengrassQuickStartGroup" already existed, reusing it
Successfully added Thing into Thing Group: [GreengrassQuickStartGroup]
Setting up resources for aws.greengrass.TokenExchangeService ...
Attaching TES role policy to IoT thing...
IAM policy named "GreengrassV2TokenExchangeRoleAccess" already exists. Please attach it to the IAM role if not already
Configuring Nucleus with provisioned resource details...
Downloading Root CA from "https://www.amazontrust.com/repository/AmazonRootCA1.pem"
Error while trying to setup Greengrass Nucleus
java.net.UnknownHostException: www.amazontrust.com
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:607)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:284)
        at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
        at java.net.URL.openStream(URL.java:1068)
        at com.aws.greengrass.easysetup.DeviceProvisioningHelper.downloadFileFromURL(DeviceProvisioningHelper.java:286)
        at com.aws.greengrass.easysetup.DeviceProvisioningHelper.downloadRootCAToFile(DeviceProvisioningHelper.java:278)
        at com.aws.greengrass.easysetup.DeviceProvisioningHelper.updateKernelConfigWithIotConfiguration(DeviceProvisioningHelper.java:310)
        at com.aws.greengrass.easysetup.GreengrassSetup.provision(GreengrassSetup.java:425)
        at com.aws.greengrass.easysetup.GreengrassSetup.performSetup(GreengrassSetup.java:294)
        at com.aws.greengrass.easysetup.GreengrassSetup.main(GreengrassSetup.java:242)

aws-greengrass / aws-greengrass-nucleus Goto Github PK

aws-greengrass-nucleus's Introduction

Greengrass Nucleus

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

SPDX-License-Identifier: Apache-2.0

A quick tour through com.aws.greengrass

Learn more

aws-greengrass-nucleus's People

Contributors

Stargazers

Watchers

Forkers

aws-greengrass-nucleus's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs