awslabs / aws-greengrass-group-setup

File-driven creation of an entire AWS Greengrass group

aws-greengrass-group-setup

Greengrass gg_group_setup is an example file-driven approach to the creation of an entire AWS Greengrass group.

This repository demonstrates how one might automate the following discrete steps, but if you're doing full, complex, deployment management of Greengrass Groups you will want to explore the new Greengrass CloudFormation support and read this blog post.

Usually the following discrete steps are necessary to setup and deploy a Greengrass group.

• create_thing
• create_core_definition
• create_device_definition
• create_function_definition
• create_logger_definition
• create_subscription_definition
• create_deployment

gg_group_setup provides functioning example code of how a Greengrass Group is created. It also provides a:

• gg_group_setup config file parser GroupConfigFile which can be sub-classed
• file-driven command line interface encapsulated in the GroupComnands class
  • The GroupCommands class is also an example implementation of the steps necessary to create a Greengrass Group.
• GroupType which can be sub-classed for more complex GroupCommands scenarios

gg_group_setup includes multiple commands: create, deploy, create-core, create-devices, clean-core, clean-devices, clean-file, and clean-all.

After installation you can use these commands from the Command Line Interface, or you can use them from within a program via the GroupCommands class.

Installation

The quickest way to get gg_group_setup is to install the latest stable version via pip.

pip install gg-group-setup

After installation, for command line help type:

gg_group_setup create -- --help
gg_group_setup deploy -- --help
gg_group_setup create-core -- --help
gg_group_setup create-devices -- --help
gg_group_setup clean-core -- --help
gg_group_setup clean-devices -- --help
gg_group_setup clean-file -- --help
gg_group_setup clean-all -- --help
gg_group_setup associate_devices -- --help

Quick Start

The high-level process to create a Greengrass group using gg_group_setup is as follows:

1. Execute $ gg_group_setup create-core <thing_name> <config_file> -- to create a Greengrass Core named thing_name
2. Execute $ gg_group_setup create-devices '[<device_thing_name_01>,<device_thing_name_02,...]' <config_file> -- to create things for use as devices in your Greengrass Group.
3. Create and alias your Lambda function(s)
4. Update the group <config_file>. Example: cfg.json
   1. update the lambda_functions section

      1. In the lambda_functions section of the configuration, replace <function_name> with the name of the Lambda function configured and aliased previously. Then for that function enter the arn and arn_qualifier of the function.

         "lambda_functions": {
           "<function_name>": {
             "arn": "<lambda_ARN>",
             "arn_qualifier": "<alias>"
             "environment_variables": {
               "<name>": "<value>",
             }
           }
         },

      2. For example, if the Lambda function is created in us-west-2, named MyFirstGreengrassLambda, and the alias named dev pointing to version 1, the lambda_functions section would contain these values.

         "lambda_functions": {
           "MyFirstGreengrassLambda": {
             "arn": "arn:aws:lambda:us-west-2:<account_id>:function:MyFirstGreengrassLambda:dev",
             "arn_qualifier": "dev"
             "environment_variables": {
               "SAY_HELLO_TO": "Miss. Mocking Bird",
             }
           }
         },

         • If you need more than one function in the group named MyFirstGreengrassLambda and MockDeviceLambda, the lambda_functions section would contain these values.

           "lambda_functions": {
             "MyFirstGreengrassLambda": {
               "arn": "arn:aws:lambda:us-west-2:<account_id>:function:MyFirstGreengrassLambda:dev",
               "arn_qualifier": "dev"
               "environment_variables": {
                 "SAY_HELLO_TO": "Miss. Mocking Bird",
               }
             },
             "MockDeviceLambda": {
               "arn": "arn:aws:lambda:us-west-2:<account_id>:function:MockDeviceLambda:dev",
               "arn_qualifier": "dev"
               "environment_variables": {
                 "SAY_GOODBYE_TO": "Mr. Sleepy Owl",
                 "SAY_GOODBYE_AT": "8:00 PM",
               }
             }
           },

   2. update the subscriptions section
      1. the subscriptions section should reflect the topics the Lambda functions and devices in the group use to communicate with each other and the cloud. For example, the MockDevice Lambda function expects to use the following subscriptions:

         "subscriptions": {
           "errors": "/errors",
           "telemetry": "/telemetry"
         }

5. Download the Greengrass software and follow these instructions to extract the software onto the Greengrass core.
6. Install the Greengrass core's certificates onto the core device
7. Start your Greengrass core
8. Execute $ gg_group_setup create <group_type> <config_file> -- to create the Greengrass group
9. Execute $ gg_group_setup deploy <config_file> -- to deploy the Greengrass group

Note: gg_group_setup also includes a Mock Device Lambda function you can use to get started.

Using gg_group_setup as a Library

After the Quick Start, you will probably want to configure your own unique Greengrass group with its own Lambda functions, devices, and subscription topology. To do this you will need to implement a sub-class of GroupType.

In the gg_group_setup folder you will see an example mock_device Lambda function and a mock_group.

After implementing a sub-class of GroupType called CustomGroupType, update the group <config_file> to reflect the custom group. The custom group can then be used in code as follows:

    config_file = "<filename>"  # filename of the group's <config_file>
    group_name = "<group_name>"  # if `None`, the group_type value will be used
    region = "<aws_region>"  # AWS region in which the group will be created
 
    gc = GroupCommands(group_types={
        CustomGroupType.CUSTOM_TYPE: CustomGroupType
    })
    gc.create(
        config_file, group_type=CustomGroupType.CUSTOM_TYPE, 
        group_name=group_name, region=region
    )

Entering an account_id

If you add your account ID to the config_file manually before running any commandsgg_group_setup will use the account ID to make a notch more restrictive Core and Device policies. Specifically, if you enter a value here:

  ...
  "misc": {
    "account_id": "<account_id_here>",
    "iot_endpoint": ""
  },...

These instructions show you how to find your AWS account ID.

Using the config_file manually

Although the helper commands create-core and create-devices will create things and store the proper information in the given config_file, you can also ignore those commands if you want to enter all the information manually. The create, deploy, and clean-all Greengrass group commands will use the config_file information as stored.

Projects using gg_group_setup

The aws-greengrass-mini-fulfillment repository creates and uses pretty complex custom group types.

aws-greengrass-group-setup's Issues

Create Quick Start guide

Create a quick start guide that covers the example file format and CLI use.

incompatible with recent awscli versions

It seems the botocore version pulled in by gg_group_setup (via boto3) is botocore-1.5.95.
The latest awscli package seems to want botocore-1.7.0.

With the old botocore the new cli barfs with:
Traceback (most recent call last):
File "/usr/local/bin/aws", line 19, in
import awscli.clidriver
File "/home/jim/.local/lib/python2.7/site-packages/awscli/clidriver.py", line 35, in
from awscli.help import ProviderHelpCommand
File "/home/jim/.local/lib/python2.7/site-packages/awscli/help.py", line 27, in
from awscli.clidocs import ProviderDocumentEventHandler
File "/home/jim/.local/lib/python2.7/site-packages/awscli/clidocs.py", line 18, in
from botocore.utils import is_json_value_header
ImportError: cannot import name is_json_value_header

clean does not work

--clean-all doesnot work properly. We still need to do a --clean-file prior to the next run.

create group issue when Core created previously

create group incorrectly assesses is_fresh when Core created previously.

Profile name not actually being used

Looks like GroupCommands accepts a profile_name parameter, but afterwards it never uses it.

_get_iot_session function also takes in a profile_name as a parameter, but no usages of this function actually pass in the profile_name.

This should be remedied so that people can pass in different profiles to edit/create Greengrass groups in different accounts.

The current work around is to change the default in ~/.aws/credentials depending on which account you want to modify.

Blank region name defaults to us-west-2

If gg_group_setup is executed without --region parameter, the core and the group always gets created in us-west-2. It does not retrieve the region from AWS user profile.

Unable to add deivces to GG group using CLI Commands

These are the commands that I used to create a group:
gg_group_setup create-core --thing-name GGCore --config-file sampleconfig.json --region us-east-1 --cert-dir corecerts
gg_group_setup create-devices --thing-names '[device1, device2, device3]' --config-file sampleconfig.json --region us-east-1 --cert-dir devicecerts
Started greengrass daemon in the core
gg_group_setup create --group-type mock --config-file sampleconfig.json --region us-east-1
gg_group_setup deploy --config-file sampleconfig.json --region us-east-1

Deployment is successful but the group is created with a core and no devices. The things has been created in IoT but not attached to the group.

Is there any limitations while using the gg_group_setup CLI or is it possible only by writing a custom groupType using gg_group_setup as a library?

Make a new patch release version v0.5.4 with latest PR merges.

I need a stable release tag to pull for my project that contains the following PR merge:

• Issue #22 and Pull Request #23

Can you kindly make a release version v0.5.4 or v0.6.0 accordingly please?

Support Creating Mock Group without Devices

It would be great if gg_group_setup could support create-core followed immediately by create-group. At this point in time there's an assumption in the code that there will always be devices created before a create-group is called. Came from conversation with: @neelmitra

Add an EmptyGroupType

Add a group type that is as empty as possible.

Check for valid regions in GroupCommands

The create, clean_all, and deploy functions should ensure they are given valid regions if the region value is not None. Likely this is a simple comparison between the result of invoking boto3.client('ec2').describe_regions() with the given region value.

create-devices issue when Core created previously

When one creates a Core prior to executing create-devices, the create-devices command thinks the configuration file is not "fresh".

Group Commands > Create Function Definition doesn't honor region

Steps to reproduce:

1. Configure AWS credentials with something other than us-west-2 (e.g. us-east-1).
2. Create a GreenGrass Group with at least one lambda function (some code path executes GroupCommand.create) and make sure the specified region doesn't match your AWS config.

Expected result:

Lambda function lookups should be correctly scoped to the region specified as parameters.

Actual result:

Lambda function lookups aren't performed in the region specified as parameters; the default specified in AWS credentials is used.

Suggested fix:

cmd.py
GroupCommands._create_function_definition

Change this line –
aws = boto3.client('lambda')
– to this –
aws = boto3.client('lambda', region_name=region)

How to download the Greengrass software?

Greetings,

I'm working on Greengrass deployment automation and have created a script similar to this one (didn't know it existed when I started). And now I found that I cannot find any Greengrass software download links in the documentation. Do you know of any way to download the software without having to create the group and core using AWS Console?

Thank you,
Shavkat

No support for environment variables for Lambda Function Definition, under Greengrass Group.

I use Environment Variables on Lambda Function Definitions to keep my apps aligned with 12factor app principles.

The current cfg.json#lambda_functions/<functionName> and gg_group_setup create doesn't allow me to do this in anyway.

Functions: Cannot specify 'Pinned' value

As the API is currently exposed, lambda functions referenced with gg_group_setup are on-demand by default with no option to set them to pinned. I need a way to specify that a lambda function is long-lived / pinned – see create_function_definition.

I've been able hack in support by overriding GroupCommands._create_function_definition in a subclass.

After subclassing, the relevant config –
"lambda_functions": { "function_name": { "arn": "function_arn", "arn_qualifier": "dev" }
– is updated to look like this –
"lambda_functions": { "function_name": { "arn": "function_arn", "arn_qualifier": "dev", "pinned": false }

Are there plans to expose this functionality? I can package these changes up into a PR if that's helpful.

Auto activate "Automatically detect and override connection information" in a Group

The default behavior of an instantiated group should support auto-discovery of the Greengrass Core.
A command like this works:

aws greengrass create-function-definition-version \
--function-definition-id 4d941bc7-92a1-4f45-8d64-EXAMPLEf76c3 \
--functions '[{"FunctionArn":"arn:aws:lambda:::function:GGIPDetector:1","Id":"1","FunctionConfiguration":{"Pinned":true,"MemorySize":32768,"Timeout":3}}]' \
--region us-west-2

Essentially, use create-function-defintion-version and add arn:aws:lambda:::function:GGIPDetector:1 just like any other Lambda with the function configuration as shown above.

Mock service role does not work

Originally reported by @neelmitra in Issue #15 as "the mock service role does not work". Specifically:

In addition the mock service role does not work, we still need to create a separate role with AWSGreengrassResourceAccessRolePolicy and associate to the account as below.

aws greengrass associate-service-role-to-account --role-arn arn:aws:iam::xxxxxxxxxxxxx:role/GreengrassRole

Configurable group type

Can the group type be configurable as empty , mock or any name the user chooses ?

In addition the mock service role does not work, we still need to create a seperate role with AWSGreengrassResourceAccessRolePolicy and associate to the account as below.

aws greengrass associate-service-role-to-account --role-arn arn:aws:iam::xxxxxxxxxxxxx:role/GreengrassRole

gg_group_setup create-core parameter --profile_name not work, also --region.

Hi,

Thank you for create this tool, quit handy!
But I have to change ak/sk to the default, otherwise gg_group_setup create-core not work and return error: The security token included in the request is invalid.

2021-01-29 15:04:23,105|botocore.credentials|INFO: Found credentials in shared credentials file: ~/.aws/credentials
Traceback (most recent call last):
  File "/opt/homebrew/bin/gg_group_setup", line 8, in <module>
    sys.exit(main())
  File "/opt/homebrew/lib/python3.9/site-packages/gg_group_setup/cmd.py", line 900, in main
    fire.Fire(gc)
  File "/opt/homebrew/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/homebrew/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/homebrew/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/gg_group_setup/cmd.py", line 711, in create_core
    keys_cert, thing = self.create_thing(thing_name, region, cert_dir)
  File "/opt/homebrew/lib/python3.9/site-packages/gg_group_setup/cmd.py", line 627, in create_thing
    keys_cert = iot_client.create_keys_and_certificate(setAsActive=True)
  File "/opt/homebrew/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the CreateKeysAndCertificate operation: The security token included in the request is invalid.