edgexfoundry / app-service-configurable Goto Github PK

app-service-configurable uses profile command line option to specify the different config(s) to use when running multiple instances of the service. It also appends the profile name to the service key so each running instance has its own unique service key in the registry.

When running app-service-configurable or any other service as a snap service changing the command line options is not possible. Thus the need to add an environment override for specifying the profile.

This environment override should replace the value in the command-line args if profile is specified or add it if it hasn't been specified. There are two command-line flags for specifying the profile which are -p or --profile.

This should be implemented in the SDK and the use the latest SDK to resolve this issue.

The POSIX SIGINT/SIGTERM signals seem to be ignored by app-service-configurable. This prevents systemd from cleanly shutting down the service via SIGTERM when deployed via the snap or a native systemd service unit. Likewise, if a developer runs the service natively on a POSIX compliant system, ctrl-C (which generates SIGINT) will be ignored.

Here's the journal output on an Ubuntu 18.04 system with the latest geneva edgexfoundry snap (from the edge channel) installed:

09:10:01 edgexfoundry.app-service-configurable[8162]: level=INFO ts=2020-05-14T13:10:01.95915984Z app=AppService-rules-engine source=sdk.go:173 msg="Terminating: terminated"
09:10:32 systemd[1]: snap.edgexfoundry.app-service-configurable.service: State 'stop-sigterm' timed out. Killing.
09:10:32 systemd[1]: snap.edgexfoundry.app-service-configurable.service: Killing process 8162 (app-service-con) with signal SIGKILL.
09:10:32 systemd[1]: snap.edgexfoundry.app-service-configurable.service: Main process exited, code=killed, status=9/KILL
09:10:32 systemd[1]: snap.edgexfoundry.app-service-configurable.service: Failed with result 'timeout'.

Steps to reproduce (natively):

• build the service from the latest v1.1.0 tag
• bring up the latest EdgeX via the snap (see below) or docker-compose
• start the service in the foreground
• enter ctrl-C

Steps to reproduce with the edgefoundry snap (which bundles app-service-configurable for builtin rules-engine integration) on an Ubuntu LTS release (eg. 16.04, 18.04, or 20.04 or corresponding Core version):

• in a separate terminal window run: $ sudo journalctl -f
• install the snap: $ sudo snap install edgexfoundry --edge
• enable one of the rules engines: $ sudo snap set edgexfoundry kuiper=on
• disable the rules engine: $ sudo snap set edgexfoundry kuiper=off
• notice that the command takes a long time to complete, and review the log output in the other terminal window

The other related issue is that after the service is finally stopped, Consul continually logs failed health checks for the service. I can open a separate issue for this...

May 14 09:36:58 sumo edgexfoundry.consul[5778]: 2020-05-14T09:36:58.893-0400 [WARN] agent: Check is now critical: check="Health Check: AppService-rules-engine"

Add rules-engine-redis profile that configures the message bus for using Redis Streams

Please update the following files to reference Go 1.13 instead of 1.12

• Dockerfile
• Dockerfile.build
• Jenkinsfile
• go.mod

The new AddTags transform needs to be added to the following profiles so user cam configure them and use them in the pipeline.

• sample
• http-export
• mqtt-export

With the new environment variable overrides capability, we can remove all the docker profiles and apply the differences using environment variables in the docker compose files.

It was decided to hold off one removal of Mongo implementations for Hanoi.
#92

For redisstreams to work in secure mode, the Secret Store configuration is needed so the Redis Password can be retrieved from the Secret Provider.

In other EdgeX services, we can always see the log like:

Service started in: 2.729020742s

However, app-service-configurable doesn't print this log message.

Update to latest SDK that has new ServerBindAddr setting for value used for ListenAndServe.

All profile config files need ServerBindAddr added with a blank value and comment:

[Service]
...
Host = 'localhost'
ServerBindAddr = '' # Leave blank so default to Host value unless different value is needed.
...

Latest SDK has the following V2 bug fixs:

• Metrics Response scheme now has the metrics property as specified in the Swagger
• SecretsRequest requestId is now optional as now defined in the Swagger

Related issue in SDK
edgexfoundry/app-functions-sdk-go#408

Update to use latest SDK with recent fix and update comments in all profiles to state the setting is blank to preserve backward compatibility .

Update to use latest SDK which implements UseTargetTypeOfByteArray and document how to use it.

Update the http-export, mqtt-export and sample profiles to add the new [SecretStoreExclusive] section
Update the http-export, mqtt-export and sample profiles to add the new [InsecureSecrets] section
Update the http-export and sample profiles to add the SecretHeaderName and SecretPath to the parameters for the HTTPPost functions .
Update the mqtt-export profile to add MQTTSecretSend function
Update documentation of the profiles

Each repo needs to have an Attribution file containing the licence information for each dependency including indirect dependencies.

Also need to update the make test target to do a go mod tidy and to validate the Attribution.txt file and fail if missing dependencies. This has been done for edgex-go in PR edgexfoundry/edgex-go#1667 so just duplicate what was done here.

With the new TargetType feature added to the App-Functions-SDK (edgexfoundry/app-functions-sdk-go#79), the app-service-configurable can provide option to set the TargetType to *[]byte.

This allows the configurable functions pipeline to be seeded with any raw data rather than forcing it to be JSON decode Event. Custom JSON decoded type are out of scope for app-service-configurable since the type would have to be know at build time.

App-service should provide mongo database Username and Password even if the security is disabled. It will become mandatory once docker-edgex-mongo issue_77 is fixed. By fixing this issue, mongo will be started with access controll enabled. This will guarantee that each service is able to manipulate the data only of their databases.
If credentials are not provided after issue_77 is fix authorization exceptions will appear.

Configuration files that need to be updated with the default mongo credentials:

• http-export configuration file

• mqtt-export conf file

• sample config file

Add json logic function to configurable pipeline and update sample profile to show how to use it.

NOTE: SDK's configuration.go must first be updated to allows function to be used in the configurable pipeline.

The logging service is be deprecated in Geneva and removed in Hanoi releases.

All profiles need to be updated to not use remote or file logging. This can be done by updating the logging section to:

# Remote and file logging disabled so only stdout logging is used
[Logging]
EnableRemote = false
File = ''

Update to use latest SDK with code cleaned-up for logging service removed.
Config profiles need [Clients.Logging] and [Logging] sections removed.

Add the PR template and Commit guidelines that are used in app-functions-sdk
https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/.github

The following environment variables described in README.md:

• Writable_Pipeline_Functions_HTTPPostJSON_Parameters_Url
• Writable_Pipeline_Functions_HTTPPostJSON_Parameters_PersistOnError
• Writable_Pipeline_Functions_MQTTSend_Parameters
  • _Qos
  • _Key
  • _Cert
  • _Autoreconnect
  • _Retain
  • _PersistOnError

should be:

• Writable_Pipeline_Functions_HTTPPostJSON_Parameters_url
• Writable_Pipeline_Functions_HTTPPostJSON_Parameters_persistOnError
• Writable_Pipeline_Functions_MQTTSend_Parameters
  • _qos
  • _key
  • _cert
  • _autoreconnect
  • _retain
  • _persistOnError

The name of those parameters has to have exactly the same upper/lower case as declared in configuration.toml. It's case-sensitive.
It didn't work even if declared as described in README, I had to change its name as above.

Update to latest SDK with Mongo removed
Update profile config to remove any references to mongo

Per docker documentation:
The main purpose of a CMD is to provide defaults for an executing container
https://docs.docker.com/engine/reference/builder/#cmd
While An ENTRYPOINT allows you to configure a container that will run as an executable
https://docs.docker.com/engine/reference/builder/#entrypoint

Here is an example for Consul which has an entrypoint script and a command
https://github.com/hashicorp/docker-consul/blob/master/0.X/Dockerfile#L75

We have to options:

1. Simply change ENTRYPOINT to CMD
   Overrides of CMD in compose file have to include the executable.
2. Leave the executable in ENTRYPOINT and move command-line options to CMD
   Overrides of CMD in compose file are then just the command-line options.

Now that we have a MQTT implementation for the Message Bus, we need a sample profile that has the Message Bus configured to use MQTT rather than ZMQ. The profile name should be named mqtt-message-bus and be a copy of the http-export with MessageBus section changed. README needs to be updated to add description of this new profile and make the profile list its own section in TOC.

the current go.mod still uses the old version of app-function-sdk-go. Needs to use the latest one.

Update ASC w/ app-functions-sdk-go#359 - Filter function to reverse the logic to filter out specific names

The ASC testcases have been failed on Jenkins since Apr 23.

I tested app-service-configurable in blackbox-testing docker-compose-test-tools.yml with docker-compose-nexus-redis-no-secty.yml locally, and AppService-blackbox-tests didn't show on Consul ( no /v1/kv/edgex/appservices/1.0/AppService-blackbox-tests).

The following is the app-service-configurable(AppService-blackbox-tests) in docker-compose-test-tools.yml:

  app-service-configurable:
    image: ${appService}
    ports:
      - "48095:48095"
    entrypoint: ["/app-service-configurable"]
    command: ["--registry","--confdir=/res", "--skipVersionCheck=true"]
    container_name: edgex-app-service-configurable
    hostname: app-service-configurable
    environment:
      edgex_registry: consul://edgex-core-consul:8500
      edgex_service: http://edgex-app-service-configurable:48095
      edgex_profile: blackbox-tests
      Service_Host: edgex-app-service-configurable
      Clients_CoreData_Host: edgex-core-data
      Clients_Logging_Host: edgex-support-logging
      Logging_EnableRemote: "true"
      Database_Host : edgex-mongo
      Database_Username : appservice
      Database_Password : password
    networks:
      edgex-network:
        aliases:
          - edgex-app-service-configurable

We are testing V2 APIs, but they are not available in the latest image.
We found out the reason is that app-func-sdk-go stopped tagging after July 31st, and app-service-configurable is still using
https://github.com/edgexfoundry/app-functions-sdk-go/tree/v1.2.1-dev.17

All the new changes in August are not included into the current app-service-configurable image.

The current snap install hook is forced to use the environment variable SNAP_INSTANCE_NAME when it disables the service on install. At the moment this is required as the snapctl command doesn't have the capability to introspect the instance name for service commands. This is slated to be fixed soon, so this bug is a placeholder to revise this install hook and bump the minimum required version of snapd once fixed.

The usage of SNAP_INSTANCE_NAME was introduced by the following PR:

#26

For more details please refer to the following forum post:

https://forum.snapcraft.io/t/run-configure-hook-of-nextcloud-1-snap-run-hook-configure-error-error-running-snapctl-unknown-service-nextcloud-apache/11422/7

The environment variables are shown in the README are shown with "Parameters_Addressable" in their names, when they should all either be just "Parameters" or "Addressable" to mirror the hierarchy in the .toml files.

Update to latest SDK implements Vault Integration for DB credentials. The [SecretStore] config section needs to be added to the default and export profile configurations.

This issue is dependent on PR #edgexfoundry/app-functions-sdk-go#244 being merged.

The port for app-service-configurable-rules in nightly-build compose files is 48100 (for example, docker-compose-nexus-mongo-no-secty.yml ) but it shows 50001 on Consul.

Besides, after I started up the edgex-app-service-configurable in blackbox-testing (using docker-compose-test-tools.yml), the AppService was been overwritten.

The log of app-service-configurable-rules and edgex-app-service-configurable:

Starting Configurable Application Service...
level=INFO ts=2020-04-24T11:35:43.4336862Z app=AppService source=environment.go:327 msg="Environment override of '-p/-profile' by environment variable: edgex_profile=rules-engine"
level=INFO ts=2020-04-24T11:35:43.4402771Z app=AppService source=config.go:219 msg="Loaded configuration from /res/rules-engine/configuration.toml"
level=INFO ts=2020-04-24T11:35:43.4533681Z app=AppService source=environment.go:327 msg="Environment override of 'MessageBus.SubscribeHost.Host' by environment variable: MessageBus_SubscribeHost_Host=edgex-core-data"
level=INFO ts=2020-04-24T11:35:43.4534531Z app=AppService source=environment.go:327 msg="Environment override of 'Binding.PublishTopic' by environment variable: Binding_PublishTopic=events"
level=INFO ts=2020-04-24T11:35:43.4535003Z app=AppService source=environment.go:327 msg="Environment override of 'Service.Host' by environment variable: Service_Host=edgex-app-service-configurable-rules"
level=INFO ts=2020-04-24T11:35:43.4535295Z app=AppService source=environment.go:327 msg="Environment override of 'Registry.Host' by environment variable: Registry_Host=edgex-core-consul"
level=INFO ts=2020-04-24T11:35:43.4559668Z app=AppService source=environment.go:327 msg="Environment override of 'Clients.Logging.Host' by environment variable: Clients_Logging_Host=edgex-support-logging"
level=INFO ts=2020-04-24T11:35:43.4560449Z app=AppService source=environment.go:327 msg="Environment override of 'Clients.CoreData.Host' by environment variable: Clients_CoreData_Host=edgex-core-data"
level=INFO ts=2020-04-24T11:35:43.4580591Z app=AppService source=config.go:112 msg="Config Provider URL created from Registry configuration"
level=INFO ts=2020-04-24T11:35:43.4586326Z app=AppService source=environment.go:327 msg="Environment override of 'Configuration Provider Information' by environment variable: edgex_registry=consul://edgex-core-consul:8500"
level=INFO ts=2020-04-24T11:35:43.4588498Z app=AppService source=config.go:167 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/appservices/1.0/AppService"
level=INFO ts=2020-04-24T11:35:43.4993621Z app=AppService source=environment.go:327 msg="Environment override of 'MessageBus.SubscribeHost.Host' by environment variable: MessageBus_SubscribeHost_Host=edgex-core-data"
level=INFO ts=2020-04-24T11:35:43.4994368Z app=AppService source=environment.go:327 msg="Environment override of 'Binding.PublishTopic' by environment variable: Binding_PublishTopic=events"
level=INFO ts=2020-04-24T11:35:43.4994644Z app=AppService source=environment.go:327 msg="Environment override of 'Registry.Host' by environment variable: Registry_Host=edgex-core-consul"
level=INFO ts=2020-04-24T11:35:43.49949Z app=AppService source=environment.go:327 msg="Environment override of 'Clients.Logging.Host' by environment variable: Clients_Logging_Host=edgex-support-logging"
level=INFO ts=2020-04-24T11:35:43.4998126Z app=AppService source=environment.go:327 msg="Environment override of 'Service.Host' by environment variable: Service_Host=edgex-app-service-configurable-rules"
level=INFO ts=2020-04-24T11:35:43.5001591Z app=AppService source=environment.go:327 msg="Environment override of 'Clients.CoreData.Host' by environment variable: Clients_CoreData_Host=edgex-core-data"
level=INFO ts=2020-04-24T11:35:43.5012474Z app=AppService source=config.go:321 msg="Configuration has been pulled from Configuration provider (6 environment overrides applied)"
level=ERROR ts=2020-04-24T11:35:43.5016497Z app=AppService source=logger.go:74 msg="logTarget cannot be blank, using stdout only"
level=INFO ts=2020-04-24T11:35:43.5025424Z app=AppService source=environment.go:327 msg="Environment override of 'Registry Provider Information' by environment variable: edgex_registry=consul://edgex-core-consul:8500"
level=INFO ts=2020-04-24T11:35:43.5026189Z app=AppService source=registry.go:79 msg="Using Registry (consul) from http://edgex-core-consul:8500"
level=INFO ts=2020-04-24T11:35:43.5197263Z app=AppService source=config.go:308 msg="Writeable configuration has been updated from the Configuration Provider"
level=INFO ts=2020-04-24T11:35:43.607927Z app=AppService source=version.go:66 msg="Skipping core service version compatibility check"
level=INFO ts=2020-04-24T11:35:43.6080151Z app=AppService source=server.go:280 msg="Registering standard routes..."
level=INFO ts=2020-04-24T11:35:43.6084648Z app=AppService source=main.go:43 msg="Loading Configurable Pipeline..."
level=INFO ts=2020-04-24T11:35:43.6085554Z app=AppService source=sdk.go:411 msg="MessageBus trigger selected"
level=INFO ts=2020-04-24T11:35:43.6086268Z app=AppService source=messaging.go:47 msg="Initializing Message Bus Trigger. Subscribing to topic: events on port 5563 , Publish Topic: events on port 5566"
level=INFO ts=2020-04-24T11:35:43.6091195Z app=AppService source=sdk.go:147 msg="StoreAndForward disabled. Not running retry loop."
level=INFO ts=2020-04-24T11:35:43.60916Z app=AppService source=sdk.go:150 msg="AppServiceConfigurable-rules-engine has Started"
level=INFO ts=2020-04-24T11:35:43.60921Z app=AppService source=server.go:350 msg="Starting HTTP Web Server on port :50001"

• Update the Dockerfile to use golang:1.15-alpine base image
• Update the Dockerfile.build to use golang:1.15-alpine base image
• Update go.mod to use 1.15
• Update Jenkinsfile to specify 1.15
• Update snap/snapcraft.yaml to pull 1.15.1 tar files

Running edgex-app-service-configurable-rules with nightly-build compose files, I got the error in the log:

Starting Configurable Application Service...
level=INFO ts=2020-07-24T05:34:39.660147Z app=AppService-rules-engine source=environment.go:331 msg="Environment override of '-p/-profile' by environment variable: EDGEX_PROFILE=rules-engine"
level=INFO ts=2020-07-24T05:34:39.6640869Z app=AppService-rules-engine source=config.go:219 msg="Loaded configuration from /res/rules-engine/configuration.toml"
level=INFO ts=2020-07-24T05:34:39.6708673Z app=AppService-rules-engine source=environment.go:331 msg="Environment override of 'Registry.Host' by environment variable: REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2020-07-24T05:34:39.6709418Z app=AppService-rules-engine source=environment.go:331 msg="Environment override of 'Binding.PublishTopic' by environment variable: BINDING_PUBLISHTOPIC=events"
level=INFO ts=2020-07-24T05:34:39.6709804Z app=AppService-rules-engine source=environment.go:331 msg="Environment override of 'Clients.CoreData.Host' by environment variable: CLIENTS_COREDATA_HOST=edgex-core-data"
level=INFO ts=2020-07-24T05:34:39.6710474Z app=AppService-rules-engine source=environment.go:331 msg="Environment override of 'Service.Host' by environment variable: SERVICE_HOST=edgex-app-service-configurable-rules"
level=INFO ts=2020-07-24T05:34:39.67119Z app=AppService-rules-engine source=environment.go:331 msg="Environment override of 'MessageBus.SubscribeHost.Host' by environment variable: MESSAGEBUS_SUBSCRIBEHOST_HOST=edgex-core-data"
level=INFO ts=2020-07-24T05:34:39.6712267Z app=AppService-rules-engine source=environment.go:331 msg="Environment override of 'Service.Port' by environment variable: SERVICE_PORT=48100"
level=INFO ts=2020-07-24T05:34:39.6715048Z app=AppService-rules-engine source=config.go:167 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/appservices/1.0/AppService-rules-engine"
level=INFO ts=2020-07-24T05:34:44.7587653Z app=AppService-rules-engine source=config.go:330 msg="Configuration has been pushed to into Configuration Provider (6 environment overrides applied)"
level=ERROR ts=2020-07-24T05:34:44.7593739Z app=AppService-rules-engine source=logger.go:74 msg="logTarget cannot be blank, using stdout only"
level=INFO ts=2020-07-24T05:34:44.7596967Z app=AppService-rules-engine source=registry.go:79 msg="Using Registry (consul) from http://edgex-core-consul:8500"
level=INFO ts=2020-07-24T05:34:44.7894689Z app=AppService-rules-engine source=telemetry.go:79 msg="Starting CPU Usage Average loop"
level=ERROR ts=2020-07-24T05:34:44.7928798Z app=AppService-rules-engine source=version.go:128 version=1.2.1-dev.1 msg="Core Services version is malformed"
level=ERROR ts=2020-07-24T05:34:44.7930143Z app=AppService-<profile> source=main.go:34 msg="SDK initialization failed: boostrapping failed\n"

The same errors happened when running app-service-configurable with blackbox-testing docker-compose-test-tools.yml, and that causes the blackbox-testing failures on Jenkins currently.

Since the ClouldEvent will have been add to the SDK for the conversion function it can now be used as the TargeType in App Service Configurable.

Added a configuration setting to specify ClouldEvent as TargetType.

On-hold unit Cloud Event package added via Cloud conversion functions.

Now that the MQTT & HTTP export functions no longer call MarkAsPushed by default and this functionality has been capsuled in a function the export profiles need to be update to add MarkAsPushed function to the configured pipelines.

It would be great to have this as a snap. I can work on this when the service is considered feature-complete and has jobs setup in Jenkins

Make any tweaks needed to work with release version of SDK:

• Service key changes to AppService-
• Tweaks to mqtt-export profile for secrets configuration

Profiles that use Database still specify mongo as default. Need to change to use Redis
Dockerfile still uses old command line flags, etc. Need to update to add use of -cp flag for the config provider.

Since ASC is profile base, there shouldn't be a default configuration. To force users to select a profile, the default configuration needs to be moved to the sample profile. The provided profiles need to be documented in the README.

Per https://docs.edgexfoundry.org/1.2/microservices/application/AppServiceConfigurable/#mqttsend, MQTTSend has been deprecated and the community suggests to use MQTTSecretSend. However, the sample configuration under res/mqtt-export still uses MQTTSend. The community shall consider to update the sample configuration to use MQTTSecretSend.

Once PR for edgexfoundry/app-functions-sdk-go#385 is merged. ASC needs to be updated to use the latest SDK and remove all instances of Service.ClientMonitor setting from all profiles

With the use of go-mod-bootstrap so profiles need addition config elements.

This might be more of edge case, but running with a profile will correctly populate consule (i.e. docker-rules-engine) but if you remove the profile, then it will continue to use the same key despite the name changing.

Steps to Reproduce:

1. With consul up and running, Run app service configurable in docker specifying a profile: --profile=docker-rules-engine (i.e using docker run or docker-compose up)
2. Observe the correct key/values in consul under the correct key of AppService-docker-rules-engine
3. Rerun the command without a profile.
4. Observe that no new entry is in Consul and the old key is still being used and observed for changes for the new container.

The permissions setup by Vault Worker do not allow secrets to be created at the root level for the specified client. To allow user to store secretes w/o specify a sub path (as is documented) the [SecretStoreExclusive] configuration needs to have a sub-path specified on the Path property in all the profiles that have [SecretStoreExclusive] specified.

Path = '/v1/secret/edgex/appservice-http-export/'

changes to:

Path = '/v1/secret/edgex/appservice-http-export/app'

The shared [SecretStore] client doesn't need this since it is read only and has the DB secrets stored a the redisdb sub-path.

The latest SDK also is required as it has a fix from go-mod-secrets to ensure there is no trailing slash on the resulting Vault URL.

Batch is a new transform available in the SDK, need to update the sample configuration.toml to show how to configure it.

SDK's configuration.go must first be updated to allows function to be used in the configurable pipeline.

Not sure how we missed this, but LogLevel is in the TOML for docker configuration.toml twice.

SDK v1.2.0 has been released with bug fixes and enhancements
Logging client is never nil after return from call to Initialize(), so error handling code can be simplified as was done in all the examples.

The snap description has the wrong link, it has:

when it should just be https://github.com/edgexfoundry/app-functions-sdk-go