suneetnangia / kedge Goto Github PK

Container needs privileged access to host to be able to run edge and Moby container engine. We should make every attempt to avoid this elevated access.

We are observing intermittent exception which results in restart of edgeAgent:

█████╗ ███████╗██╗ ██╗██████╗ ███████╗
██╔══██╗╚══███╔╝██║ ██║██╔══██╗██╔════╝
███████║ ███╔╝ ██║ ██║██████╔╝█████╗
██╔══██║ ███╔╝ ██║ ██║██╔══██╗██╔══╝
██║ ██║███████╗╚██████╔╝██║ ██║███████╗
╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝

██╗ ██████╗ ████████╗ ███████╗██████╗ ██████╗ ███████╗
██║██╔═══██╗╚══██╔══╝ ██╔════╝██╔══██╗██╔════╝ ██╔════╝
██║██║ ██║ ██║ █████╗ ██║ ██║██║ ███╗█████╗
██║██║ ██║ ██║ ██╔══╝ ██║ ██║██║ ██║██╔══╝
██║╚██████╔╝ ██║ ███████╗██████╔╝╚██████╔╝███████╗
╚═╝ ╚═════╝ ╚═╝ ╚══════╝╚═════╝ ╚═════╝ ╚══════╝

<6> 2021-07-08 13:36:53.174 +00:00 [INF] - Experimental features configuration: {"Enabled":false,"DisableCloudSubscriptions":false}
<6> 2021-07-08 13:36:53.366 +00:00 [INF] - Installing certificates [CN=iotedged workload ca buildkitsandbox:10/06/2021 12:58:26] to Root
<6> 2021-07-08 13:36:53.578 +00:00 [INF] - Starting metrics listener on Host: *, Port: 9600, Suffix: /metrics
<6> 2021-07-08 13:36:53.841 +00:00 [INF] - Updating performance metrics every 05m:00s
<6> 2021-07-08 13:36:53.849 +00:00 [INF] - Started operation Get system resources
<6> 2021-07-08 13:36:53.851 +00:00 [INF] - Collecting metadata metrics
<6> 2021-07-08 13:36:53.974 +00:00 [INF] - Set metadata metrics: 1.1.4.44136317 (cbeb53cf8b5889718c4724936f6d13ef5ceaac2e), {"Enabled":false,"DisableCloudSubscriptions":false}, {"OperatingSystemType":"linux","Architecture":"x86_64","Version":"1.2.3","Provisioning":{"Type":"ProvisioningType","DynamicReprovisioning":false,"AlwaysReprovisionOnStartup":false},"ServerVersion":"20.10.7+azure","KernelVersion":"5.4.0-1051-azure","OperatingSystem":"Ubuntu 18.04.5 LTS (containerized)","NumCpus":1,"Virtualized":"yes"}, True
<6> 2021-07-08 13:36:54.005 +00:00 [INF] - Started operation Checkpoint Availability
<6> 2021-07-08 13:36:54.021 +00:00 [INF] - Started operation refresh twin config
<6> 2021-07-08 13:36:54.048 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
<6> 2021-07-08 13:36:55.031 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
<6> 2021-07-08 13:36:55.197 +00:00 [INF] - Started operation Metrics Scrape
<6> 2021-07-08 13:36:55.197 +00:00 [INF] - Started operation Metrics Upload
Scraping frequency: 01:00:00
Upload Frequency: 1.00:00:00
<6> 2021-07-08 13:36:56.720 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
<6> 2021-07-08 13:36:56.896 +00:00 [INF] - Registering request handler UploadModuleLogs
<6> 2021-07-08 13:36:56.896 +00:00 [INF] - Registering request handler GetModuleLogs
<6> 2021-07-08 13:36:56.896 +00:00 [INF] - Registering request handler UploadSupportBundle
<6> 2021-07-08 13:36:56.904 +00:00 [INF] - Registering request handler RestartModule
<6> 2021-07-08 13:36:57.013 +00:00 [INF] - Initialized new module client with subscriptions enabled
<6> 2021-07-08 13:36:57.152 +00:00 [INF] - Obtained Edge agent twin from IoTHub with desired properties version 5 and reported properties version 19014.
<6> 2021-07-08 13:36:58.208 +00:00 [INF] - Plan execution started for deployment 5
<6> 2021-07-08 13:36:58.240 +00:00 [INF] - Executing command: "Command Group: (\n [Create module edgeHub]\n [Start module edgeHub]\n)"
<6> 2021-07-08 13:36:58.248 +00:00 [INF] - Executing command: "Create module edgeHub"
<6> 2021-07-08 13:36:58.546 +00:00 [INF] - Executing command: "Start module edgeHub"
<6> 2021-07-08 13:36:59.344 +00:00 [INF] - Executing command: "Command Group: (\n [Create module SimulatedTemperatureSensor]\n [Start module SimulatedTemperatureSensor]\n)"
<6> 2021-07-08 13:36:59.351 +00:00 [INF] - Executing command: "Create module SimulatedTemperatureSensor"
<6> 2021-07-08 13:36:59.737 +00:00 [INF] - Executing command: "Start module SimulatedTemperatureSensor"
<6> 2021-07-08 13:37:01.115 +00:00 [INF] - Plan execution ended for deployment 5
<6> 2021-07-08 13:37:01.733 +00:00 [INF] - Updated reported properties
<6> 2021-07-08 13:37:06.953 +00:00 [INF] - Updated reported properties
<6> 2021-07-08 13:37:32.417 +00:00 [INF] - Termination requested, initiating shutdown.
<6> 2021-07-08 13:37:32.420 +00:00 [INF] - Waiting for cleanup to finish
<4> 2021-07-08 13:37:32.434 +00:00 [WRN] - Reconcile failed because of the an exception
System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.CancellationToken.ThrowIfCancellationRequested()
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.IO.BufferedStream.ReadFromUnderlyingStreamAsync(Memory1 buffer, CancellationToken cancellationToken, Int32 bytesAlreadySatisfied, Task semaphoreLockTask) at Microsoft.Azure.Devices.Edge.Util.Uds.HttpBufferedStream.ReadLineAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpBufferedStream.cs:line 62 at Microsoft.Azure.Devices.Edge.Util.Uds.HttpRequestResponseSerializer.SetResponseStatusLine(HttpResponseMessage httpResponse, HttpBufferedStream bufferedStream, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpRequestResponseSerializer.cs:line 131 at Microsoft.Azure.Devices.Edge.Util.Uds.HttpRequestResponseSerializer.DeserializeResponse(HttpBufferedStream bufferedStream, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpRequestResponseSerializer.cs:line 66 at Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/uds/HttpUdsMessageHandler.cs:line 36 at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.GeneratedCode.EdgeletHttpClient.ListModulesAsync(String api_version, CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/generatedCode/EdgeletHttpClient.cs:line 85
at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter[T](Task1 task, TimeSpan timeout) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 126 at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 135
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.HandleException(Exception exception, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 248
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Versioning.ModuleManagementHttpClientVersioned.Execute[T](Func`1 func, String operation) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/versioning/ModuleManagementHttpClientVersioned.cs:line 153
at Microsoft.Azure.Devices.Edge.Agent.Edgelet.Version_2020_07_07.ModuleManagementHttpClient.GetModules[T](CancellationToken cancellationToken) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Edgelet/version_2020_07_07/ModuleManagementHttpClient.cs:line 166
at Microsoft.Azure.Devices.Edge.Agent.Docker.DockerEnvironment.GetModulesAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Docker/DockerEnvironment.cs:line 51
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.GetCurrentModuleSetAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 248
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 128
<4> 2021-07-08 13:37:32.483 +00:00 [WRN] - Agent reconcile concluded with errors.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Microsoft.Azure.Devices.Edge.Agent.IoTHub.Reporters.IoTHubReporter.ReportAsync(CancellationToken token, ModuleSet moduleSet, IRuntimeInfo runtimeInfo, Int64 version, DeploymentStatus updatedStatus) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.IoTHub/reporters/IoTHubReporter.cs:line 44
at Microsoft.Azure.Devices.Edge.Agent.Core.Agent.ReconcileAsync(CancellationToken token) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Core/Agent.cs:line 205
at Microsoft.Azure.Devices.Edge.Util.TaskEx.TimeoutAfter(Task task, TimeSpan timeout) in /home/vsts/work/1/s/edge-util/src/Microsoft.Azure.Devices.Edge.Util/TaskEx.cs:line 143
at Microsoft.Azure.Devices.Edge.Agent.Service.Program.MainAsync(IConfiguration configuration) in /home/vsts/work/1/s/edge-agent/src/Microsoft.Azure.Devices.Edge.Agent.Service/Program.cs:line 348
<6> 2021-07-08 13:37:32.483 +00:00 [INF] - Main thread terminated
<6> 2021-07-08 13:37:32.484 +00:00 [INF] - Initiating shutdown cleanup.
<6> 2021-07-08 13:37:32.505 +00:00 [INF] - Stopping all modules...
<4> 2021-07-08 13:37:32.622 +00:00 [WRN] - Updating reported properties failed with error error(condition:amqp:unauthorized-access,description:Put token failed. status-code: 401, status-description: The specified SAS token has an invalid signature. It does not match either the primary or secondary key..) type Microsoft.Azure.Devices.Client.Exceptions.UnauthorizedException

Currently container image is baked with IoT Edge connection string which is not ideal as it can be leaked and less flexible when it comes to deploying container in multiple environments dynamically.