elastic / azure-vm-extension

Elastic Agent Azure VM Extension

Elastic Agent VM extensions

ElasticAgent.windows for Windows systems ElasticAgent.linux for Linux systems

The ElasticAgent VM extensions are small applications that provide post-deployment configuration and automation on Azure VMs. Once installed, the ElasticAgent VM extension will download the Elastic Agent artifacts, install the Elastic Agent on the virtual machine, enroll it to Fleet and then start the agent service.

Platforms supported

Platform	Version
Windows	2008r2 +
Centos	6.10+
Debian	9,10
Oracle	6.8+
RHEL	7+
Ubuntu	16+

Elastic Cloud dependency

To automate the installation and configuration of the Elastic Agent, the Azure VM extension code makes several API calls which requires specific Elastic stack version.

VM extension version	Elastic Cloud dependency
1.3.0.0	7.13.0 or later
1.2.0.0	7.13.0 or later
1.1.1.0	7.13.0 or later
1.1.0.0	7.13.0 or later
1.0.0.0	7.13.0 or later

Configuration

For a successful installation the following configuration settings are required:

Public settings:

• username - a valid username that can have access to the elastic cloud cluster
• cloudId - the elastic cloud ID (deployment ID)

Protected settings:

• password - a valid password that can be used in combination with the username public setting to access the elastic cloud cluster

Managing the Elastic Agent VM extensions

The Elastic Agent VM extensions can be managed using the Azure CLI, PowerShell, Resource Manager templates, and in the future the Azure portal.

For Windows Azure VM's users will need to install the ElasticAgent.windows VM extension.

Example installation from CLI:

 az vm extension set -n ElasticAgent.windows --publisher Elastic --version {version number} --vm-name "{resource name}" --resource-group "{resource group name}" --protected-settings '{\"password\":\"{elastic password}\"}' --settings '{\"username\":\"{elastic username}\",\"cloudId\":\"{elastic cloud ID}\"}'

For Linux based VM's users will need to install the ElasticAgent.linux VM extension.

Example installation from CLI:

 az vm extension set -n ElasticAgent.linux --publisher Elastic --version {version number} --vm-name "{resource name}" --resource-group "{resource group name}" --protected-settings '{\"password\":\"{elastic password}\"}' --settings '{\"username\":\"{elastic username}\",\"cloudId\":\"{elastic cloud ID}\"}'

azure-vm-extension's Issues

Replace deprecated api/fleet/enrollment-api-keys to api/fleet/enrollment_api_keys

API enrollment-api-keys will be deprecated and change to enrollment_api_keys in 8.0 (https://petstore.swagger.io/?url=https://raw.githubusercontent.com/elastic/kibana/main/x-pack/plugins/fleet/common/openapi/bundled.json#/default/get-enrollment-api-keys). We should also replace the old api/fleet/enrollment-api-keys in azure vm extension.

how to debug/diagnose when no metrics being received by ESS stack

Using 7.17.2 on ESS via our instructions

VM Extension added to a Windows VM but No metrics being received by stack

How is it possible to debug/diagnose/fix to get metrics flowing?

Improve uninstall/unenroll

For unenroll we need to retrieve the agent id, this is curently done through reading the fleet.yml file, this is error prone, would like to have an additional flag for inspect command where we retrieve specific information, there could be plans in the future for the uninstall command to first unenroll so this might not be necessary

Check package integrity before installing

When installing the agent in Install_ElasticAgent_OTHER we don't actually run the check on the SHASUM for the .tar.gz we download.

Since the sum file is downloaded and since there's some error handling already in place, the actual check should be executed.

Expand on documentation with more examples

Add linux example , make clear that there are 2 versions of the vm extension

Azure VM extension UI in the Azure Portal

The ElasticAgent vm extension is still not visible in the Azure Portal.
Initial work here https://github.com/elastic/azure-vm-extension/tree/master/src/packages

Add support for several OS's

Based on:
https://docs.microsoft.com/en-US/troubleshoot/azure/virtual-machines/extension-supported-os
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/endorsed-distros

Also support matrix for the elastic-agent https://www.elastic.co/support/matrix#elastic-agent

Testing contains the steps:

• install vm ext on machine
• update configuration to different cloud env
• test on at least 7.10, 7.12
• re/uninstall

Windows:

• Windows Server 2012
• Windows Server 2012R2
• Windows Server 2016 (Gen 1+2)
• Windows Server 2019 (Gen 1+2)
• Windows 10

RPM:

Red Hat Enterprise Linux by Red Hat

• [N] 6.7+ ( encountering https://serverfault.com/questions/606135/curl-35-ssl-connect-error, would require an update of the nss package which will could have substantial effect on the vm)
• 7.2
• 7.3
• 8

CentOS by Rogue Wave Software

• 6.10
• 7.4+
• 8.3

Oracle Linux by Oracle

• 6.x
• 7.x
• 8.x

Debian:

Debian by Credativ

• 8.x (could not find version)
• 9.x (Credativ missing used Automized INC instead)
• 10.x (Credativ missing used Automized INC instead)

Ubuntu by Canonical: Ubuntu Server and Pro.

• 16.x
• 18.x
• 20.x

Not supported:

[N] Flatcar Container Linux by Kinvolk
[N] SUSE Linux Enterprise by SUSE
[N] openSUSE

Add support for 7.13 Fleet Server

Few things have changed in the installing/enrolling/unenrolling steps:

1. Only this call is necessary now KIBANA_URL/api/fleet/setup (the other one still exists but it's doing exactly the same)
2. Nothing changed for enrollment API keys
3. For the enrollment Elastic Agent do not communicate with Kibana anymore so --kibana-url=$kibanaUrl should be replaced by --url=$fleetServerUrl
4. The unenroll API is still available, there was a breaking change introduced (the force parameter is now call revoke)
5. You can get the Fleet Server hosts by call the settings API in fleet, (note the settings should be set before in Kibana)
   GET /api/fleet/settings

Migrage to Buildkite

We need to move from Jenkins to Buildkite by the end of December 2023.

Problem with the Azure VM Extension for Windows VMs using a private link

A user reported an install failure with the following logs:

2022/11/01 11:04:38:141 INFO Install command has been executed. Elastic Agent will be installed on enable
...
2022/11/01 12:47:06:883 INFO The latest configuration file is C:\Packages\Plugins\Elastic.ElasticAgent.windows\1.2.0.0\RuntimeSettings\2.settings
2022/11/01 12:47:06:899 ERROR Elastic Agent installation failed after 3 retries
2022/11/01 12:47:06:914 ERROR Cannot bind parameter 'Uri'. Cannot convert value "https:// 4 a b e c 5 1 e 6 0 3 c 1 1 e d 9 b 6 a 0 2 4 2 a c 1 2 0 0 0 2 .p r i v a t e l i n k . n o r t h e u r o p e . a z u r e . e l a s t i c - c l o u d . c o m " to type "System.Uri". Error: "Invalid URI: The hostname could not be parsed."
2022/11/01 12:47:06:914 ERROR at Get-Stack-Version, C:\Packages\Plugins\Elastic.ElasticAgent.windows\1.2.0.0\scripts\helper.ps1: line 103
at Install-ElasticAgent, C:\Packages\Plugins\Elastic.ElasticAgent.windows\1.2.0.0\scripts\enable.ps1: line 34
at <ScriptBlock>, C:\Packages\Plugins\Elastic.ElasticAgent.windows\1.2.0.0\scripts\enable.ps1: line 272
at <ScriptBlock>, <No file>: line 1
2022/11/01 12:47:06:930 INFO The latest configuration file is C:\Packages\Plugins\Elastic.ElasticAgent.windows\1.2.0.0\RuntimeSettings\2.settings

Add documentation

Document how the extension can be used:

What VM Extensions for Elastic does
How to enable/disable VM Extensions
How to manage agent once VM Extension is enabled
OS compatibility matrix

Add dedicated policy for agents started through the vm extension

We had some issues with the renaming of Default policies that we can avoid by creating a dedicated policy inside the install process of the vm extension and enrolling the agent to it.

• investigate if this could be done with the current kibana fleet apis
• could be named azure-vm-extension-policy and contain the system integration.
• a check will be done if the policy exists first and then create if not

Flaky test: OS_VERSION = 'windows-2016' / Validate / test_indice_ds_logs_windows_diskio – validate.TestIndices

Test failed: ITs / Matrix - OS_VERSION = 'windows-2016' / Validate / test_indice_ds_logs_windows_diskio – validate.TestIndices
Error detail:

[](https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fazure-vm-extension-mbp%2FPR-59/detail/PR-59/1//pipeline)[](https://github.com/elasticmachine)[](https://github.com/elasticmachine)[](https://github.com/apps/cla-checker-service)[](https://github.com/elasticmachine)[](https://github.com/elasticmachine)[](https://github.com/elasticmachine)[](https://github.com/elasticmachine)[](https://github.com/elasticmachine)[](https://github.com/elasticmachine)[](https://github.com/elasticmachine)[](https://github.com/elasticmachine)[](https://github.com/kaiyan-sheng)[](https://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax)[](https://github.com/kaiyan-sheng)[](https://github.com/kaiyan-sheng)[](https://github.com/elasticmachine)[](https://github.com/)False is not true : Expected at least one entry in index .ds-logs-system.application-default-*, got 0

Stacktrace:

 Traceback (most recent call last):
  File "/app/test/ats/validate.py", line 139, in test_indice_ds_logs_windows_diskio
    self.count_and_test('.ds-logs-system.application-default-*', self.hostname, 1)
  File "/app/test/ats/validate.py", line 111, in count_and_test
    self.assertTrue(records_count >= compare_with, "Expected at least one entry in index {}, got {}".format(index_name, records_count))
AssertionError: False is not true : Expected at least one entry in index .ds-logs-system.application-default-*, got 0

Link: https://beats-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/Beats/pipelines/azure-vm-extension-mbp/pipelines/PR-59/runs/1/steps/923/log/?start=0

Decouple start and enroll on windows

On Windows elastic-agent install cmd installs the agent, enrolls it and starts it. Nice to have a install -no start command which only installs the elastic agent as a service and we can separate the enroll and start steps

Automate the publishing process

Implement continuous delivery for the vm extension

Add integration tests

Figure out a way to provide some degree of integration testing to the project