izmaxxsun / elastic-stack-deployment

Installing Elasticsearch, Kibana, Fleet Server and APM without orchestration

Deploying Elastic Stack on Windows

Installing Elasticsearch, Kibana, Fleet Server and APM without orchestration. The other ways to deploy are using Elastic Cloud (managed service), or ECE/ECK which provide orchestration for easier install, upgrades, and maintenance.

Elastic Stack version 8.4.

Create Virtual Machine

Created on Azure with Windows Server 2022 Datacenter Azure Edition Standard B4ms, 4vCPU, 16GiB RAM

Gameplan

Following along with Installing the Elastic Stack, we need to install the stack in order and we'll focus on the minimum needed here:

• Elasticsearch
• Kibana
• APM

Installing Elasticsearch

1 - Following the Windows install instructions, download the .zip file and extract it

2 - Run Elasticsearch from the command line

.\bin\elasticsearch.bat

3 - Copy and keep the password and enrollment token

4 - Check it is running curl --cacert http_ca.crt https://elastic:IVMTu3b5RF+-E3F7kOxY@localhost:9200 --ssl-no-revoke

Installing Kibana

5 - Following the Kibana install instructions, download the .zip file and extract it

6 - Run Kibana from the command line

7 - Open link to Kibana in browser

8 - Paste enrollment token and click button to connect Kibana to Elasticsearch

9 - Log in to Kiban with "elastic" user and password noted in Step 3

Configure Elasticsearch and Kibana for APM

Complete prerequisites from APM quick start

10 - navigate to $ES_HOME\config and open elasticsearch.yml configuration file

11 - add the following line

xpack.security.authc.api_key.enabled: true

12 - generate Kibana encryption keys

$KIBANA_HOME\bin\kibana-encryption-keys generate

12 - navigate to $KIBANA_HOME\config and open kibana.yml configuration file

13 - Copy and paste Kibana encryption keys into kibana.yml

xpack.encryptedSavedObjects.encryptionKey: fc2caad44285034fa89b7aadcaac750b
xpack.reporting.encryptionKey: 68873dd130e6eaef07bb28c5a23b720b
xpack.security.encryptionKey: 5fc4dfde27048ea90e4cb1ce27aa1786

Set up Fleet

14 - Go to Management > Fleet > Settings

15 - Click Edit hosts and add Fleet Server URL (use IP that is accessible by hosts we plan to monitor, so in this case use Azure VM public IP)

16 - Click Agents tab and follow guide to add Fleet Server

.\elastic-agent.exe install `
  --fleet-server-es=https://172.27.0.5:9200 `
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NjM4OTk3NzA2MTM6VUJNYXZtMHpSSy1BSk56djNxeFYxZw `
  --fleet-server-policy=fleet-server-policy `
  --fleet-server-es-ca=C:\elasticsearch-8.4.2\config\certs\http_ca.crt

Dealing with Certificates

Keep in mind that Elastic Cloud takes care of all this for you. We're just doing this the hard way.

When you start Elasticsearch it creates the following certificates and keys (see Configuring Stack Security)

• http_ca.crt - used to sign certificates for HTTP layer of ES cluster
• keystore with key and cert for HTTP layer for this node
• keystore that contains key and cert for transport layer of all nodes in cluster

Elastic Agent requires a PEM-formatted certificate to send encrypted data to ES (see Configure SSL/TLS for self-managed Fleet Servers).

Run the following to convert...

openssl pkcs12 -in path.p12 -out cert.crt -clcerts -nokeys
openssl pkcs12 -in path.p12 -out private.key -nocerts -nodes

Note: For development purposes, run with --insecure flag to acknowledge certificate chain cannot be verified (https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html). Also hod to add fleet-server-es-ca to point to cert generated by Elasticsearch.

Allow Access to Kibana Port

17 - Update kibana.yml to update server.host to a non-loopback address

18 - Restart Kibana from command line

19 - Create inbound rule for Windows Defender Firewall (from Firewall & network protection, select Advanced Settings)

Allow Access to Fleet Server Port

20 - Update inbound rules on Azure VM networking

21 - Add Windows Defender inbound port rule

Add Elastic Agent to IIS Server

This collects both infrastructure data using System integration along with IIS Server logs and metrics.

-> Password for the elastic user (reset with bin/elasticsearch-reset-password -u elastic): vW3doqO4ZucvLVeE__-q

-> HTTP CA certificate SHA-256 fingerprint: 1e4cc28c7551dab8fba3f1b96ecd7972d41b9d3bba5f16634755e8a9f932412a

-> Configure Kibana to use this cluster:

• Run Kibana and click the configuration link in the terminal when Kibana starts.
• Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes): eyJ2ZXIiOiI4LjQuMiIsImFkciI6WyIxNzIuMjcuMC41OjkyMDAiXSwiZmdyIjoiMWU0Y2MyOGM3NTUxZGFiOGZiYTNmMWI5NmVjZDc5NzJkNDFiOWQzYmJhNWYxNjYzNDc1NWU4YTlmOTMyNDEyYSIsImtleSI6ImlfUF9aNE1CNnNGb1RTd19wMVZEOk13dnB6akxTVHp1NFk3OWkzbzVMVGcifQ==

-> Configure other nodes to join this cluster:

• On this node:
  • Create an enrollment token with bin/elasticsearch-create-enrollment-token -s node.
  • Uncomment the transport.host setting at the end of config/elasticsearch.yml.
  • Restart Elasticsearch.
• On other nodes:
  • Start Elasticsearch with bin/elasticsearch --enrollment-token <token>, using the enrollment token that you generated.