AEM Compose
AEMC is a versatile tool for managing Adobe Experience Manager (AEM) instances. With a comprehensive set of commands, it simplifies tasks such as working CRX packages, OSGi configurations, JCR repository nodes, and more. Its seamless integration with Ansible enhances automation capabilities.
- Reusable core designed to handle advanced dev-ops operations needed to manage AEM instances
- Various distributions based on core for context-specific use cases:
- CLI - for developer workstations, shell scripting
- AEM Project Quickstart - add development environment automation to the existing AEM projects
- Docker Example - for experiments only
- Terraform AEM Provider - set up with ease higher environments (VMs and AEM instances) in the cloud (AWS, Azure, GCP, etc.)
- Pulumi AEM Provider - working the same as Terraform AEM Provider but with Pulumi
- Ansible Collection/Modules - for managing higher AEM environments
- Packer Example - starting point for baking AWS EC2 image using Ansible
- Local Example - development & testing sandbox for AEM Compose project
- CLI - for developer workstations, shell scripting
- Fast & lightweight
- Usable on all operating systems and architectures without pre-installing any dependencies
- Blog Posts
- Talk at AdaptTo 2023 Conference - Get Your AEM Together: AEM Compose, the Ultimate DevEx Tool by Tomasz Sobczyk & Krystian Panek
Provides complete set of commands to comfortably work with CRX packages, OSGi configurations, repository nodes and more.
Key assumptions:
- Idempotent and fast
- Rich configuration options
- Self-describing, both machine & human-readable
- Multiple input & output formats (text/yaml/json)
Main features:
- easy & declarative setup of:
- JDK (isolated, version tied to project)
- AEM instances (run modes, JVM & start opts, env & secret vars, Sling props, custom admin password)
- OSGi (configurations, bundles, components)
- Crypto Support, Trust Store, SSL by Default
- replication agents
- any repository nodes
- deploying AEM packages with:
- efficiently upload with optimized memory usage (constant and independent of package size)
- automatic workflow toggling - avoiding DAM asset renditions regeneration
- advanced snapshot handling - avoiding redeploying the same package by checksum verification
- customizable instance health checking
- building AEM packages with:
- source code change detection - avoiding rebuilding application when it is not needed (handled by Taskfile)
- making AEM instance backups (with restoring)
- advanced archive format to speed up performance and storage efficiency (ZSTD used by default)
- instance state aware - stopping, archiving then starting again AEM instances automatically (if needed)
- content synchronization between running instance and local file system
Worth knowing:
- On Windows use it with Git Bash (CMD and PowerShell are not supported nor tested)
Help command:
Instance commands:
Package commands:
Setup task:
- AEM instances are provisioned only when the configuration is changed
- Maven AEM application build executed only when there are code changes
- AEM dispatcher set up using Docker Compose, redeployed only when there are configuration changes as well
- Health checks executed to confirm the stability of the environment as a whole
Supported project types:
-
with structure based on Adobe AEM Project Archetype, compatibility:
AEM Compose (init) AEM Project Archetype (tested) >= 1.2.0 41, 42, 43, 47, 48 -
with any type of structure, however afterwards only a little customizations in Taskfile.yml need to be done to reflect configuration related to built AEM application artifact path and AEM dispatcher files location
-
empty folder; the project kind will be needed to be specified explicitly during initialization
Run command below to initialize the AEM Compose tool in your project:
curl https://raw.githubusercontent.com/wttech/aemc/main/project-init.sh | shand then:
sh aemw initAfter successful initialization, remember to always use the tool via wrapper script in the following way:
sh aemw [command]For example:
sh aemw versionProject initialization sets up ready-to-use tasks powered by Task tool which are aggregating one or many AEM Compose CLI commands into useful procedures.
To list all available tasks, run:
sh taskw --listFor example:
sh taskw setupSome tasks like aem:build may accept parameters.
For example, to build AEM application with:
- Applying frontend development mode Maven profile
- Unit tests skipping
- UI tests skipping
Simply run command with appending task variable to the end:
sh taskw aem:build AEM_BUILD_ARGS="-PfedDev -DskipTests -pl '!ui.tests'"The tool is designed to be used in Infrastructure as a Code (IaaC) solutions such as Terraform or Pulumi.
See separate projects based on AEM Compose:
- Terraform Provider: https://github.com/wttech/terraform-provider-aem
- Pulumi Provider: https://github.com/wttech/pulumi-aem
The tool is designed to be used in Ansible playbooks and roles.
See a separate project based on AEM Compose: https://github.com/wttech/aemc-ansible
Consider implementing any application on top of AEM Compose API like using snippet below:
File: aem.go
package main
import "fmt"
import "os"
import aemc "github.com/wttech/aemc/pkg"
func main() {
aem := aemc.DefaultAEM()
instance, _ := aem.InstanceManager().NewByIDAndURL("remote_author", "http://admin:[email protected]:4502")
changed, err := instance.PackageManager().DeployWithChanged("/tmp/my-package.zip")
if err != nil {
fmt.Printf("cannot deploy package: %s\n", err)
os.Exit(1)
}
if changed {
if err := aem.InstanceManager().AwaitStartedOne(*instance); err != nil {
fmt.Printf("instance not stable after deploying package: %s\n", err)
os.Exit(1)
}
}
fmt.Println("package deployed properly")
os.Exit(0)
}Then to run application use command:
go run aem.goThis tool is written in Go. Go applications are very often self-sufficient which means that they are not relying on platform-specific libraries/dependencies. The only requirement is to use proper tool binary distribution for each operating system and architecture. Check out releases page to review available binary distributions.
The tool tries to make configuration as much explicit as it could be to allow customization in an easy way.
Below are listed all available configuration options (source):
# AEM instances to work with
instance:
# Full details of local or remote instances
config:
local_author:
active: [[.Env.AEM_AUTHOR_ACTIVE | default true ]]
http_url: [[.Env.AEM_AUTHOR_HTTP_URL | default "http://127.0.0.1:4502" ]]
user: [[.Env.AEM_AUTHOR_USER | default "admin" ]]
password: [[.Env.AEM_AUTHOR_PASSWORD | default "admin" ]]
run_modes: [ local ]
jvm_opts:
- -server
- -Djava.awt.headless=true
- -Djava.io.tmpdir=[[canonicalPath .Path "aem/home/tmp"]]
- -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=[[.Env.AEM_AUTHOR_DEBUG_ADDR | default "0.0.0.0:14502" ]]
- -Duser.language=en
- -Duser.country=US
- -Duser.timezone=UTC
start_opts: []
secret_vars:
- ACME_SECRET=value
env_vars:
- ACME_VAR=value
sling_props: []
local_publish:
active: [[.Env.AEM_PUBLISH_ACTIVE | default true ]]
http_url: [[.Env.AEM_PUBLISH_HTTP_URL | default "http://127.0.0.1:4503" ]]
user: [[.Env.AEM_PUBLISH_USER | default "admin" ]]
password: [[.Env.AEM_PUBLISH_PASSWORD | default "admin" ]]
run_modes: [ local ]
jvm_opts:
- -server
- -Djava.awt.headless=true
- -Djava.io.tmpdir=[[canonicalPath .Path "aem/home/tmp"]]
- -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=[[.Env.AEM_PUBLISH_DEBUG_ADDR | default "0.0.0.0:14503" ]]
- -Duser.language=en
- -Duser.country=US
- -Duser.timezone=UTC
start_opts: []
secret_vars:
- ACME_SECRET=value
env_vars:
- ACME_VAR=value
sling_props: []
# Tuning performance & reliability
# 'auto' - for more than 1 local instances - 'serial', otherwise 'parallel'
# 'parallel' - for working with remote instances
# 'serial' - for working with local instances
processing_mode: auto
# HTTP client settings
http:
timeout: 10m
debug: false
disable_warn: true
# State checking
check:
# Time to wait before first state checking (to avoid false-positives)
warmup: 1s
# Time to wait for next state checking
interval: 6s
# Number of successful check attempts that indicates end of checking
done_threshold: 4
# Max time to wait for the instance to be healthy after executing the start script or e.g deploying a package
await_started:
timeout: 30m
# Max time to wait for the instance to be stopped after executing the stop script
await_stopped:
timeout: 10m
# Max time in which socket connection to instance should be established
reachable:
timeout: 3s
# Bundle state tracking
bundle_stable:
symbolic_names_ignored: []
# OSGi events tracking
event_stable:
# Topics indicating that instance is not stable
topics_unstable:
- "org/osgi/framework/ServiceEvent/*"
- "org/osgi/framework/FrameworkEvent/*"
- "org/osgi/framework/BundleEvent/*"
# Ignored service names to handle known issues
details_ignored:
- "*.*MBean"
- "org.osgi.service.component.runtime.ServiceComponentRuntime"
- "java.util.ResourceBundle"
received_max_age: 5s
# OSGi components state tracking
component_stable:
skip: false
pids:
include: ['com.day.crx.packaging.*', 'org.apache.sling.installer.*']
exclude: ['org.apache.sling.installer.hc.*', 'org.apache.sling.installer.core.impl.console.*']
match:
"disabled": []
"no config": []
"unsatisfied (reference)": []
"satisfied": []
# Sling Installer tracking
installer:
# JMX state checking
state: true
# Pause Installation nodes checking
pause: true
# Managed locally (set up automatically)
local:
# Wait only for those instances whose state has been changed internally (unaware of external changes)
await_strict: true
# Current runtime dir (Sling launchpad, JCR repository)
unpack_dir: "aem/home/var/instance"
# Archived runtime dir (AEM backup files '*.aemb.zst')
backup_dir: "aem/home/var/backup"
# Oak Run tool options (offline instance management)
oak_run:
download_url: "https://repo1.maven.org/maven2/org/apache/jackrabbit/oak-run/1.44.0/oak-run-1.44.0.jar"
store_path: "crx-quickstart/repository/segmentstore"
# Source files
quickstart:
# AEM SDK ZIP or JAR
dist_file: "aem/home/lib/{aem-sdk,cq-quickstart}-*.{zip,jar}"
# AEM License properties file
license_file: "aem/home/lib/license.properties"
# Status discovery (timezone, AEM version, etc)
status:
timeout: 500ms
# JCR Repository
repo:
property_change_ignored:
# AEM assigns them automatically
- "jcr:created"
- "cq:lastModified"
# AEM encrypts it right after changing by replication agent setup command
- "transportPassword"
# CRX Package Manager
package:
# Force re-uploading/installing of snapshot AEM packages (just built / unreleased)
snapshot_patterns: [ "**/*-SNAPSHOT.zip" ]
# Use checksums to avoid re-deployments when snapshot AEM packages are unchanged
snapshot_deploy_skipping: true
# Disable following workflow launchers for a package deployment time only
toggled_workflows: [/libs/settings/workflow/launcher/config/asset_processing_on_sdk_*,/libs/settings/workflow/launcher/config/update_asset_*,/libs/settings/workflow/launcher/config/dam_*]
# Also sub-packages
install_recursive: true
# Use slower HTML endpoint for deployments but with better troubleshooting
install_html:
enabled: false
# Print HTML directly to console instead of writing to file
console: false
# Fail on case 'installed with errors'
strict: true
# Number of changes after which the commit to the repository is performed
install_save_threshold: 1024
# Allows to relax dependency handling if needed
install_dependency_handling: required
# Controls how 'rep:policy' nodes are handled during import
install_ac_handling: ''
# OSGi Framework
osgi:
bundle:
install:
start: true
start_level: 20
refresh_packages: true
# Crypto Support
crypto:
key_bundle_symbolic_name: com.adobe.granite.crypto.file
# Workflow Manager
workflow:
launcher:
lib_root: /libs/settings/workflow/launcher
config_root: /conf/global/settings/workflow/launcher
toggle_retry:
timeout: 10m
delay: 10s
java:
# Require following versions before e.g running AEM instances
version_constraints: ">= 11, < 12"
# Pre-installed local JDK dir
# a) keep it empty to download open source Java automatically for current OS and architecture
# b) set it to absolute path or to env var '[[.Env.JAVA_HOME]]' to indicate where closed source Java like Oracle is installed
home_dir: ""
# Auto-installed JDK options
download:
# Source URL with template vars support
url: "https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.18%2B10/OpenJDK11U-jdk_[[.Arch]]_[[.Os]]_hotspot_11.0.18_10.[[.ArchiveExt]]"
# Map source URL template vars to be compatible with Adoptium Java
replacements:
# Var 'Os' (GOOS)
"darwin": "mac"
# Var 'Arch' (GOARCH)
"x86_64": "x64"
"amd64": "x64"
"386": "x86-32"
# enforce non-ARM Java as some AEM features are not working on ARM (e.g Scene7)
"arm64": "x64"
"aarch64": "x64"
base:
# Location of temporary files (downloaded AEM packages, etc)
tmp_dir: aem/home/tmp
# Location of supportive tools (downloaded Java, OakRun, unpacked AEM SDK)
tool_dir: aem/home/opt
log:
level: info
timestamp_format: "2006-01-02 15:04:05"
full_timestamp: true
input:
format: yml
file: STDIN
output:
format: text
log:
# File path of logs written especially when output format is different than 'text'
file: aem/home/var/log/aem.log
# Controls where outputs and logs should be written to when format is 'text' (console|file|both)
mode: consoleNote that environment variables may be injected in any part of config file. Environment variables could be defined in one or many dotenv files.
By default, the VCS-tracked file is loaded (aem/default/etc). However, occasionally developers might want to override the default config file and load a VCS-ignored file instead (aem/home/etc/aem.yml).
To do so, run the command:
sh aemw config initAll configuration options specified in file aem.yml could be overridden by environment variables.
Simply add prefix AEM_ then each level of nested YAML object join with _ and lowercase the name of each object.
For example: instance.local.quickstart.dist_file could be overridden by environment variable AEM_INSTANCE_LOCAL_QUICKSTART_DIST_FILE
Also note that some configuration options may be ultimately overridden by CLI flags, like --output-format.
By default, fail-safe options are in use. However, consider using the configuration options listed below to achieve a more desired tool experience.
export AEM_INSTANCE_PROCESSING_MODE=parallelThis setting will significantly reduce command execution time. Although be aware that when deploying heavy AEM packages like Service Packs on the same machine in parallel, a heavy load could be observed, which could lead to unpredicted AEM CMS and AEM Compose tool behavior.
export AEM_OUTPUT_VALUE=ALLSetting this environment variable will instruct the tool to request from the AEM instance descriptive information about the recently executed command subject.
For example, if a recently executed command was sh aemw package deploy my-package.zip -A the AEM Compose tool after doing the actual package deployment will request from CRX Package Manager the exact information about just deployed package.
This feature is beneficial for clarity and debugging purposes.
To install larger AEM packages that may include content pages, assets, and more, you can adjust the HTTP timeout setting.
By default, the timeout is set to 10m, but you have the option to increase it (e.g., to 3h) or disable it completely (by using 0).
To set the timeout for a single AEMC command, use the following syntax:
AEM_INSTANCE_HTTP_TIMEOUT=0 sh aemw package deploy --file my-package.zipIt's important to be aware that AEMaaCS also has its own timeout for requests made to the Package Manager UI. For detailed information, please refer to the documentation.
To skip the instance health check after running a command that would normally trigger it, AEM_INSTANCE_CHECK_SKIP environment variable can be used. This can save a lot of time when deploying a lot of packages sequentially.
To skip the instance health check for a single AEMC command, use the following syntax:
AEM_INSTANCE_CHECK_SKIP=true sh aemw package deploy --file my-package.zipStarting from version 1.4.0 (see #177), AEMC now supports AEM package installations using an HTML report serving endpoint, similar to CRX Package Manager's. While this method may result in slightly extended installation times, it provides a comprehensive HTML report detailing the package installation process.
This new feature offers two distinct modes for leveraging its benefits:
-
Saving HTML report to file:
AEM_INSTANCE_PACKAGE_INSTALL_HTML_ENABLED=true sh aemw package deploy --file my-package.zip
-
Direct console output of HTML report:
AEM_INSTANCE_PACKAGE_INSTALL_HTML_ENABLED=true AEM_INSTANCE_PACKAGE_INSTALL_HTML_CONSOLE=true sh aemw package deploy --file my-package.zip
-
Configuring publish agent on AEM author:
PROPS=" enabled: true transportUri: http://localhost:4503/bin/receive?sling:authRequestLogin=1 transportUser: admin transportPassword: admin userId: admin " echo "$PROPS" | sh aemw repl agent setup -A --location "author" --name "publish"
-
Configuring flush agent on AEM publish:
PROPS=" enabled: true transportUri: http://localhost/dispatcher/invalidate.cache protocolHTTPHeaders: - 'CQ-Action: {action}' - 'CQ-Handle: {path}' - 'CQ-Path: {path}' - 'Host: flush' " echo "$PROPS" | sh aemw repl agent setup -P --location "publish" --name "flush"
If needed, update
localhostto the value on which AEM dispatcher is available, e.g.localhost:8080.
AEM Compose supports SSL by Default feature of AEM.
This feature requires:
- certificate file in PEM format
- private key file in DER or PEM format
- password for keystore (has to be the same for each invocation of the tool)
- password for truststore (can be different for each invocation of the tool)
- hostname for HTTPS connector (used by AEM to check if the setup was successful; has to be reachable by AEM)
- port for HTTPS connector
To set up SSL by Default, run:
sh aemw ssl setup \
--instance-author
--keystore-password password \
--truststore-password password1 \
--certificate-file localhost.crt \
--private-key-file localhostprivate.key \
--https-hostname localhost \
--https-port 8443
sh aemw ssl setup \
--instance-publish
--keystore-password password \
--truststore-password password1 \
--certificate-file localhost.crt \
--private-key-file localhostprivate.key \
--https-hostname localhost \
--https-port 9443Above example uses the self-signed certificate created as per the AEM docs.
See the reference documentation: AEM 6.5 > Administering Guide > SSL by Default
For local environment remember to set different port numbers for author and publish instances.
AEM Compose supports managing the trust store of AEM instances.
This feature supports:
-
creation of the general trust store if it does not exist
sh aemw gts create --password PASSWORD_HERE -A
-
getting the general trust store status
sh aemw gts status -A
-
adding a certificate to the general trust store
sh aemw gts certificate add --path <path> -A
This command will add a certificate to the general trust store only if not exists in trust store and will return the alias of the certificate.
Command certificate add supports certificates in PEM and DER formats.
-
reading a certificate from the general trust store (by alias)
sh aemw gts certificate read --alias <alias> -A
-
removing a certificate from the general trust store (by alias)
sh aemw gts certificate remove --alias <alias> -A
If you're migrating from Gradle AEM Plugin you may run into issue with setting up dispatcher. In case of error when running task dispatcher:start or dispatcher:setup:
2024-01-22 16:37:23 dispatcher | 2024/01/22 15:37:23 Dispatcher configuration validation failed:
2024-01-22 16:37:23 dispatcher | 2024/01/22 15:37:23 /mnt/dev/src/conf.d/variables/default.vars: lstat /etc/httpd.extra: no such file or directory
2024-01-22 16:37:23 dispatcher |
2024-01-22 16:37:23 dispatcher | ERROR Mon Jan 22 15:37:23 UTC 2024 Configuration invalid, please fix and retry,
2024-01-22 16:37:23 dispatcher | Line numbers reported are correct for your configuration files.
Check if there is a broken symlink/file in dispatcher/src/variables/default.vars. If the file exists, delete it and
run the task again.
Issues reported or pull requests created will be very appreciated.
- Fork plugin source code using a dedicated GitHub button.
- See development guide
- Do code changes on a feature branch created from main branch.
- Create a pull request with a base of main branch.
- Creator, owner, and maintainer: Krystian Panek
- Consultancy: Tomasz Sobczyk, Maciej Majchrzak
- Contributors: <see all>
AEM Compose is licensed under the Apache License, Version 2.0 (the "License")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent