wttech / gradle-aem-multi

Example Multi-Module AEM application built by Gradle Build System

AEM Multi-Project Example

Description

This project could be used to start developing long-term project based on AEM.

To start developing application/library based on AEM it is recommended to use Gradle AEM Single instead.

Documentation for AEM plugin is available in project Gradle AEM Plugin.

Screenshot

Table of Contents

• Features
• Quickstart
• Environment
• Structure
• Building
• Tips & tricks
• Running tests
• Attaching debugger
• Extending build

Features

Main motivation of this project is to automate all aspects of AEM development and make it a breeze.

---

Archetyping:

• Generating project from archetype using Gradle Fork Plugin,
• Generating user-specific build properties using friendly GUI

Environment:

• Automatic native local AEM instance(s) setup,
• Automatic HTTPD server with AEM dispatcher setup (based on Docker),
• Hosts file amendment
• Health checking

Back-end:

• Building OSGi bundle ready for AEM sites development

Front-end:

• Setup of popular UI build toolkit: NodeJS, Yarn and Webpack for advanced assets bundling (modular JS, ECMAScript6 transpilation, SCSS compilation with PostCSS, code style checks etc).
• Integrated SCSS compilation on AEM side using AEM Sass Compiler.

Testing:

• Stubbing using AEM Stubs Tool
• Unit tests
• Integration tests using Karate Framework and JSoup.
• Functional tests using Cypress
• Performance tests using Gradle Lighthouse Plugin

Maintenance:

• Automatic AEM migration scripts execution using AEM Easy Content Upgrade,
• Automatic AEM access control configuration applying using Access Control Tool,
• Interactive incident monitoring / logs monitoring with filtering.

Quickstart

1. Fork project using command:

   git clone https://github.com/wttech/gradle-aem-multi.git && cd gradle-aem-multi && sh gradlew fork

   and specify properties:

   

   and wait until project is forked then enter configured target directory.

2. Setup user specific project configuration using command:

   sh gradlew props

   and specify properties:

   

3. Setup local AEM instances with dependencies and AEM dispatcher (see prerequisites) then build application using command:

   sh gradlew environmentHosts
   sh gradlew setup

   and wait till complete AEM environment will be ready to use.

4. Develop continuously application using command:

   sh gradlew

   which is an alias for:

   sh gradlew develop

   or to just deploy AEM application (without running anything else):

   sh gradlew :app:aem:all:packageDeploy

Prerequisites

Tested on:

• Java 1.8, 11
• Gradle 6.7
• Adobe AEM 6.5
• Docker 2.4.0.0

Structure

• app - source code generated from Adobe AEM Archetype 23 and adapted to Gradle / Gradle AEM Plugin,
• env - resources and configuration related with setting up local AEM instances and AEM dispatcher,
• test - integration and functional tests requiring full environment setup.

Environment

Project is configured to have local environment which consists of:

• native AEM instances running on local file system,
• virtualized Apache HTTP Server with AEM Dispatcher module running on Docker (official httpd image).

Assumptions:

• AEM author available at http://localhost:4502
• AEM publish available at http://localhost:4503
• Apache web server with Virtual hosts configured for domains:
  • http://example.com -> which maps to /content/example content root on publish

Building

1. Use command gradlew so that Gradle in version according to project will be downloaded automatically.
2. Deploy application:
   • Full assembly and run all tests
     • sh gradlew <=> sh gradlew :develop
   • Only assembly package:
     • sh gradlew :app:aem:all:packageDeploy
   • Only single package or bundle:
     • sh gradlew :app:aem:core::bundleInstall,
     • sh gradlew :app:aem:ui.apps:packageDeploy,
     • sh gradlew :app:aem:ui.content:packageDeploy.
3. Rebuilding front-end only: sh gradlew :app:aem:ui.frontend:build

Tooling

1. Monitoring errors in logs: sh gradlew instanceTail
2. Synchronizing JCR content from AEM to local file system: sh gradlew :app:aem:ui.content:packageSync
3. Interactively updating HTTPD Virtual-Host & AEM Dispatcher configuration: sh gradlew environmentDev
4. Copying JCR content between AEM instances: sh gradlew instanceRcp -Pinstance.rcp.source=http://user:[email protected]:4502 -Pinstance.rcp.target=local-author -Pinstance.rcp.paths=[/content/example,/content/dam/example]

Tips & tricks

• To run some task only for subproject, use project path as a prefix, for instance: sh gradlew :app:aem:ui.content:packageDeploy.
• According to recommendations, Gradle daemon should be:
  • enabled on development environments,
  • disabled on continuous integration environments.
• To see more descriptive errors or want to skip some tasks, see command line documentation.

Running tests

IntelliJ

Certain unit tests may depend on the results of running gradle tasks. One such example is the testing of OSGi Services using OSGi Mocks where in order to run a test, the SCR metadata must be available for a class. Running a test like this in IntelliJ results in errors because the IDE is not aware of the Bundle plugin.

This can be worked around by configuring IntelliJ to delegate test execution to Gradle. In order to set this up, go to Settings > Build, Execution, Deployment > Gradle > Runner and set your IDE to delegate IDE build/run actions to Gradle. Alternatively, you can use a dropdown menu to use a specific runner or to decide on a test-by-test basis.

Attaching debugger

1. Execute build with options -Dorg.gradle.debug=true --no-daemon, it will suspend,
2. Attach debugger on port 5005,
3. Suspension will be released and build should stop at breakpoint.

Extending build

For defining new tasks directly in build see:

• Build Script Basics
• More about Tasks

The easiest way to implement custom plugins and use them in project is a technique related with buildSrc/ directory. For more details please read documentation.

gradle-aem-multi's Issues

Integrate htl plugin

as in https://github.com/Cognifide/aem-project-archetype/blob/master/src/main/archetype/ui.apps/pom.xml#L147
using https://github.com/Cognifide/gradle-htl-plugin

Upgrade UI toolkit set up in design project

• Node, Yarn versions to be updated
• Packages: https://github.com/Cognifide/gradle-aem-multi/blob/master/app/design/package.json
• Webpack configuration: https://github.com/Cognifide/gradle-aem-multi/blob/master/app/design/src/main/content/jcr_root/etc/designs/example/publish/src/webpack.config.js

Feel free to add any linters etc to make that module even more up to date / using best recent UI toolkit as it is possible.

hosts.bat file stoped working on Windows 10

Use newer DSL: `plugins` and `pluginManagement`

• https://docs.gradle.org/4.3/release-notes.html#more-use-cases-supported-using-the-plugins-{}-block
• https://docs.gradle.org/4.3/userguide/plugins.html#sec:plugin_management

Consider using alternative URL to dispatcher module being downloaded

currently used is not working after each dispatcher module release

e.g https://www.adobeaemcloud.com/content/companies/public/adobe/dispatcher/dispatcher/_jcr_content/top/download_8/file.res/dispatcher-apache2.4-linux-x86-64-4.2.3.tar.gz

Introduce sample servlet using Sling Servlet Annotations

I tried to add dependencies to gradle project for new Servlet Annotations (described here:
https://sling.apache.org/documentation/the-sling-engine/servlets.html#registering-a-servlet-using-java-annotations),
but with no success. I think I miss proper plugin. How to do this the right way?

Integrate Context-Aware Config

http://wcm.io/caconfig/editor/usage.html

Support for KnotX in GAP environment plugin

@Skejven FYI

gradle-aem-multi setups KnotX service side by side with dispatcher

• it is described in docker-compose.yml - additional KnotX container
• KnotX is behind Apache and only page requests get forwarded to it
• there is example content added to multi project thats allows additional rendering to be made by KnotX
• data source for KnotX will be static files configured and mounted
• KnotX configuration files would be bind-mounted

GAP support:

• GAP should be able to perform health checks on KnotX endpoint 8092
• GAP creates all needed logs and

Always resolve IP of AEM in dispatcher configuration

https://docs.adobe.com/content/help/en/experience-manager-dispatcher/using/configuring/dispatcher-configuration.html#enabling-secure-sessions-sessionmanagement

/alwaysResolve may help with not working dispatcher ; could fix an issue when dispatcher is down and only helping thing in that situation is to run environmentRestart task;

quite sure that after adding that the issue will be gone

Docker compose file has incorrect pahts on Windows environments

Hi!
I noticed that paths created in docker-compose.yml file are incorrect on Windows. They contain both backslashes and forward slashes. This leads to docker compose script failure due to "unknown escape character".

In my case the template from docker-compose.yml.peb file:

    volumes:
      - "{{ environment.dockerRootPath }}/distributions/mod_dispatcher.so:/usr/local/apache2/modules/mod_dispatcher.so"
      - "{{ environment.dockerRootPath }}/httpd/cache:/opt/aem/dispatcher/cache"
      - "{{ environment.dockerRootPath }}/httpd/logs:/usr/local/apache2/logs"
      - "{{ environment.dockerConfigPath }}/httpd/conf:/usr/local/apache2/conf"

Gets resolved to:

    volumes:
      - "C:\testwindows\aem\.environment/distributions/mod_dispatcher.so:/usr/local/apache2/modules/mod_dispatcher.so"
      - "C:\testwindows\aem\.environment/httpd/cache:/opt/aem/dispatcher/cache"
      - "C:\testwindows\aem\.environment/httpd/logs:/usr/local/apache2/logs"
      - "C:\testwindows\aem\gradle\environment/httpd/conf:/usr/local/apache2/conf"

Introduce sample backup task

task aemBackup {
   doLast {
      def response = aem.buildPackage(listOf("/content/example"))
      aem.downloadPackage(response.path, project.file("content-backup-${System.currentTimeMilis()}.zip")
   }
}

this will be a little demonstration of writing custom tasks

Configure release

Consider:

• https://github.com/allegro/axion-release-plugin
• https://github.com/researchgate/gradle-release

Configure example to be using plugin artifact stored on BinTray

In progress. See: wttech/gradle-aem-plugin#1

Integrate functional tests

configure https://github.com/smooth-code/jest-puppeteer in https://github.com/Cognifide/gradle-aem-multi/tree/flat-structure/test.functional/ build.gradle.kts

Integrate markup analysis

Using: https://webhint.io/ called via Node task: https://www.npmjs.com/package/hint

Integrate mock server

based on docker with nice GUI for creating mocks

https://github.com/iridakos/duckrails/wiki/Setup-DuckRails-via-Docker

the goal will be to easily setup mocks for local development... by GUI ;)
and all mocks will be saved in VCS and integrated via Docker volume & DuckRails

Plugin not found

* Exception is:
org.gradle.api.plugins.UnknownPluginException: Plugin [id: 'com.cognifide.aem.package', artifact: 'com.cognifide.gradle:aem-plugin:4.1.1'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact 'com.cognifide.gradle:aem-plugin:4.1.1')
  Searched in the following repositories:
    MavenLocal(file:/C:/Users/x/.m2/repository/)
    BintrayJCenter
    maven(https://plugins.gradle.org/m2/)
    maven2(http://dl.bintray.com/cognifide/maven-public)
    maven3(https://dl.bintray.com/neva-dev/maven-public)

Should I update plugin version to 5.0.0 in settings.gradle?
https://github.com/Cognifide/gradle-aem-multi/blob/master/settings.gradle#L14

Rename project to 'gradle-aem-project'

as of this will be project template similar to aem-multimodule-project-archetype

Integrate integration tests

https://github.com/intuit/karate

Migrate dispatcher to centos/httpd

Use https://hub.docker.com/r/centos/httpd/ image instead of official httpd

Create more robust archetype

Maybe use: https://plugins.gradle.org/plugin/com.orctom.archetype

Structure

Root (with https://github.com/researchgate/gradle-release or https://github.com/allegro/axion-release-plugin)

Bundle projects:

• app/core
• app/commons

Content projects:

• app/config (config,config.author,config.publish)
• app/design (with https://github.com/srs/gradle-node-plugin)

Extract dispatcher configuration to project archetypes

Allow users of project archetypes to also fork dispatchers configuration and provide its details specific for the project.

AC

• when forking gradle-aem-multi
  • dev is asked to provide names for demo and live local domains
  • aemEnvHosts task is configured accordingly
  • reverse mappings are generated
  • invalidation handler is configured
  • whole httpd and dispatchers configuration files are generated
  • basic health checks are generated
  • user can start dispatchers development out of the box

Related #78, #355.

Align example to code from Adobe Archetype

aem/sites should have code from https://github.com/adobe/aem-project-archetype/tree/master/src/main/archetype/ui.apps/src/main/content/jcr_root/apps/__appsFolderName__

dispatcher code should be copied from https://github.com/adobe/aem-project-archetype/tree/master/src/main/archetype/dispatcher.cloud and also bundled into ZIP via dedicated Gradle task

Integrate migration scripts run automatically

Imagine

sh gradlew :aem-webapp:aemDeploy :aemAwait :aem-migration:aemDeploy :aemDeploy :test-integration-karate:test

Integrate automated ACL

via Vault hook used as in migration project
https://github.com/Netcentric/accesscontroltool

Upgrade to AEM 6.3

wttech/gradle-aem-plugin#86

TODO

• update Groovy console version working on AEM 6.3
• update uber jar to 6.3

Replication issues

The replication agent on publish in disabled by default which is causing some trouble while working with the instances.

Temporary workflows toggling during package deployment

To prevent renditions being regenerated for every package deploy, it is very often that projects are using such filter:

    <filter root="/content/dam/example-demo" mode="merge">
        <exclude pattern="/content/dam/.*/renditions/.*"/>
        <include pattern="/content/dam/.*/renditions/original(.*)?"/>
    </filter>

This has serious side effect. The images will be never removed and messy images may left on instance(s) unintentionally.

But simply, imagine:

aemDeploy {
    def workflows = [
         "/etc/workflow/launcher/config/update_asset_create"
         "/etc/workflow/launcher/config/update_asset_mod"
    ]
    doFirst {
        aem.sync { it.disableWorkflow(workflows) }
    }
    doLast {
        aem.sync { 
            it.enableWorkflow(workflows)
            // possibility to use any method from: https://github.com/Cognifide/gradle-aem-plugin/blob/master/src/main/kotlin/com/cognifide/gradle/aem/instance/InstanceSync.kt
            // it.restartBundle('x.x.x')
        }
    }
}

Integrate performance testing

Create task that will launch lighthouse: https://www.npmjs.com/package/lighthouse#using-the-node-cli

Introduce server-side junit integration tests

sample:
https://github.com/adobe/aem-project-archetype/blob/master/src/main/archetype/it.tests/src/main/java/it/tests/HelloWorldModelServerSideTest.java
docs:
https://sling.apache.org/documentation/bundles/org-apache-sling-junit-bundles.html

Replace aecu with vault-upgrade-hook

Upgrade kotlin-gradle-plugin due to incompatibility with task configuration avoidance

JetBrains/kotlin#1812 needed when compiling Kotlin using Gradle

aemSync not working

I assume the aemSync gradle task will synchronize the files from the AEM instance with the gradle project files. However, when I run the error that the JCR repositories don't exists, also, I keep getting errors such as

Execution failed for task ':aem:aemSatisfy'.
> Cannot list packages on LocalInstance(httpUrl='http://localhost:4502', user='admin', password='*****', typeName='author', debugPort=14502). Reason: request failed.

Integrate regression testing

Using: https://github.com/BBC-News/wraith
Due to requirement about having ImageMagick installed on machine, it should be deployed on Docker via GAP environment plugin:
http://bbc-news.github.io/wraith/os-install.html
https://hub.docker.com/r/bbcnews/wraith/dockerfile/

Alternatively consider using AET https://github.com/Skejven/aet-docker , but seems that it is just more heavy for local dev machine (but better for production :) / but it is not the case here)

Configure encrypted credentials

https://github.com/etiennestuder/gradle-credentials-plugin

when private repositories are needed to be configured later

Fix model test

https://github.com/Cognifide/gradle-aem-multi/blob/master/app/aem/core/src/test/java/com/company/example/core/models/HelloWorldModelTest.java#L35