shinesolutions / aem-opencloud-manager Goto Github PK

Describe the bug
Jenkins shows this error message when create-aem-aws-stack-builder-resources pipeline is executed:

Console Output
Started by user [email protected]
ERROR: Unable to find provisioners/jenkins/jenkinsfiles/aem-opencloud/installation/aws/create-aem-aws-stack-builder-resources/ from git https://github.com/shinesolutions/aem-opencloud-manager
Finished: FAILURE

To Reproduce
Steps to reproduce the behavior:

1. Go to AEM OpenCloud Manager folder
2. Click on installation, aws, and then create-aem-aws-stack-builder-resources
3. Start the build pipeline, it will fail
4. View the console output result and it shows that error message

Expected behavior
Pipeline should succeed and create an AEM AWS Stack Builder resources stack

Environment (please complete the following information if relevant):

• This bug occurs on main branch and latest release.

Additional context
This error is basically caused by AEM OpenCloud Manager codebase not having the Jenkinsfile for that pipeline.

There is a missing environment variable not set in the pipeline:
env.AOC_CONFIG_PROFILE_DIR_AEM_AWS_STACK_BUILDER

which is needed by GenerateOfflineSnapshotConfig.groovy, as it's empty a null value is substituted causing the job to fail.

Need to add new pipeline for creating CDN resource on Manage Environments section.

Is your feature request related to a problem? Please describe.
Currently a deployment involves graceful restart of Apache httpd which reloads most configurations, however, there are certain changes e.g. MaxRequestWorker and ServerLimit which require non-graceful restart (stop and start to be on the safe side).

Describe the solution you'd like
Add another pipeline on Operational Task section to trigger Apache httpd restart.

Additional context
The reason why graceful restart is used by default is because if we do hard restart, it actually causes several seconds where Apache httpd will be down and this causes monitoring to detect downtime and triggers alerts subsequently. Hence AOC's stance is to perform graceful restart by default to avoid downtime detection. And by having Operational Task to perform hard restart, it indicates that the user explicitly decide to do the hard restart and taking the responsibility of managing the alerts that might be triggered.

For AEM Consolidated and Full-Set creation pipelines, the deployment stage needs to be skipped when deployment descriptor URL is not supplied / empty.

Check out this article for an example implementation https://comquent.de/en/skipped-stages-in-jenkins-scripted-pipeline/

When Switch DNS Consolidated pipeline is executed, it updates the DNS record with a target value, however this target value is a Full-Set author-dispatcher's and not a Consolidated target.

Currently mirror libraries only support versioned releases, but we also need to add support for branch download.

Need to refactor AocMirrorGithubLibrary to do the following:

• check if branch exists, then download branch artifact at https://github.com/${user}/${repo}/archive/${branch}.tar.gz , and then repackage this artifact to remove the ${repo}/${branch} subdir, this way it's consistent with the released version which doesn't have such subdir
• check if version exists, then download released version at https://github.com/${user}/${repo}/releases/download/${version}/${repo}-${version}.tar.gz
• otherwise, log error message

As part of Release category, we need to add a new pipeline that will copy artifacts defined in a descriptor file from a source bucket to a destination bucket.

The idea is that this pipeline would be useful and be flexible enough to move deployable artifacts between buckets across AWS accounts.

The usage of the GenerateOfflineSnapshotConfig or GenerateOfflineSnapshotYaml fails if the dir it tries to save the yaml file to does not exists.

We should make sure that the dir get's created before saving the file

To help ensure better compliance with security requirements in various sites we need to add creation of a secure string to SSM using Ansible for later consumption in the license lookup stage with Puppet.

In short, the creation process needs to occur before consumption in the Cloud Formation template that Ansible uses.

The automation around generating the offline-snapshot yaml configuration file via source stack prefix isn't working as expected. The problem is that the automation always picks up the snapshot from the author-standby instance for creating the AEM Author components for Full-Set & Consolidated.

This script

https://github.com/shinesolutions/aem-opencloud-manager/blob/master/src/cloud/aws.groovy#L35

needs to be updated to filter out the snapshot of the author-standby instance.

The AEM AWS Stack Builder allows it to use the deployment descriptor of the green stack to create the blue AEM Stack

https://github.com/shinesolutions/aem-aws-stack-builder/blob/master/provisioners/ansible/playbooks/apps/aem/stack-data.yaml#L101-L109

However the blue/green pipeline only use the source stack prefix atm to generate a configuration file for using the correct snapshot ids. We need to improve this process to also generate a configuration file which allows it to use the deployment descriptor of the green stack.

Jenkins password is currently a user configuration (specified in YAML), this should be moved to AWS resources as a secret on AWS Secrets Manager.

This needs to be optional (resource created with a condition), and only retrieve secret from AWS Secrets Manager if the password is not supplied in YAML file.
This is to allow users who don't currently approve the use of AWS Secrets Manager to use the YAML user config as a fallback mechanism.

Hello,

I'm trying to run the create-full-set job in eu-west-1 however the job fails during the test-readiness-full-set: Poll to check if command was executed step and the 'TestReadinessFullset' SSM command results in timeout, even though the resources are deployed successfully. The only output from the SSM command is:

      Testing if AEM Full-Set is ready using InSpec...
      inspec exec . --show-progress --controls=\
        orchestrator-instances-provisioned-successful

After doing some digging, I saw that the command is being run on the orchestrator, using the inspec-aem-aws included in the stack-provisioner tarball, under the files/inspec directory. This includes a default configuration aem-aws.yml file with empty values. The helper.rb library for inspec-aem-aws will try to ready the config file and since there are no values it will call ruby_aem_aws with an empty config and ruby_aem_aws will default to the constant ap-southeast-2 for the region. As a result inspec will not be able to query the AWS resources in eu-west-1 to determine successful deployment.
I tried to figure out a way to set the region for inspec-aem-aws, but in any case, the helper.rb library will not look for a 'region' variable. I was able to manually run the check-readiness-full-set after adding an aws.region variable to aem-aws.yml and customizing helper.rb to read it and pass it to ruby_aem_aws.

I got past this step using a modified stack-provisioner tarball which included the above aem.region variable changes, but then I had a similar issue during the Run acceptance tests Jenkins stage, where aem-test-suite will use the vendored inspec-aem-aws, default to ap-southeast-2 and as a result fail to query the required cloudwatch alarms.

Has anyone seen this issue before? Have I missed some required configuration variable somewhere which causes the issue?

As part of Stack Manager Messenger testing with AEM OpenCloud Manager, let's list down the exact permissions required.

These permissions need to be added to:

• AWS Permissions wiki page (will be added to future user handbook) https://github.com/shinesolutions/aem-opencloud-manager/blob/master/docs/aws-permissions.md#aem-stack-manager-messenger
• Example AWS Permissions CloudFormation template (used by user's cloud/infra team) https://github.com/shinesolutions/aem-opencloud-manager/blob/master/examples/aws-permissions.yaml

Is your feature request related to a problem? Please describe.
If an offline-snapshot gets triggered at the same time when a live-snapshot is running the offline-snapshot process will fail.

Describe the solution you'd like
The AEM Offline-snapshot should wait until the live-snapshot process finishes and than run the offline-snapshot.

Describe alternatives you've considered
none

Additional context
no additional context.

Custom Image Provisioner, Custom Stack Provisioner, and descriptor URLs are currently configurable at pipeline runs.

Need to add configuration properties for them so users can fill in default values.
This will then help future pipeline runs where the person triggering the build no longer needs to provide URLs to those artifacts, and instead, the person only needs to modify the default - usually this involves changing a version number.

Is your feature request related to a problem? Please describe.
Configuration parameters in aem-aws-stack-builder e.g. aem.[author|publish].jvm_opts & aem.[author|publish].jvm_mem_opts are only getting set when you setup an AEM OpenCoud Vanilla stack.

Those configuration parameters were set correctly on existing AOC environments in the past but since the whole AEM installation is part of the snapshot those configuration parameters are not getting set anymore when you create an AOC environment based on AOC snapshots.

Describe the solution you'd like
To make sure we always set the values which were provided in the configuration parameters we should use the aem::config manifest to reset the binaries during the provisioning process. This follows the same process as what is currently implemented in the installation manifests of puppet-aem_curator.

Describe alternatives you've considered
no alternatives.

Additional context
A fix will also fix these:
shinesolutions/puppet-aem-curator#193
shinesolutions/puppet-aem-curator#152

We need to hide a number of parameters which are set up once-off during pipeline provisioning. The idea is that we shouldn't confuse users (specially non-ops folks) when they see the build parameters list, some of those parameters don't need to be modified during on-demand execution of the build pipelines.

The ones that need to be hidden are:

• JENKINS_AGENT_DOCKER_IMAGE
• JENKINS_AGENT_DOCKER_ARGS
• AWS_LIBRARY_S3_BUCKET
• AWS_LIBRARY_S3_PATH

we should introduce a flag to give the user the control to enable/disable the deletion of the pre-requisite stack as well.

Describe the bug
The pipeilne create-full-set does not support the creation of full-sets with a disabled Chaos Monkey component. The acceptance test stage in the pipeline only supports full-sets with enabled chaos monkey component.

To Reproduce
Steps to reproduce the behavior:

1. Disable chaos monkey in the configuration profile
2. Go to e.g. aem-opencloud-5.12.1-pre.0/manage-environments/aws/aem-full-set-rhel7-aem65-jdk8-sandpit-sandpit/create-full-set
3. Click on Build with parameters
4. See error

Expected behavior
The create-full-set pipeline should support the creation of FUll-Sets with disabled Chaos monkey component

Environment (please complete the following information if relevant):

• Any AEM OpenCloud Version

Additional context
Error mesage:

Profile: AEM-AWS InSpec profile (aem-aws)
Version: 2.4.0
Target:  local://

[38;5;9m  ×  full-set-cloudwatch-alarms-exists: Check if full-set cloudwatch alarms exist (7 failed)[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorDispatcher-AtLeastOneUnHealthyAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorDispatcher-MoreThanOneUnHealthyAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorDispatcher-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorDispatcher-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorDispatcher-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorDispatcher-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorDispatcher-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorDispatcher-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorDispatcher-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorDispatcher-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Author-SyncDelayAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Author-MultiAuthorInstanceAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Author-NoAuthorInstanceAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorPrimary-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorPrimary-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorPrimary-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorPrimary-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorPrimary-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorPrimary-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorPrimary-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorPrimary-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorStandby-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorStandby-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorStandby-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorStandby-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorStandby-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorStandby-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-AuthorStandby-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-AuthorStandby-MemoryHighAlarm"[0m
[38;5;9m     ×  acceptance should have all full set cloudwatch alarms "Warning-ChaosMonkey-NoInstanceAlarm"
     expected #has_all_full_set_cloudwatch_alarms?("Warning-ChaosMonkey-NoInstanceAlarm") to return true, got false[0m
[38;5;9m     ×  acceptance should have all full set cloudwatch alarms "Warning-ChaosMonkey-RootDiskSpaceHighAlarm"
     expected #has_all_full_set_cloudwatch_alarms?("Warning-ChaosMonkey-RootDiskSpaceHighAlarm") to return true, got false[0m
[38;5;9m     ×  acceptance should have all full set cloudwatch alarms "Critical-ChaosMonkey-RootDiskSpaceHighAlarm"
     expected #has_all_full_set_cloudwatch_alarms?("Critical-ChaosMonkey-RootDiskSpaceHighAlarm") to return true, got false[0m
[38;5;9m     ×  acceptance should have all full set cloudwatch alarms "Warning-ChaosMonkey-CPUHighAlarm"
     expected #has_all_full_set_cloudwatch_alarms?("Warning-ChaosMonkey-CPUHighAlarm") to return true, got false[0m
[38;5;9m     ×  acceptance should have all full set cloudwatch alarms "Critical-ChaosMonkey-CPUHighAlarm"
     expected #has_all_full_set_cloudwatch_alarms?("Critical-ChaosMonkey-CPUHighAlarm") to return true, got false[0m
[38;5;9m     ×  acceptance should have all full set cloudwatch alarms "Warning-ChaosMonkey-MemoryHighAlarm"
     expected #has_all_full_set_cloudwatch_alarms?("Warning-ChaosMonkey-MemoryHighAlarm") to return true, got false[0m
[38;5;9m     ×  acceptance should have all full set cloudwatch alarms "Critical-ChaosMonkey-MemoryHighAlarm"
     expected #has_all_full_set_cloudwatch_alarms?("Critical-ChaosMonkey-MemoryHighAlarm") to return true, got false[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Orchestrator-NoInstanceAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-Orchestrator-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Orchestrator-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-Orchestrator-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Orchestrator-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-Orchestrator-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Orchestrator-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-PublishDispatcher-AtLeastOneUnHealthyAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-PublishDispatcher-MoreThanOneUnHealthyAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-PublishDispatcher-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-PublishDispatcher-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-PublishDispatcher-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-PublishDispatcher-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-PublishDispatcher-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-PublishDispatcher-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-PublishDispatcher-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-PublishDispatcher-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-Publish-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Publish-RootDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-Publish-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Publish-DataDiskSpaceHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-Publish-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Publish-CPUHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Warning-Publish-MemoryHighAlarm"[0m
[38;5;41m     ✔  acceptance should have all full set cloudwatch alarms "Critical-Publish-MemoryHighAlarm"[0m


Profile Summary: 0 successful controls, [38;5;9m1 control failure[0m, 0 controls skipped
Test Summary: [38;5;41m54 successful[0m, [38;5;9m7 failures[0m, 0 skipped
make[1]: *** [test-acceptance-full-set] Error 1
make[1]: Leaving directory `/tmp/shinesolutions/aem-opencloud-manager/aem-test-suite/vendor/inspec-aem-aws'
make: *** [test-acceptance-architecture-full-set] Error 2

Describe the bug
After promoting the author-standby to author-primary using the promote-author pipeline the author instance isn't accessible on AEM 6.4 & AEM 6.5

To Reproduce
Steps to reproduce the behavior:

1. Run the operational-task promote-author
2. Access the author instance
3. AEM Login page appears
4. after login nothing happens or you are able to login but no assets can be load

Expected behavior
The promoted author-primary instance should act like the original author-primary instance

Environment (please complete the following information if relevant):

• AEM 6.4 & AEM 6.5

Additional context
Solution to fix is to generate the config file org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config & org.apache.jackrabbit.oak.segment.standby.store.StandbyStoreService.config with the content of an author-primary instance

Reference: https://docs.adobe.com/content/help/en/experience-manager-65/deploying/deploying/tarmk-cold-standby.html

A solution implementation should also fix the issue shinesolutions/puppet-aem-curator#200 reported by @henrykuijpers.

Is your feature request related to a problem? Please describe.
With plenties of variables in a project, it is easy to encounter pipeline failure due to configuration errors, especially when Opencloud is upgraded with variable modification.

Describe the solution you'd like
Before running pipelines, all required variables should be checked, making sure them meet the criteria.

Describe the bug
Some variables are not described in the documentation.

The current aws-resources pipelines are related to Packer AEM and AEM AWS Stack Builder life cycles.
There are some aws-resources that would follow the user's lifecycle (the user's AWS account) and not Packer AEM / AEM AWS Stack Builder. For example, the EC2 keypair only need to be created once for each account, and rotated as necessary.
We need a Jenkins pipeline that will run a playbook and apply CF template to provision the following AWS resources:

• EC2 keypair
• S3 bucket for artifacts
• S3 bucket for Packer AEM
• S3 bucket for AEM AWS Stack Builder
• AEM OpenCloud Manager IAM roles
• ACM cert for storing a self-signed cert