Currently a given Devfile version corresponds to a given Open Liberty version, which will appear in all of:

• the prop in the mvn invocations within the Devfile: mvn -Dliberty.runtime.version=20.0.0.10 ....
• the Open Liberty version cached in the stack image
• the Open Liberty version used in the (stage 2) base of the outer loop build.

To use a different version, Champ, the architect, could customize the stack, using the build.sh script and eventually more of the Devfile "parent" extension mechanisms.

For Jane to customize the stack, she needs to:

1. Override the version in the devfile
2. Pass the version to the outer loop build as a parameter

Note this is going to potentially result in a slower startup time and outer loop build time as we no longer have cached features and OL install.

This could perhaps be mitigated in one place by having stage 1 outer loop build continue to use the stack image OL. For packaging a WAR and generating config, the OL version would typically not matter. (I'm sure there could be a corner case down the road).

Finally, we should consider, turning back to the Champ perspective, what Champ could do to prevent Jane from override the OL version, at least in the outer loop build, and forcing a given OL version there.

Using configure.sh for our outer loop bulid will set us up for better-performance on startup, etc.

It might be nice to add a (non-default) option to skip this step, which would result in a quicker outer loop build, as an alternative for those willing to make that tradeoff.

We should be releasing under
http://dockerhub.com/r/openliberty

The upfront work should include:

• creating the Dockerhub repo
• adding a script to automate build and release of a new image
  • Possibly consider a GitHub Action

(Copied over from appsody/stacks#771)

Some things to focus on:

• Setting up the Maven repository including openliberty runtime/feature installation (including service / ifixes potentially)
• The development image adding maven: kabanero/ubi8-maven:0.3.1 (maybe mention how this is a generic Java/Maven image not especially tied to Open Liberty.

I also wonder if there's any value in defining an "offline Maven" path through the build? By that I mean, instead of assuming there's an internal Maven repository, assume there's no other Maven repository beyond that built into the stack, i.e. all the dependencies will be built into the stack up front.

Worth reading this issue: eclipse-che/che#15518 (comment)

OpenLiberty/ci.docker#193

Even with dev mode certain changes, like some to JVM options, might require a server restart. How can we do this without running through the devBuild command? Is there a better way than odo delete; odo create ... ?

The templates/stackimage/Dockerfile is a bit hard to follow for someone who's trying to follow the example to customize their stack image.

It could use some high-level grouping/organization into:

1. populating Maven repo with dependencies
2. running liberty:dev mode to populate dev mode, indirect dependencies
3. creating directories to be used in the outer loop Dockerfile build (we create them here to take advantage of the fact we're running as root, so Jane doesn't have to when building the app image).
4. Creating the OL install on disk with features.

The customer (Champ) may not want to do all these four or maybe not to the same extent or in the same manner. It would help to document better to understand what they may or may not want to skip or do differently.

E.g. running liberty:dev could simply be skipped and you'd still build in the majority of mvn deps..it's a relative few extra you get.

There should be more doc on this, either in comments or elsewhere.

Need to get more detail, but we need to do this for 4Q20.

Need to ensure:
* deployed apps show up on kappnav
* github repo of deployed apps show up on kappnav

It should be at the level matching the Maven built into the stack image. Bonus points if we can add a single variable to the stack build to only have to specify this value in a single place.

This may happen automatically since our Java 8 devfile is an extension, but we need to still look at how to manage the Java version between inner and outer loop. The outerloop Dockerfile is using Java 11.

Some variants:

• using parent POM provided by the ci.maven project, or otherwise binding start/stop server into test lifecycle
• using liberty-maven-plugin but not in the above manner
• not using liberty-maven-plugin at all, but with an existing Maven WAR project

It's possible the last variant should fall out into its own unique case but my guess it is fits with the other two.

A bit better to use the (final) released version rather than an M4 milestone.

We should also add an entry to:
https://github.com/OpenLiberty/application-stack/wiki/Versioning-stack-artifacts

to describe the checklist of updates we need to make here.

Provide a devfile.yaml which uses the base devfile.yaml to allow building the starter application in a container with java 8

Could be Open Shift only

No need to include DB yet (still prototyping).

Format TBD. I think we need something more than README.md, but not sure exactly which format. @awisniew90 let me know what you think. We also want to get this reviewed by Don Bourne and team as we get closer.

Assuming there's not a special way to ignore /target for Maven projects, then just add .odoignore of /target dir to our starters.

No need for client side ports with MST (I believe). Also let's just use / as ctx root.

SHORT TERM

In the short term we'll take a pass at this in #60, but will revisit post-4Q.

LATER

Since we don't have a stack update/build tool, we need to explicitly and transparently execute a process for making stack updates

E.g. at a minimum, we might expect:

• devfile updates
• starter artifact updates (some reference to stack version?)
• Docker stack image build, push scripts
• Travis testing? / GitHub Action testing?
• PR to devfile registry repo

Of course, in the addition to the above baseline for any change at all, there will be of course changes depending on the content being updated

If we assume a Java 11 pre-req, we can explicitly bring in the jdbc-4.3 feature

OTOH, if we stay at a Java 8 pre-req, then, since the jpa-2.2 feature brings in jdbc-4.2, we could avoid explicitly specifying the jdbc feature in our starter server.xml.

This could result in a slightly different DS config:
https://www.ibm.com/support/knowledgecenter/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_dep_configuring_ds.html

Maybe it is a good thing to require users install the same level of Java as the one in the containers, rather than a burden to be avoided.

There are a lot of options you can use with odo create and so not hard to misunderstand and do the wrong thing...

E.g. if someone should have used the starter to create but created an "empty" component with just:
odo create java-openliberty

you would get this, on odo push

◑ Executing devbuild command "if [ -e /projects/.disable-bld-cmd ]; then echo "found the disable file" &&... ✗
Unable to exec command [/bin/sh -c cd /projects && if [ -e /projects/.disable-bld-cmd ]; then echo "found the disable file" && echo "devBuild command will not run" && exit 0; else echo "will run the devBuild command" && echo "...moving liberty" && mkdir -p /projects/target/liberty && mv /opt/ol/wlp /projects/target/liberty && mvn -Dmaven.repo.local=/mvn/repository package && touch ./.disable-bld-cmd; fi]:
will run the devBuild command
...moving liberty
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.189 s
[INFO] Finished at: 2020-08-20T18:21:17Z
[INFO] ------------------------------------------------------------------------
[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/projects). Please verify you invoked Maven from the correct directory. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MissingProjectException

✗ Executing devbuild command "if [ -e /projects/.disable-bld-cmd ]; then echo "found the disable file" && echo "devBuild command will not run" && exit 0; else echo "will run the devBuild command" && echo "...moving liberty" && mkdir -p /projects/target/liberty && mv /opt/ol/wlp /projects/target/liberty && mvn -Dmaven.repo.local=/mvn/repository package && touch ./.disable-bld-cmd; fi", if not running [5s]
✗ Failed to start component with name java-openliberty. Error: Failed to create the component: command execution failed: unable to execute the run command: error while streaming command: command terminated with exit code 1
[root@a38191284286 ocw2]#

It would be nice to give a clearer, less noisy error.

This is probably only internal/Wiki doc.

To track updates to each of:

1. devfile itself
2. starter
3. stack image

we should describe the workflow for each of:

• The project developer releasing one or more of these
• The application developer "Jane" trying to reference the GitHub source for the project they have created from devfile and possibly starter (understanding they could have made changes we can't track).

We could employ some combo of: version ids, tags, numbers, image labels, etc. in the appropriate artifacts: devfile, pom, Git repo, etc within GitHub repos, Docker Hub repos, devfile registry, etc.

Add the basic:

.vscode/launch.json
.vscode/tasks.json

Possibly using OpenShift Connector plugin if there's anything special added by it.

Log from pipeline when doing docker build:

step-build
STEP 1: FROM openliberty/application-stack:0.2 AS compile
Getting image source signatures
Copying blob sha256:ffc6e0e42cd81d0e1330e0d62c9c327cbe502fbbd75f1cc2096b51cba31b77fa
Copying blob sha256:ec1681b6a383e4ecedbeddd5abc596f3de835aed6db39a735f62395c8edbff30
Copying blob sha256:c4d668e229cd131e0a8e4f8218dca628d9cf9697572875e355fe4b247b6aa9f0
Copying blob sha256:62ce84dcd6db67c652f91b733dbf24a48eed62735866a801c98bcc6f463c9827
Copying blob sha256:07d56f29999d747adefdac6ab4cb22a82495aacf233fc1d8947aef4f088d381c
Copying blob sha256:5136161c2195466b3ad1191bc1212d3c57d52a27da5b804c4b7e56928ef9f42c
Copying blob sha256:e636531b9fc9d9152735e0598096be65eac77eb43eac010b5d394267ef3b0b69
Copying blob sha256:c7395efd8cd62a39cd6b388309ef90a413de5fdbdf822e2b384e8d6b0c3a4cb1
Copying blob sha256:d4b5e2a72a9ce660270ca823173eb40f0b54bfc8e6dfa68d71a2fd4a4faafddf
Copying blob sha256:5313e5eb605f5a04c7837a7505dfa4175a06cd332b48b62c529dd720d652daec
Copying blob sha256:1db0db446e350d6b7ce76724cd7f469447060deb095c8e8a57518163b0f37b6e
Copying blob sha256:4df258a59cc9390baca73bb77e53c97f42040fb26d6535f7cedee471bc28b30b
Copying blob sha256:5b20fc98b693edb787ab0361cb8e3e258147eca36402f36d164991703b920e74
Copying blob sha256:21b550370a2317e29b0bff83150cbb1be5dcfe2510fb104c7b49eb25f2657232
Copying blob sha256:b8fabb2a82151bfa81ea8f8c7c0ed9cefa32572ab656da1f1fe6c46062a30e16
Copying config sha256:52594e517d37fe606642ac5cc352f19f81cbc43ece339af40bd355e5985dc0e0
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir /configlibdir && mkdir /config && mkdir /shared
mkdir: cannot create directory ‘/configlibdir’: Permission denied
error building at STEP "RUN mkdir /configlibdir && mkdir /config && mkdir /shared": error while running runtime: exit status 1

Also from manual test:

$ docker run -ti openliberty/application-stack:0.2 /bin/sh
Unable to find image 'openliberty/application-stack:0.2' locally
0.2: Pulling from openliberty/application-stack
ec1681b6a383: Already exists
c4d668e229cd: Already exists
07d56f29999d: Pull complete
5136161c2195: Pull complete
ffc6e0e42cd8: Pull complete
62ce84dcd6db: Pull complete
c7395efd8cd6: Pull complete
e636531b9fc9: Pull complete
d4b5e2a72a9c: Pull complete
5313e5eb605f: Pull complete
1db0db446e35: Pull complete
4df258a59cc9: Pull complete
5b20fc98b693: Pull complete
21b550370a23: Pull complete
b8fabb2a8215: Pull complete
Digest: sha256:be82e25642f83649ef8223508fa5298fa3ebe40bd3ebaaa76328cffce2a52705
Status: Downloaded newer image for openliberty/application-stack:0.2
sh-4.4$ whoami
java_user
sh-4.4$ mkdir /configlibdir
mkdir: cannot create directory ‘/configlibdir’: Permission denied
sh-4.4$ mkdir /config
mkdir: cannot create directory ‘/config’: Permission denied
sh-4.4$ mkdir /shared
mkdir: cannot create directory ‘/shared’: Permission denied
sh-4.4$ exit

I'm a bit confused exactly how debug is supposed to work.. these links:

redhat-developer/odo#3187
redhat-developer/odo#3059

seem to suggest that you either push in debug mode or non-debug mode whereas going back a ways, this:
redhat-developer/odo#822
discussion suggests debug mode maybe should always be enabled.

As a stack user, the use case is simply to be able to connect an IDE debugger.

I'm not clear though if we need to force the user to restart the container in order to switch, or maybe if we'd only do that if we really wanted to optimize non-debug perf. startup time and run with -Ddebug=false.

(We may or may not include "debug configurations" for Eclipse, VSCode, etc. .. I'd probably break that out into a separate issue.)

More investigation needed, I started a Slack conversation here: https://openshiftdo.slack.com/archives/CR77F5GEQ/p1597930737002400

A number of things in the current templates/default/pom.xml
that don't align with the design in (currently slide 28 shows key excerpts, but the slide # could change) in:
https://ibm.box.com/s/0rnnsg6sh6lv6edq4f6hzkhrakc3sgu1

This doesn't necessarily need to do much more than cleanup the:
https://github.com/OpenLiberty/application-stack/blob/master/README.md
which is a bit out-of-date (e.g. the odo url create is no longer required on Open Shift).

Bonus points to have a generic k8s (vs. OCP) section.. or else just make clear the doc is OCP only.

If we add the Maven volume from: #47 that'd probably be worth explaining.

The "debug" probably is worth a quick mention (maybe a separate #59 isn't even needed).

SAMPLE VS STARTER?

We might be able to use the starter as-is or may need to extend via a "sample"..not sure yet.

See redhat-developer/odo#5141.

OTHER OBSERVATIONS

Noticing some other Devfiles use PROJECTS_ROOT where we use /projects?

https://github.com/odo-devfiles/registry/blob/0b0e73401e462324a8db32b60c625f56ad639391/devfiles/java-quarkus/devfile.yaml#L26-L31

Should we do the same? What flexibility are we missing out on now?

There was once a PROJECTS_SOURCE.. has this replaced PROJECTS_ROOT???

Currently we have a default username and password configured in our starter via quickStartSecurity. We should consider using an auto generated password for the inner loop and removing the qss config when deploying in the outerloop.

Add failsafe:verify to odo test command so odo test will return a non-zero exit status

I think the latest doc is here:

https://github.com/openshift/odo/blob/master/docs/public/debugging-using-devfile.adoc

Is there a standard way of capturing Open Liberty version?

Also add common ones like:

• org.opencontainers.image.authors
• org.opencontainers.image.vendor
• org.opencontainers.image.url

It's be easier to visually scan the build command in the form of:

build-cmd.sh 20.0.0.10

With output whether it runs fully or sees the marker file and stops.

A number of things in the current templates/default/src/main/liberty/config/server.xml
that don't align with the design in (currently slide 29 but could change) in:
https://ibm.box.com/s/0rnnsg6sh6lv6edq4f6hzkhrakc3sgu1

E.g.

, the WAR name, the httpDispatcher element.

In general, the liberty-maven-plugin dev mode has recognized that certain updates will require a restart of the Liberty server.

Does liberty-maven-plugin automatically do the restart for all appropriate cases?

Relying on a message to the user is even harder in the odo environment when the user isn't necessarily looking at the dev mode log.

Investigate...

See OpenLiberty/open-liberty#10021 (comment)

See: https://github.com/OpenLiberty/application-stack/wiki/Links for list of related repos

Add travis automated verification to application stack

With a typical flow creating from a starter, after doing:
$ odo create java-openliberty --starter
you should have a devfile referencing, as its parent, a specific version of the java-openliberty application stack devfile.

However, the starter artifacts will not necessarily be versioned along with the stack devfile and Docker image. The starter could be updated and another project could be created from the same stack version, but with a different starter.

This shouldn't be a big deal, since a user will be customizing the starter themselves.

Every time the stack is updated, it should be tested to ensure a project created from the latest version of EACH starter will worth with this most recent stack version. It won't effect existing projects which should be pinned to an existing version of the devfile.

Every time a starter is update it should be tested to ensure it works with the latest version of the stack.

ASSUMPTION: Once a project is created, it does not use starters again, so it isn't possible for a change in starter to break an existing project. Note, this assumes something like an "application project". You could say here's my "test project", which always does a create from starter, so yes, it is effected by a change in starter at a given stack

At the very least, we should include, within the starter project (POM?) a reference to the SHA that shows the original starter source cloned from the Git repo hosting it.

We should provide a way to test HTTPS endpoints. This currently fails because the test client cannot accept the default self signed certificate from the server.

The stack currently uses MST as an Integration Test framework which has this known issue (MicroShed/microshed-testing#17). The proposed solution for MST is to start a pebble container, which would not work for us in a K8s environment.

Potential options:

1. Have tests auto-accept certs or bypass cert verification.

I looked into plugins like "wagon" for this, but could not get this to work. This also may not be a pattern we want to promote.

2. Get a signed cert for the server to use during testing.

OL has an acmeCA-2.0 feature for grabbing certs on behalf of the server. https://openliberty.io/docs/20.0.0.12/reference/feature/acmeCA-2.0.html. We could explore this and use LetsEncrypt as a CA.

There is also a LetsEncrypt Operator (needs to be built and deployed) here: https://www.redhat.com/en/blog/dynamic-ssl-certificates-using-letsencrypt-openshift. With this you can easily edit your HTTPS route to have a cert generated automatically. This may not be the right option since HTTPS should work internally without needing a route.

Since the starters aren't versioned along with the stack (see #32) it will be confusing to see their changes mixed in with the changes that do get included within the stack.

We should move them into a new repo, e.g. application-stack-starters.

The only reason we might hold off is to think through #32 more and make sure we know we want to do this.

Right now we build a stack image with

• mvn repo
• OL install
• inner/outer loop work/output dirs

and we use the stack image both for the inner loop and stage 1 of the outer loop build.

This doesn't seem awful.. but we need to consider if there is an even better way to do things.

E.g. another idea would be to have three images:

• a maven cache
• an inner loop stack image building on this maven cache
• an outer loop stack image with working directories, also building on this maven cache.

How do the pros/cons of having a more fine-grained build measure up against the complexities of managing more releases with more different versions? What if we want to service one without disturbing the other ?

Maybe just a placeholder to think about it.

Also the OL version defines the cache contents: the OL install and all the Maven Liberty features, so it arguably should be part of the tag name.

Once #72 is merged.. we'll have a chmod establishing group write access in our stack image Dockerfile.

I'd think we could have a smaller image and quicker build by setting the umask to provide group write access rather than doing a chmod at the end.

Investigate...

With the plans for devfile/api#239 (perhaps in Devfile v2.1?), we will have a global "variable" in devfile.

This would most obviously be useful for parameterizing the Open Liberty version in the devfile.

Of course, the template with {{.OL_RUNTIME_VERSION}} and build.sh is simple enough, but this would be even cleaner, and a nice example of how to make use of this feature.

• Update devfile
• Update customization doc
• Update application-stack-intro
• Update application-stack-samples

As a developer using a stack-provided Devfile I can customize the run/build maven cmds in a few ways:

E.g. consider the 'id: run' command:

mvn -Dmaven.repo.local=/mvn/repository -Dliberty.runtime.version=20.0.0.9 -Ddebug=false -DhotTests=true -DcompileWait=3 liberty:dev

I could:

1. Edit the original 'run' command
2. Add a second, alternate, 'run-alt' command
3. Extend the devfile by creating a child referencing it as parent, making the changes in the child.

In addition to these it might also be nice to have an extra placeholder in the command like MVN_ARGS here:

mvn -Dmaven.repo.local=/mvn/repository -Dliberty.runtime.version=20.0.0.9 -Ddebug=false -DhotTests=true -DcompileWait=3 $MVN_ARGS liberty:dev

which I could set via:

odo config set --env MVN_ARGS="-Dliberty.var.a1=A1 -Dliberty.var.b1=B1"

and then view with:

odo config view

(Note this would have the side effect that MVN_ARGS becomes an env var of the run container).

Inner loop

m2 container volume links:

• https://github.com/odo-devfiles/registry/blob/4a622f827ed87cd368c7c5819d3ad9b2b6c3c9d5/devfiles/java-quarkus/devfile.yaml
• redhat-developer/odo#3704
• redhat-developer/odo#2510

Outer loop

buildah overlay mounts:  https://www.redhat.com/sysadmin/speeding-container-buildah

Other considerations

1. generic k8s vs Open Shift
2. We need to document the lifecycle of this PV (i.e. what would cause it to be deleted?)