openliberty / ci.common Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 28.0 1.81 MB

Common functionality for ci.maven and ci.gradle

License: Apache License 2.0

Java 99.99% Shell 0.01%

ci.common's People

Contributors

Stargazers

Watchers

ci.common's Issues

suggest to make DOCKER_TIMEOUT configurable

Hi Team,
I'm having issues on using the liberty maven pluign in devc mode.
It uses the ci.common to execut docker commands.
On doing so I always run into timeout issues as the connection from docker for windows with wsl2 enabled is not fast enough to finish the commands within the fixed 20 seconds timeout.
I'd suggest to make this configurable by a env variable or so, that I can override it for my needs.

Best regards
Chris

maven-artifact should be "provided"

Maven artifact should be "provided".
The version is not synchronized among other projects, e.g. ci.maven. I would recommend to extact its version into a property which can then be consumed by ci.maven.

Update variable name to match meaning of docker timeout parameter

dockerTimeout becomes dockerBuildTimeout

Support variables in server.xml include location

server.xml can have an <include location="..."> where it includes another xml file. The location string can have a variable. See https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_setup_includes.html

When installing features from the server.xml and the included xmls, resolve the variables in the location strings.

Handle `onConflict` setting for `include` elements related to application config

When gathering application names and locations from Liberty config files, they are added to a common collection with no regard to the onConflict setting for the <include> element that referenced the Liberty config file. The elements that are gathered from the Liberty config files are:

XPATH_SERVER_APPLICATION = xPath.compile("/server/application");
XPATH_SERVER_WEB_APPLICATION = xPath.compile("/server/webApplication");
XPATH_SERVER_ENTERPRISE_APPLICATION = xPath.compile("/server/enterpriseApplication");

The methods that would need to query the onConflict setting and honor it are ServerConfigDocument.parseIncludeVariables and ServerConfigDocument.parseInclude. Also, the parseInclude method should probably call parseNames on the included file, just as it was called on the original config file in initializeAppsLocation. When merging multiple app elements for the same app, the id attribute is what connects them. Our code does not consider that attribute currently.

Dev mode bug - adding a feature manager block

Adding a feature manager block to the server.xml after dev mode has started results in an error, when dev mode should trigger installing any newly specified features.

Steps to Reproduce

Start dev mode with a server.xml that does not have a featureManager block.
Add a feature manager block, with a feature specified, ie.

    <featureManager>
        <feature>jakartaee-9.0</feature>
    </featureManager>

Notice the error in the dev mode console:

[INFO] Copied file: /Users/kathrynkodama/devex/sampleProjects/vaccination-demo-app/src/main/liberty/config/server.xml to: /private/var/folders/gr/3p093xzj0f9fj9psx80z_zw00000gn/T/tempConfig18101587657383742248/server.xml
[INFO] Parsing the server file /private/var/folders/gr/3p093xzj0f9fj9psx80z_zw00000gn/T/tempConfig18101587657383742248/server.xml for features and includes.
[INFO] Parsing the server file /private/var/folders/gr/3p093xzj0f9fj9psx80z_zw00000gn/T/tempConfig18101587657383742248/configDropins/overrides/liberty-plugin-variable-config.xml for features and includes.
[ERROR] Could not file process changes for /Users/kathrynkodama/devex/sampleProjects/vaccination-demo-app/src/main/liberty/config/server.xml: null

It seems the server.xml is not successfully copied over to the usr/servers directory. The error message is coming from DevUtil and the getServerXmlFeatures method in ServerFeatureUtil

Expected Behaviour
After completing step 2 above, the expected behaviour would be that Liberty installs the corresponding feature. This would match the behaviour of starting Liberty with liberty:run and adding the featureManager block in the usr/server/serverName/server.xml file.

Allow multiple user features for prepare-feature

OpenLiberty/ci.gradle#730

Ignore whitespace within Dockerfile lines for devc

Currently devc parses Dockerfile for content such as RUN configure.sh or RUN features.sh. We should enhance that parsing to handle the cases where there is extra whitespace within such commands, e.g. within DevUtil.getCleanedLines()

Refactor startServer and stopServer into ci.common

Refactor the implementations of starting/stopping server into ci.common instead of in ci.maven and ci.gradle. For ci.maven's case, this would also involve refactoring most of StartDebugMojoSupport into ci.common.

Handle additional conditions for resolving variables

In ServerConfigDocument, there is logic that resolves variables referenced by server.xml for the application location and name. This logic processes all the config files that Liberty looks in, but does not look at system properties or environment variables. The document here (https://openliberty.io/docs/ref/config/) indicates those should be used. In a build environment, perhaps that does not make sense. But we should look into it and ensure this behavior is intentional.

That document also covers additional syntax for resolving variable references that we are not handling. We should add logic for this. Here is an excerpt:

Replace all non-alpha num characters with _

Change all characters to upper case.

If you enter ${my.env.var} in server.xml it will look for environment variables with the following names:

my.env.var

my_env_var

MY_ENV_VAR

Set `load-external-dtd` to false when loading xml files

Check for places that load xml files and set the load-external-dtd flag to false.

One placed I identified already: XmlDocument.createDocument(File)

DocumentBuilderFactory f = DocumentBuilderFactory.newInstance();
f.setFeature("http://apache.org/xml/features/nonvalidating/load-dtd-grammar", false); 
f.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false);

Open Liberty server does not start using devmode when messages.log format is JSON

I set the following property to set messages.log to JSON:
com.ibm.ws.logging.console.format=json

and I get the following error when I start the server:
Could not parse the host name from the log message: {"type":"liberty_message","host":"localhost","ibm_userDir":"\/Users\/kartiksaravana\/workspace\/logsinsight\/target\/liberty\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"CWWKT0016I: Web application available (default_host): http:\/\/192.168.0.10:9080\/logsinsight\/","ibm_threadId":"0000003b","ibm_datetime":"2020-06-29T16:46:48.037-0400","ibm_messageId":"CWWKT0016I","module":"com.ibm.ws.http.internal.VirtualHostImpl","loglevel":"AUDIT","ibm_sequence":"1593463608037_000000000000C"}

I tried to make some changes to fix the issue:
https://github.com/k-rtik/ci.common/tree/json-message

Localize code that generates docker build context

In dev mode for containers we call docker build with a build context. We should determine the context in one place to avoid mistakes if we ever decide to change the build context. We could write a method to do this but for now we can just determine the context in one method and pass it to other methods as needed.

Refactor logging into a common logger interface

Pull logging from DevUtil and InstallFeatureUtil out into a common logger interface, implement it as a singleton in both Maven and Gradle, and pass that singleton into DevUtil and InstallFeatureUtil.

https://github.com/WASdev/ci.common/blob/devMode/src/main/java/net/wasdev/wlp/common/plugins/util/DevUtil.java#L74-L115

https://github.com/WASdev/ci.common/blob/devMode/src/main/java/net/wasdev/wlp/common/plugins/util/InstallFeatureUtil.java#L104-L133

Remove JSON dependencies from pom

Two dependencies containing 'javax.json'

Dev mode prints message repeatedly with debug logging

When you use the -X option you see this message printed over and over in the console:

[DEBUG] (org.codehaus.mojo.pluginsupport.ant.AntHelper) CWWKM2014I: Now looking for CWWKZ0003I: in messages.log.

Update Warning message when there is a port conflict using container mode

When port 9080 is used, dev mode with containers uses port 9081 instead and prints a warning message "The Liberty container is using non-default host ports." We need to add words to say this is done to avoid an error.

productInfo - command timed out

task 'installFeature' does not work on the win10 machine (it is working fine on Linux machine).

Condition:

Gradle project
buildscript net.wasdev.wlp.gradle.plugins:liberty-gradle-plugin:2.7
dependencies
libertyRuntime group: 'com.ibm.websphere.appserver.runtime', name: 'wlp-javaee8', version: '19.0.0.9'
libertyFeature 'io.openliberty.features:mpReactiveMessaging-1.0:19.0.0.9'
installLiberty.finalizedBy 'installFeature'

During the productInfoValidate() the plugin throws "productInfo command timed out "
The feature is actually installed but the validation fails. If I run manually bin\productInfo.bat the validation works fine.

ArrayIndexOutOfBoundException with latest ci.common-1.2-20180719.194151-6 snapshot

Hit this with recent JDKs at V8 from IBM and Oracle with latest snapshot... when building WASdev/ci.maven. Problem went away reverting to most recent, earlier snapshot

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.2:descriptor (default-descriptor) on project liberty-maven-plugin: Execution default-descriptor of goal org.apache.maven.plugins:maven-plugin-plugin:3.2:descriptor failed: 55297 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:3.2:descriptor (default-descriptor) on project liberty-maven-plugin: Execution default-descriptor of goal org.apache.maven.plugins:maven-plugin-plugin:3.2:descriptor failed: 55297
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:191)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default-descriptor of goal org.apache.maven.plugins:maven-plugin-plugin:3.2:descriptor failed: 55297
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
        ... 20 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 55297
        at org.objectweb.asm.ClassReader.readClass(Unknown Source)
        at org.objectweb.asm.ClassReader.accept(Unknown Source)
        at org.objectweb.asm.ClassReader.accept(Unknown Source)
        at org.apache.maven.tools.plugin.annotations.scanner.DefaultMojoAnnotationsScanner.scanFile(DefaultMojoAnnotationsScanner.java:139)
        at org.apache.maven.tools.plugin.annotations.scanner.DefaultMojoAnnotationsScanner.scan(DefaultMojoAnnotationsScanner.java:85)
        at org.apache.maven.tools.plugin.annotations.JavaAnnotationsMojoDescriptorExtractor.scanAnnotations(JavaAnnotationsMojoDescriptorExtractor.java:125)
        at org.apache.maven.tools.plugin.annotations.JavaAnnotationsMojoDescriptorExtractor.execute(JavaAnnotationsMojoDescriptorExtractor.java:104)
        at org.apache.maven.tools.plugin.scanner.DefaultMojoScanner.populatePluginDescriptor(DefaultMojoScanner.java:108)
        at org.apache.maven.plugin.plugin.AbstractGeneratorMojo.execute(AbstractGeneratorMojo.java:233)
        at org.apache.maven.plugin.plugin.DescriptorGeneratorMojo.execute(DescriptorGeneratorMojo.java:92)
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        ... 21 more

Container support requires Docker version 18.03.0 or higher

Dev mode currently reports the version number required in an invalid format: "18.03.00". To avoid confusion with the version numbers reported on the Docker web site we should update our plugin to use the correct format.

open liberty server does not start with "mvn liberty:dev" when http port is disabled

With my configuration

		httpsPort="9080" id="defaultHttpEndpoint" />

open liberty server does not start with "mvn liberty:dev" when http port is disabled. Or the server starts but is stopped immediately with the following message.

[INFO] (org.codehaus.mojo.pluginsupport.ant.AntHelper) [AUDIT   ] CWWKT0016I: Web application available (default_host): https://localhost:9443/mydomain/
[INFO] (org.codehaus.mojo.pluginsupport.ant.AntHelper) [AUDIT   ] CWWKZ0001I: Application mydomainEAR.ear started in 6.297 seconds.
[INFO] (org.codehaus.mojo.pluginsupport.ant.AntHelper) [AUDIT   ] CWWKF0012I: The server installed the following features: [appClientSupport-1.0, appSecurity-2.0, appSecurity-3.0, batch-1.0, beanValidation-2.0, cdi-2.0, concurrent-1.0, distributedMap-1.0, ejb-3.2, ejbHome-3.2, ejbLite-3.2, ejbPersistentTimer-3.2, ejbRemote-3.2, el-3.0, j2eeManagement-1.1, jacc-1.5, jaspic-1.1, javaMail-1.6, javaee-8.0, jaxb-2.2, jaxrs-2.1, jaxrsClient-2.1, jaxws-2.2, jca-1.7, jcaInboundSecurity-1.0, jdbc-4.2, jms-2.0, jndi-1.0, jpa-2.2, jpaContainer-2.2, jsf-2.3, jsonb-1.0, jsonp-1.1, jsp-2.3, managedBeans-1.0, mdb-3.2, servlet-4.0, ssl-1.0, wasJmsClient-2.0, wasJmsSecurity-1.0, wasJmsServer-1.0, webProfile-8.0, websocket-1.1].
[INFO] (org.codehaus.mojo.pluginsupport.ant.AntHelper) [AUDIT   ] CWWKF0011I: The mydomainEARServer server is ready to run a smarter planet. The mydomainEARServer server started in 14.743 seconds.
[DEBUG] (org.codehaus.mojo.pluginsupport.ant.AntHelper) CWWKM2014I: Now looking for CWWKF0011I: in messages.log.
[INFO] (org.codehaus.mojo.pluginsupport.ant.AntHelper) CWWKM2015I: Match number: 1 is [11/28/19 14:54:02:527 CET] 00000021 com.ibm.ws.kernel.feature.internal.FeatureManager            A CWWKF0011I: The mydomainEARServer server is ready to run a smarter planet. The mydomainEARServer server started in 14.743 seconds..
[DEBUG] (org.codehaus.mojo.pluginsupport.ant.AntHelper) CWWKM2014I: Now looking for CWWKO0221E: in messages.log.
[DEBUG] (org.codehaus.mojo.pluginsupport.ant.AntHelper) CWWKM2014I: Now looking for CWWKT0016I: in messages.log.
[DEBUG] (org.codehaus.mojo.pluginsupport.ant.AntHelper) CWWKM2015I: Match number: 1 is [11/28/19 14:54:02:463 CET] 00000031 com.ibm.ws.http.internal.VirtualHostImpl                     A CWWKT0016I: Web application available (default_host): https://localhost:9443/mydomain/.
[DEBUG] Web app available message: [11/28/19 14:54:02:463 CET] 00000031 com.ibm.ws.http.internal.VirtualHostImpl                     A CWWKT0016I: Web application available (default_host): https://localhost:9443/mydomain/
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  17.708 s
[INFO] Finished at: 2019-11-28T14:54:02+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.openliberty.tools:liberty-maven-plugin:3.1:dev (default-cli) on project mydomainEAR: Could not parse the host name from the log message: [11/28/19 14:54:02:463 CET] 00000031 com.ibm.ws.http.internal.VirtualHostImpl                     A CWWKT0016I: Web application available (default_host): https://localhost:9443/mydomain/ -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal io.openliberty.tools:liberty-maven-plugin:3.1:dev (default-cli) on project mydomainEAR: Could not parse the host name from the log message: [11/28/19 14:54:02:463 CET] 00000031 com.ibm.ws.http.internal.VirtualHostImpl                     A CWWKT0016I: Web application available (default_host): https://localhost:9443/mydomain/

and the Stacktrace where this comes from

Caused by: io.openliberty.tools.common.plugins.util.PluginExecutionException: Could not parse the host name from the log message: [11/28/19 14:54:02:463 CET] 00000031 com.ibm.ws.http.internal.VirtualHostImpl                     A CWWKT0016I: Web application available (default_host): https://localhost:9443/mydomain/
    at io.openliberty.tools.common.plugins.util.DevUtil.parseHostName (DevUtil.java:516)
    at io.openliberty.tools.common.plugins.util.DevUtil.parseHostNameAndPorts (DevUtil.java:504)
    at io.openliberty.tools.common.plugins.util.DevUtil.startServer (DevUtil.java:493)

It looks like the messages are not interpreted in the right way. And if I start with mvn liberty:run there us no problem... everything works as expected in NON DEV mode.

I tried to use 1.8-SNAPSHOT of.common along with liberty-maven-plugin version 3.2-SNAPSHOT and did the following change in DevUtil class

    protected int parseHostName(String webAppMessage) throws PluginExecutionException {
        int httpPrefixIndex = webAppMessage.indexOf(HTTP_PREFIX);
        int httpsPrefixIndex = webAppMessage.indexOf("https://");
        if (httpPrefixIndex < 0 && httpsPrefixIndex < 0) {
            throw new PluginExecutionException("Could not parse the host name from the log message: " + webAppMessage);
        }
        int hostNameIndex = httpPrefixIndex + HTTP_PREFIX.length();
        int portPrefixIndex = webAppMessage.indexOf(":", hostNameIndex);
        if (portPrefixIndex < 0) {
            throw new PluginExecutionException("Could not parse the http port number from the log message: " + webAppMessage);
        }
        hostName = webAppMessage.substring(hostNameIndex, portPrefixIndex);
        debug("Parsed host name: " + hostName);
        return portPrefixIndex;
    }

So if the error only pops up if https:// is also not found. I tried this and mvn liberty:dev starts the server just like before if both ports (http and https) are enabled.

openliberty / ci.common Goto Github PK

ci.common's People

Contributors

Stargazers

Watchers

Forkers

ci.common's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs