xebialabs-community / xlr-xldeploy-plugin

The xlr-xldeploy-plugin is a XL Release plugin that allows to start a control task, start a deployment, migrate a deployment package to another server or get the latest version of an application on XL Deploy.

XL Release - XL Deploy interface

Preface

This document describes the functionality provided by the xlr-xldeploy-plugin.

See the XL Release Documentation for background information on XL Release and release concepts.

Overview

The xlr-xldeploy-plugin is a XL Release plugin that allows to start a control task, start a deployment, migrate a deployment package to another server or get the latest version of an application on XL Deploy.

Installation

Place the latest released version under the plugins dir for XL Release 7.x or under the plugins/__local__ for XL Release 8+. If needed append the following to the script.policy under conf:

permission java.io.FilePermission "plugins/*", "read";
permission java.io.FilePermission "conf/logback.xml", "read";

• This plugin (2.3.x+) requires XLR 5.x+
• When using XL Release 7.x:
  • v2.9.1 is compatible with XLR 7.x. Use this version if you don't want to go experimental.
  • v3.x will only work with XLR 7.x+ and has breaking changes. This means shared configurations and tasks will need to be reconfigured. When an upgrader has been written this will also be mentioned here.
  • v3.x is dependent upon the out-of-box xlr-xld-plugin-7.x plugin bundled with the v7.x server. v2.9.1 and lower are independent.
  • If you're using a v7.x of the plugin, these are deprecated and should be replaced with a v3.x version.

Types

• ControlTask (compatible with XL Deploy 4.5.2 and up)

  • ciId
  • controlTaskName
  • parameters
  • continueIfStepFails (Will try to continue if a step in a control task fails)
  • numberOfContinueRetrials (Number of times to retry a step)
  • pollingInterval
  • numberOfPollingTrials

• DeployTask (compatible with XL Deploy 4.5.1 and up)

  • deploymentPackage (ID of the deployment package to deploy e.g.: Applications/XL Release/XLR/1.0)
  • environment (ID of the environment to deploy to e.g.: Environments/Xl Release/XL Release)
  • orchestrators (Comma separated list of orchestrators to be used: parallel-by-deployment-group, parallel-by-container)
  • deployedApplicationProperties (Dictionary containing all the deployed application properties to be set (except orchestrators). e.g.: {"maxContainersInParallel": "2"})
  • overrideDeployedProps (Map contain xpath expression to override deployed properties.) e.g.: {'deployeds/openshift.ResourceModule[@id="Infrastructure/Server/Openshift/project/myapp"]/placeholders/entry[@key="openshift.placeholder"]': 'value'}
  • deployedProperties DEPRECATED - (use overrideDeployedProps)(Dictionary containing all the properties to be set. Remark: Each key is an xlrTag in the deployeds - See also https://github.com/xebialabs-community/xld-xlrelease-plugin, e.g.: {"Gate1": "{'taskId':'1234567890'}"})
  • continueIfStepFails (Will try to continue if a step in the deployment task fails)
  • numberOfContinueRetrials (Number of times to retry a step)
  • rollbackOnError (Whether rollback should be done if the deployment fails)
  • pollingInterval (Number of seconds to wait before polling the task status)
  • numberOfPollingTrials (Number of times to poll for task status)
  • failOnPause (If checked task will fail if the deployment enters a STOPPED state, for example if the xld-pause-plugin is in use. Set to True by default for backwards compatibility)

• UndeployTask (compatible with XL Deploy 4.5.1 and up)

  • deployedApplication - Name of the deployed application you want to undeploy (Only the name, without Environments, etc...)
  • environment (ID of the environment to deploy to e.g.: Environments/Xl Release/XL Release)
  • orchestrators (Comma separated list of orchestrators to be used: parallel-by-deployment-group, parallel-by-container)
  • deployedApplicationProperties (Dictionary containing all the deployed application properties to be set (except orchestrators). e.g.: {"maxContainersInParallel": "2"})
  • continueIfStepFails (Will try to continue if a step in the deployment task fails)
  • numberOfContinueRetrials (Number of times to retry a step)
  • rollbackOnError (Whether rollback should be done if the deployment fails)
  • pollingInterval (Number of seconds to wait before polling the task status)
  • numberOfPollingTrials (Number of times to poll for task status)
  • failOnPause (If checked task will fail if the deployment enters a STOPPED state, for example if the xld-pause-plugin is in use. Set to True by default for backwards compatibility)

• Migrate Package (compatible with XL Deploy 6.0.0 and up)

  • server - Server to pull a package from
  • username - Override source username
  • password - Override source password
  • destinationServer - Server to pull package to
  • destinationUsername - Override destination username
  • destinationPassword - Override destination password
  • deploymentPackage - ID of the package to migrate (starting with Applications/)
  • autoCreatePath - If set the task will automatically create the path and application if it doesn't exist in the destination
  • idempotent - If True and deploymentPackage exists on destinationServer, it will be overwritten. If false, it will error.

• Import Package

  • server - Server to import a package to
  • username - Override username
  • password - Override password
  • repositoryUrl - Url of the package
  • repositoryUsername - Optional username for the repository
  • repositoryPassword - Optional password for the repository

• Get CI

  • server - Server to query
  • username - Override username
  • password - Override password
  • ciID - ID of the Configuration Item wanted
  • accept - Desired format of the return: XML or JSON
  • throwOnFail - Boolean - If True, a False return will Fail the task rather than just returning False.
  • response - Return value of the CI described in the format requested

• Get All Versions

  • server - Server to query
  • username - Override username
  • password - Override password
  • applicationId - ID of the application to query for all package versions
  • packageId - Return value with the all package IDs of the application (NOTE: this needs to be a variable of the list type, and needs to be defined beforehand)

• Get Latest Version

  • server - Server to query
  • username - Override username
  • password - Override password
  • applicationId - ID of the application to query for latest package version
  • stripApplications - Whether to strip "Applications/" from the beginning of the returned package ID
  • packageId - Return value with the latest package ID

• Get Last Version Deployed

  • server - Server to query
  • username - Override username
  • password - Override password
  • environmentId - ID of the environment to check the application version on
  • applicationName - Name of the applicaiton in the environment to get the current version of returned package ID
  • applicationId - Return value with the current application ID

• Does CI exist

  • server - Server to query
  • username - Override username
  • password - Override password
  • ciID - Repo path to CI to check for existence, e.g. Infrastructure/myHost
  • throwOnFail - Boolean - If True, a False return will Fail the task rather than just returning False.

• Create CI

  • server - Server to query
  • username - Override username
  • password - Override password
  • ciID - Repo path to CI to create, e.g. Infrastructure/myHost
  • ciType - Type of CI to create, e.g. overthere.CifsHost
  • xmlDescriptor - XML with the fields to set on the CI, e.g. <os>WINDOWS</os><connectionType>WINRM_NATIVE</connectionType><address>${address}</address><username>${username}</username><password>${password}</password>
  • or if you use Credential Objects:  <os>WINDOWS</os><connectionType>WINRM_NATIVE</connectionType><address>${address}</address><credential ref="Configuration/MyCredential"/>
  • addToEnvironment - Switch to decide if to add the CI to an environment
  • envID - Environment to add the CI to.

• Create Folder (Tree)

  • server - Server to query
  • username - Override username
  • password - Override password
  • folder_id - Path of folder structure you wish to create without top level type, e.g. TeamA/DEV/Application
  • folder_type - Top level type under which the folder should be created e.g. Applications

• Delete CI

  • server - Server to query
  • username - Override username
  • password - Override password
  • ciID - ID of the Configuration Item you want to delete
  • envID - If provided, will remove the CI from an Environment first
  • throwOnFail - Boolean - If True, the Task will fail if the Deletion does not succeed.

• Delete Infrastructure - This task will delete the specified Infrastructure CI, AND will also undeploy any deployed applications on the infrastrucutre as well as remove the infrastructure from any environments where it is referenced as a container.

  • server - Server to query
  • username - Override username
  • password - Override password
  • ci_id - Fully qualified id from the Infrastructure CI to delete

• CLI Config (Global Configuration)

  • CLI Home - Home directory where XL Deploy CLI is installed
  • XLD Host - Host the CLI should connect to DEFAULT will work if on the same server as XL Deploy
  • XLD Port - Port for XL Deploy server. DEFAULT will work if using the default XL Deploy port
  • XLD Secure - Specify secure connection for XL Deploy. If true, Port will default to 4517
  • XLD Context - XLD CLI context. If no context is needed then DEFAULT will be fine
  • XLD Proxy Host - Proxy host if needed.
  • XLD Proxy Port - Proxy Port if needed.
  • XLD Socket timeout - Connection timeout to XL Deploy
  • XLD User Name - User name to connect to XL Deploy
  • XLD Password - Password to connect to XL Deploy

  

• CLI

  • script - CLI Script to execute

  

• CLI URL

  • scriptURL - URL to CLI Script to execute

  

• Get CI Tags

  • Server: XL Deploy Server CI
  • Username
  • Password
  • XLD CI: The XL Deploy CI to get the tags from

  Outputs

  • XLD Tags: List of tags returned from the XL Deploy server

  

• Add CI Tag

  • Server: XL Deploy Server CI
  • Username
  • Password
  • XLD CI: The XL Deploy CI to get the tags from
  • XLD New Tag: New tag to add to the list of tags in the XL Deploy CI

  Outputs

  • XLD Tags: List of tags returned from the XL Deploy server

• Set CI Tags

  • Server: XL Deploy Server CI
  • Username
  • Password
  • XLD CI: The XL Deploy CI to get the tags from
  • XLD New Tags: New list of tags to add to the list of tags in the XL Deploy CI

  Outputs

  • XLD Tags: List of tags returned from the XL Deploy server

• Create Environment
  • Server: XL Deploy Server CI
  • Username
  • Password
  • CI ID: The XL Deploy Environment CI to be created
  • Members: List of Infrastructure CIs to be included
  • Dictionaries: List of Dictionaries to be included

• Create Dictionary
  • Server: XL Deploy Server CI
  • Username
  • Password
  • CI ID: The XL Deploy Dictionary CI to be created
  • Entries: Key-Value pairs for the dictionary items (map_string_string)

• Add Entry to Dictionary
  • Server: XL Deploy Server CI
  • Username
  • Password
  • Dictionary ID: The XL Deploy Dictionary CI to be created
  • Key: Key for the dictionary item
  • Value: Value for the dictionary item

If the key is already present, its value will be updated.

• Add Entries to Dictionary
  • Server: XL Deploy Server CI
  • Username
  • Password
  • CI ID: The XL Deploy Dictionary CI to be created
  • New Entries: Key-Value pairs for the dictionary items (map_string_string)

If a key is already present, its value will be updated.

• Create Simple CI
  • Server: XL Deploy Server CI
  • Username
  • Password
  • CI ID: The XL Deploy CI to be created
  • Properties: Key-Value pairs for the CI properties

This task supports only simple string properties.

Tiles

• Deployed applications on an environment at a specific moment in time

  

xlr-xldeploy-plugin's Issues

Bug: deployed_application_properties not working correctly

I am attempting to use XPath to modify XLD deployed placeholders using the XLR plugin. I am unable to do so due to a bug. After debugging I believe this is wrong:

https://github.com/xebialabs-community/xlr-xldeploy-plugin/blob/master/src/main/resources/xldeploy/XLDeployClient.py

			if not pkey_xml:
				application = root.find("application")
				for child in application:
					# print "DEBUG: Going to add key: %s" % key
					# print "DEBUG: Searching for deployed application: %s" % child
					pkey_xml = ET.SubElement(child, key)

I suspect

if not pkey_xml:

in Python 2.7 is being translated to

if not len(pkey_xml)

Where len(pkey_xml) is always 0. Hence the if statement always results in 'true'. By putting a print statement of both the pkey_xml and within that if block I have found that the if block is always entered even if pkey_xml is not null.

This should instead be:

if pkey_xml is None:

Lines 146, 169

show the response when creating an CI

When we are creating an CI, the task does not show any response. It would be nice to show the response.

Deploy Task enters endless loop

XL Deploy 5.5.5, XL Release 6.1, latest version of xlr-xldeploy-plugin

One of my teams has managed to get themselves into a situation where their DAR package does not undeploy properly. Whilst the Undeploy task completes, the repository is left with a few cmd2.DeployedCommand objects.
That means that when I try to run a Deploy, I end up in an endless loop.

To replicate:

1. Deploy any application in XL Deploy
2. Delete the udm.DeployedApplication from the Environment using the repository
3. Attempt to deploy that application through XL Deploy - it refuses
4. Attempt to deploy that application through the XL Deploy Plugin Task in XL Release... endless loop.

Judging by the message, it's treating the 2 parts of the error message as Task ID and State. XL Deploy clearly does some form of extra check via the GUI that is not run for the XL Release Plugin Task. I suspect that since the plugin is waiting for a specific return in the State field (COMPLETE,FAILED) etc then the fact that the return is not within the expected selection would cause this loop even though the return is completely outside the expected parameters.

Interestingly the XL Deploy "Deploy" Task provided with XL Release does not have this issue

The message below is what is returned to my XL Release Task.

Creating initial deploy
Mapping all deployables
Validating the deployment
The deployment specification is valid Creating a deployment task
Execute task with id: Repository entity Infrastructure/SERVER/SCRIPT already exists Task [Repository entity Infrastructure/SERVER/SCRIPT already exists] now in state [Y003210: Could not find resource for full path: https://xldeploy.mydomain.com:8080/deployit/task/Repository%20entity%20Infrastructure/SERVER/SCRIPT%20already%20exist]

Control Task Doesn't handle empty parameters

Hey There,

We have some control tasks that have parameters, but they are required=false and most times are empty. This seems to break inside the prepare_control_task function, as it fails with:
SyntaxError: no viable alternative at input '<EOF>'
The prepared control task xml has this in it:

<parameters> <wls.Domain_checkwldepstatus id="parameters"> <deploymentsToCheck/> </wls.Domain_checkwldepstatus> </parameters>

It seems it should ignore the parameters section if there are no values there. Am I missing anything?

It works fine in XLD if you supply no parameters.

Thanks!
Bernie

Migrate Task does not Fail when it fails

Version 2.1.5

We had 2 failures today, one was access denied bad password, an another where the package already exists. Both legit errors but the Migrate task did not fail it juts moved on.

Have not looked at the code yet, just wanted to get this logged.

Select Server CI

When adding a task "Latest Version" or "Deploy" the dropdown box for selecting the Server CI is empty.
XLR 4.7.1
xlr-xldeploy-plugin 1.1.5

when creating CI you have to put a space in xml description

Hi,

When trying to create a CI, you have to put a space in xml description and then save it. Otherwise, it will not work.
When CI ID and Ci Type is chosen, I do not want to touch the XML Descriptor.
See the attachment for the page.

We are using version XLR 5.0.0 and XLD 5.5.1.

Setting Polling interval on Deploy Task lower than default (10) causes failure

User reduced Polling interval on Deploy Task to 2, which caused a failure because it created a state that is not accounted for in deploytask.py code - the taskStates get skipped (DONE, EXECUTED, FAILED, STOPPED) and it jumps to sys.exit(1). Restoring polling to 10 gets us around the issue.

Update CI Task

Inside the function I have found a difficulty and I am trying to test to see if still exists in latest build but the one we have in production has rolled on this very item.

On the CI for the application we have satisifiesTicketNumber. In XLR when we go to update that field if it does not exist it gives a key error. If I go to XLD and enter some data, then in xlr it works. So if the field does not exist it will not add a new field it tries to update the old field but in the return it does not exist.
for example
http://XXXXX:4516/deployit/repository/ci/Applications/DPY/Plugins/uhg-xlr-svn-migration-plugin/1.1.2
returns
<udm.DeploymentPackage id="Applications/DPY/Plugins/uhg-xlr-svn-migration-plugin/1.1.2" token="26e12cbc-1ed9-4a22-b92d-c89aa997ec1f" created-by="dployeng" created-at="2017-03-29T10:14:01.752-0500" last-modified-by="kmcallo" last-modified-at="2017-08-23T11:11:06.659-0500">


false
false
false




LATEST
false
</udm.DeploymentPackage>

So in XLR we get Exception during execution: KeyError: u'satisfiesTicketNumber' in <script> at line number 16

But if I go to XLD and add this data, then it works because the return from the first call is
<udm.DeploymentPackage id="Applications/DPY/Plugins/uhg-xlr-svn-migration-plugin/1.1.2" token="26e12cbc-1ed9-4a22-b92d-c89aa997ec1f" created-by="dployeng" created-at="2017-03-29T10:14:01.752-0500" last-modified-by="kmcallo" last-modified-at="2017-08-23T11:11:06.659-0500">


false
false
false
CHG123




LATEST
false
</udm.DeploymentPackage>

I have went so far right now to identify
inside XLDeployClient.py

There is this
def update_ci_property(self, ci_id, ci_property, property_value):
363 if self.check_ci_exist(ci_id):
364 ci = self.get_ci(ci_id, 'json')
365 data = json.loads(ci)
366 if isinstance(data[ci_property], list):
367 data[ci_property] = eval(property_value)
368 else:
369 data[ci_property] = property_value
370 self.update_ci(ci_id, json.dumps(data), 'json')
371 else:
372 raise Exception("Did not find ci with id [%s]" % ci_id)

I see in access logs. These two run
if self.check_ci_exist(ci_id):
364 ci = self.get_ci(ci_id, 'json')

But I am not sure when you introduced the isinstance... and if it works.

Deployment task fails with UnicodeEncodeError

UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 10821: ordinal not in range(128)

CLI Operations don't work anymore with XLR 7.0.0

CLI files do not work anymore after upgrading to XLR 7.0.0 and plugin version 3.1.2.
The error that is given is:

Exception while loading script: java.io.FileNotFoundException: Script not found:
xldeploy/cliFile.py

I assume this is a result of the directory structure change.
It might be as simple as just adding the script location, or maybe changing the name of the task in the synthetic, but I am not sure what the preferred way would be.
I will try a "dirty" fix for now, and will update how that works.

7.0.2 Issue with XLD 6.0.2

Appears that deployment goes through looking at deployment report in XLD but the last entry is

Display the step logs Exception during execution: Traceback (most recent call last): File "<script>", line 48, in File "pyclasspath/xlr_xldeploy/XLDeployClient.py", line 441, in display_step_logs File "/app_1/xlr-7.0.2/lib/jython-standalone-2.7.0.jar/Lib/xml/etree/ElementTree.py", line 1303, in XML File "/app_1/xlr-7.0.2/lib/jython-standalone-2.7.0.jar/Lib/xml/etree/ElementTree.py", line 1656, in close File "/app_1/xlr-7.0.2/lib/jython-standalone-2.7.0.jar/Lib/xml/etree/ElementTree.py", line 1508, in _raiseerror File "", line None xml.etree.ElementTree.ParseError: Character reference "&#x1b" is an invalid XML character.

cli.py is printing password in case of system exit 1

Hi,

When the system exit code is 1, the err is being printed out. This also prints the password. See https://github.com/xebialabs-community/xlr-xldeploy-plugin/blob/master/src/main/resources/xlr_xldeploy/cli.py#L30

In my example, the username is tutti and the password is frutti. See below for an example output:

[com.xebialabs.overthere.RuntimeIOException: Cannot start command [/tmp/bin/cli.sh -quiet -username admin -password **** -source /appl/xlrelease/scripts/add_application.py -- "" /appl/xlrelease/scripts/xldeploy_config_passwords.json tutti frutti] on [local:] at com.xebialabs.overthere.local.LocalConnection.startProcess(LocalConnection.java:144) at

How we can suppress the password in the output?

The icons showed for the CLI tasks are incorrect

Control Task Doesn't seem to work with parameters

Hi there,

I am having trouble launching a control task with parameters via the built in task. I have 2 parameters defined in the synthetic of my type in xld. I tried passing as a dictionary in the task in the parameters field:
{'webInstances': 1, 'queueInstances': 1}, but got this Exception during execution:
TypeError: cannot serialize 1 (type int) in <script> at line number 14. Do I have to change the kind in the synth? I have another control task where the values for the parameters are all defaulted in the synth but I couldn't get that to work either. In that case I didn't put anything in the parameters field in the XLR task and it failed with Exception during execution: SyntaxError: no viable alternative at input '' in at line number 21 at column number 11.

If this is not an issue, can you give me some advice on where my configuration is wrong.

Thanks I really appreciate any info!
Bernie

Duplication of Delete CI and Delete Infrastructure

We now have Delete CI and Delete Infrastructure.
Delete CI just amends the XL Deploy Repository.
Delete Infrastructure removes all Deployables - essentially reverting all XLD Changes on the hardware.

I like the principle behind this second task (even though I can't merge the changes into my repo!) as it makes sure it cleans up after itself.

However, it does mean we have two Delete tasks with very discrete differences which will not be apparent to everyone - which is always risky. Do we think we should:

1. Keep the two actions completely separate (Repository / Underlying objects) - in which case we need to make sure the label and description of the task is spot on. Perhaps rename Delete CI to Delete CI from Repository?

2. Keep one task type "Delete CI" but improve it so that it can perform in the same way as Delete Infrastructure (i.e. clean up after itself). We could then offer a boolean for "Delete from Repository Only" to maintain the same compatibility that we currently have.

Create CI task fails to add to an empty environment

See also zendesk community

Error : UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1110: ordinal not in range(128) in <script> at line number 28' on task 'xld_deploy'

hello,

I always get the error each time the XL Deploy response contains a character such as é or à :
javax.script.ScriptException: UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1110: ordinal not in range(128) in <script> at line number 28
at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:222)
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:59)
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
at com.xebialabs.xlrelease.script.Jsr223ScriptExecutor.lambda$evalScriptPrivileged$186(Jsr223ScriptExecutor.java:60)
at java.security.AccessController.doPrivileged(Native Method)
at com.xebialabs.xlrelease.script.Jsr223ScriptExecutor.evalScriptPrivileged(Jsr223ScriptExecutor.java:60)
at com.xebialabs.xlrelease.script.jython.JythonScriptExecutor.evalScriptWithUtilities(JythonScriptExecutor.java:60)
at com.xebialabs.xlrelease.script.jython.JythonScriptExecutor.evalScriptWithApi(JythonScriptExecutor.java:49)
at com.xebialabs.xlrelease.script.DefaultScriptService.executeScript(DefaultScriptService.java:557)
at com.xebialabs.xlrelease.script.DefaultScriptService.lambda$executeCustomScriptTask$181(DefaultScriptService.java:203)
at com.xebialabs.deployit.plumbing.scheduler.Scheduler.lambda$withLoggedExceptions$0(Scheduler.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.python.core.PyException: null
at org.python.core.codecs.strict_errors(codecs.java:208)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.python.core.JavaFunc.call(Py.java:3029)
at org.python.core.PyObject.call(PyObject.java:450)
at org.python.core.codecs.encoding_error(codecs.java:1538)
at org.python.core.codecs.PyUnicode_EncodeIntLimited(codecs.java:1211)
at org.python.core.codecs.PyUnicode_EncodeASCII(codecs.java:1170)
at org.python.core.codecs.encode(codecs.java:165)
at org.python.core.PyString.encode(PyString.java:3981)
at org.python.core.PyString.str_encode(PyString.java:3989)
at org.python.core.PyUnicode.unicode_encode(PyUnicode.java:1851)
at org.python.core.PyUnicode$unicode_encode_exposer.call(Unknown Source)
at org.python.core.PyObject.call(PyObject.java:480)
at org.python.core.PyObject.call(PyObject.java:484)
at xml.parsers.expat$py.Parse$14(C:/product/xl-release-7.5.0-server/lib/jython-standalone-2.7.1.patch2609.jar/Lib/xml/parsers/expat.py:213)
at xml.parsers.expat$py.call_function(C:/product/xl-release-7.5.0-server/lib/jython-standalone-2.7.1.patch2609.jar/Lib/xml/parsers/expat.py)
at org.python.core.PyTableCode.call(PyTableCode.java:171)
at org.python.core.PyBaseCode.call(PyBaseCode.java:171)
at org.python.core.PyFunction.call(PyFunction.java:434)
at org.python.core.PyMethod.call(PyMethod.java:156)
at xml.etree.ElementTree$py.feed$109(C:/product/xl-release-7.5.0-server/lib/jython-standalone-2.7.1.patch2609.jar/Lib/xml/etree/ElementTree.py:1655)
at xml.etree.ElementTree$py.call_function(C:/product/xl-release-7.5.0-server/lib/jython-standalone-2.7.1.patch2609.jar/Lib/xml/etree/ElementTree.py)
at org.python.core.PyTableCode.call(PyTableCode.java:171)
at org.python.core.PyBaseCode.call(PyBaseCode.java:154)
at org.python.core.PyFunction.call(PyFunction.java:423)
at org.python.core.PyMethod.call(PyMethod.java:141)
at xml.etree.ElementTree$py.XML$86(C:/product/xl-release-7.5.0-server/lib/jython-standalone-2.7.1.patch2609.jar/Lib/xml/etree/ElementTree.py:1314)
at xml.etree.ElementTree$py.call_function(C:/product/xl-release-7.5.0-server/lib/jython-standalone-2.7.1.patch2609.jar/Lib/xml/etree/ElementTree.py)
at org.python.core.PyTableCode.call(PyTableCode.java:171)
at org.python.core.PyBaseCode.call(PyBaseCode.java:308)
at org.python.core.PyBaseCode.call(PyBaseCode.java:132)
at org.python.core.PyFunction.call(PyFunction.java:413)
at xlr_xldeploy.XLDeployClient$py.add_orchestrators$8(pyclasspath/xlr_xldeploy/XLDeployClient.py:100)
at xlr_xldeploy.XLDeployClient$py.call_function(pyclasspath/xlr_xldeploy/XLDeployClient.py)
at org.python.core.PyTableCode.call(PyTableCode.java:171)
at org.python.core.PyBaseCode.call(PyBaseCode.java:154)
at org.python.core.PyFunction.call(PyFunction.java:423)
at xlr_xldeploy.XLDeployClient$py.deployment_prepare_deployeds$22(pyclasspath/xlr_xldeploy/XLDeployClient.py:250)
at xlr_xldeploy.XLDeployClient$py.call_function(pyclasspath/xlr_xldeploy/XLDeployClient.py)
at org.python.core.PyTableCode.call(PyTableCode.java:171)
at org.python.core.PyBaseCode.call(PyBaseCode.java:308)
at org.python.core.PyBaseCode.call(PyBaseCode.java:199)
at org.python.core.PyFunction.call(PyFunction.java:482)
at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:237)
at org.python.core.PyMethod.call(PyMethod.java:228)
at org.python.core.PyMethod.call(PyMethod.java:218)
at org.python.pycode._pyx178.f$0(<script>:70)
at org.python.pycode._pyx178.call_function(<script>)
at org.python.core.PyTableCode.call(PyTableCode.java:171)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1614)
at org.python.core.builtin.eval(builtin.java:497)
at org.python.core.builtin.eval(builtin.java:501)
at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:57)
... 16 common frames omitted
2018-02-12 15:47:18.850 [scheduler-p
Can you help us ?

Regards

I already defined by XLD server. Why again?

Is it possible that the XLD plug-in reuses the information of XLD definition.
We have to define it again. It is confusing and error-prone. See the screenshots please.

Tiles

We have had a lot of issues lately, one has been OOM issue with this plugin. I want to make sure that there is something on record here. The tiles that are new, we had a user add some and the query was made to the entire repository. This threw an hprof in the log directory and thread starvation in the logs. Effectively making XLR not response to URL's. I left it run for 30 minutes afterwards but it never recovered so I was forced to restart the service.

Per Jeroene
We can reproduce such a large request by using the API end-point with same url: "/deployit/tasks/v2/query". This call potentially return hundreds of megabytes of data and also puts a significant load on the database for a long time. This call seems to be made by a Jython plugin as shown in the sceenshot of the memory threads of XLR. This Jython plugin may then cause big heap usage causing an OOM Exception. Our search for such a plugin has revealed that the "xlr-xldeploy-plugin" has a dashboard tile that does such a data call retrieving a lot of data. This corresponds to the tile request that was shown in the logs and this particular plugin indeed has a tile that cause such large requests to the API.

According to the logs in shown in the mail there is a user (cshaw4) that has this tile in his dashboard. To avoid this problem from re-occuring we suggest that you remove the tile from the template. You can navigate to any release and then modify the URL to point to "Release991664026" to find the corresponding template to modify.

Note that even when this tile is removed, any user can still re-add this tile and cause the problem to re-appear. We can release a version of the community plugin with this specific tile functionality disabled to avoid this. When the next version of the plugin is released you can then upgrade without the risk of running OOM caused by this plugin as we are working with the community on a new release of this plugin to avoid this problem.

Cryptic error message when an Environment does not exist

Suppose the following environment is passed in but does not exist:

Environments/sandbox/openshift/sandbox-routing

XLR will give the following error:

2017-01-25 15:22:17.105 [pool-1-thread-1] {task=Applications/Release5589468/Phase7034986/Task7012573} WARN c.x.xlrelease.script.ScriptService - Exception during execution
javax.script.ScriptException: Exception: Failed to prepare deployeds. Server return [500], with content [ : only whitespace content allowed before start tag and not E (position: START_DOCUMENT seen E... @1:1)
only whitespace content allowed before start tag and not E (position: START_DOCUMENT seen E... @1:1) ] in <script> at line number 22
at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:202) ~[jython-standalone-2.7.0.jar:na]
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42) ~[jython-standalone-2.7.0.jar:na]
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31) ~[jython-standalone-2.7.0.jar:na]
at com.xebialabs.xlrelease.script.ScriptExecutor.lambda$evalScriptPrivileged$53(ScriptExecutor.java:91) ~[server-4.8.1.jar:na]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_101]
at com.xebialabs.xlrelease.script.ScriptExecutor.evalScriptPrivileged(ScriptExecutor.java:90) ~[server-4.8.1.jar:na]
at com.xebialabs.xlrelease.script.ScriptExecutor.evalScriptWithUtilities(ScriptExecutor.java:84) ~[server-4.8.1.jar:na]
at com.xebialabs.xlrelease.script.ScriptExecutor.evalScriptWithApi(ScriptExecutor.java:73) ~[server-4.8.1.jar:na]
at com.xebialabs.xlrelease.script.ScriptService.executeScript(ScriptService.java:637) ~[REL-3352-hotfix.jar:na]
at com.xebialabs.xlrelease.script.ScriptService.lambda$executeCustomScriptTask$55(ScriptService.java:206) ~[REL-3352-hotfix.jar:na]
at com.xebialabs.deployit.plumbing.scheduler.Scheduler$1.run(Scheduler.java:84) ~[server-4.8.1.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
Caused by: org.python.core.PyException: null
at org.python.core.PyException.doRaise(PyException.java:226) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.Py.makeException(Py.java:1337) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.Py.makeException(Py.java:1341) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.Py.makeException(Py.java:1345) ~[jython-standalone-2.7.0.jar:na]
at xldeploy.XLDeployClient$py.deployment_prepare_deployeds$20(pyclasspath/xldeploy/XLDeployClient.py:217) ~[na:na]
at xldeploy.XLDeployClient$py.call_function(pyclasspath/xldeploy/XLDeployClient.py) ~[na:na]
at org.python.core.PyTableCode.call(PyTableCode.java:167) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.PyBaseCode.call(PyBaseCode.java:307) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.PyBaseCode.call(PyBaseCode.java:198) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.PyFunction.call(PyFunction.java:482) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:237) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.PyMethod.call(PyMethod.java:228) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.PyMethod.call(PyMethod.java:218) ~[jython-standalone-2.7.0.jar:na]
at org.python.pycode._pyx18.f$0(<script>:63) ~[na:na]
at org.python.pycode._pyx18.call_function(<script>) ~[na:na]
at org.python.core.PyTableCode.call(PyTableCode.java:167) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.PyCode.call(PyCode.java:18) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.Py.runCode(Py.java:1386) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.builtin.eval(builtin.java:497) ~[jython-standalone-2.7.0.jar:na]
at org.python.core.builtin.eval(builtin.java:501) ~[jython-standalone-2.7.0.jar:na]
at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259) ~[jython-standalone-2.7.0.jar:na]
at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:40) ~[jython-standalone-2.7.0.jar:na]
... 16 common frames omitted

This happens because in deployment_prepare_deployeds the passed in deployment variable (which comes from deployment_prepare_initial) is:

Environments/sandbox/openshift/sandbox-routing is not a Environment

This appears to be the raw response from XLD. There should be a better error handling when the response of deployment_prepare_undeploy, deployment_prepare_update, and deployment_prepare_initial is an error instead of an XML.

Add a property to skip the display of the task log (step output)

Question around xlr-xldeploy-plugin usage

Hi,

I was hoping someone can clarify something for me. We’ve recently upgraded our XLR version from 7.x to 8.5.x.

We’ve always used both the official XLD plugin and this community edition and, since the upgrade, can see many XLD duplicate tasks available. Looking at the .jar files, both plugins contain the same tasks so that make sense. However, I’m curious to know whether we’ve been using the plugins wrong – was the community plugin designed to replace the official one?

The way we have been using it is by running them side-by-side with each other.

Thanks

Could not find a type definition associated with type [xlrelease.Tile] with XLR 4.8.1

Is this version 2.7.0 of this plugin compatible with XLR version 4.8.1? The Readme states 4.8+ this error prevents XLR from starting up.

Support ability to pass in placeholder values when deploying

I need the ability to dynamically pass in placeholder values from XLR to XLD. It is possible by simply modifying the deployment XML post but at present the only way to do so is hacky (#31). I can't just use dictionaries since they change with every deployment. Using dictionaries would require one per each application (or else concurrent deployments become a problem) which would mean 1000's of them which isn't maintainable.

Thank you for reporting! Fixed this in v3.3.1

Thank you for reporting! Fixed this in v3.3.1

Originally posted by @jdewinne in #80 (comment)

Migrate does not support composite package

Not sure we ever logged this but we have composite packages in use, and the migrate package task does not seem to work with composites. Will this be supported? With out it we cannot automate promotion of composites between XLD instances towards production deployment

There is no Get CI task

The plugin contains a "create CI" and a "delete CI" task. There is no corresponding "get CI" task. Hence, XLR scripts are not able to perform logic based on knowledge of CI metadata.

Get Latest Deployed Version throwing exception

I am running the same setup with the 2.1.2 version of the plugin I have working with the 2.0.2 version and I am getting this error. Let me know if you need more info. I may get a chance to run more tests end of week. Thanks!

Exception during execution:
TypeError: get_ci() takes exactly 3 arguments (4 given) in <script> at line number 11

Add "getAllVersions" task

Adding a "getAllVersions" Task would be useful, if you need to be able to see if a certain version already has been imported to XLD.
Request from a client.

Migrate Task can't handle different application paths

Migrate task fails when Applications are in different paths across instances.

Source: Applications/FolderA/MyApp
Target: Applications/FolderB/MyApp

Existence checks fail because we're continuing to expect the source path on the destination.

I've fixed it in my fork, but it requires the get_all_versions function of the XLDClient to have the recursive feature.

If #63 goes in, I can make another patch branch to add the fix in

Migrate Task should not fail if package exists

Hi There,

This is more of a feature than an issue I suppose. We have a user with a migrate task in there template which they use through numerous environments. So generally the package is migrated during the release to the early environment. Then the next time the release runs the migrate task fails because the package exists. I was thinking it would be better to just give a message and move on if it already exists since no harm is done. For now I have them using "does ci exist" then using the boolean to decide whether or not to skip the migrate task.

I believe it is the fetch2 function that errors out if the package exists. Again we have a workaround, consider it a feature request :)

Thanks,
Bernie

Inconsistencies between xlr-xld becoming problematic.

The latest suite of changes have seen all the XL Deploy tasks renamed so that everythinbg comes under XL Deploy (rather than split between XL Deploy and xldeploy)

This is good... except that now it looks incredibly inconsistent. Some fields take Applications/ and some don't. Some have auto-complete. Some are duplicate functions entirely.

Technically the best thing would be to have 1 plugin. If we can't, we should try and be consistent so that inputs from one task type can seamlessly work with another.

Undeploy Tasks are broken

Worked in the 2.5.0.
When firing a task, get the following errror

Exception during execution:
AttributeError: type object 'XLDeployClientUtil' has no attribute 'get_deployment_package' in <script> at line number 13

Deployed Versions Tile - "KeyError: 'environment_id' in"

When selecting an existing environment in the Deployed Versions Tile, an error is thrown which reads:

2017-01-25 09:27:23.384 [qtp169833205-22214] {request=/tiles/Applications/Folder784315945/Release54048206/summary/Tile509385314/data, method=GET, username=admin} ERROR c.x.d.UnhandledExceptionMapper - Unhandled Exception: {}
java.lang.RuntimeException: javax.script.ScriptException: KeyError: 'environment_id' in <script> at line number 17

Caused by: javax.script.ScriptException: KeyError: 'environment_id' in <script> at line number 17
        at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:202)
        at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:42)
        at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
        at com.xebialabs.xlrelease.script.ScriptExecutor.lambda$evalScriptPrivileged$92(ScriptExecutor.java:92)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.xebialabs.xlrelease.script.ScriptExecutor.evalScriptPrivileged(ScriptExecutor.java:91)
        at com.xebialabs.xlrelease.script.ScriptExecutor.evalScriptWithUtilities(ScriptExecutor.java:85)
        at com.xebialabs.xlrelease.script.ScriptExecutor.evalScript(ScriptExecutor.java:66)
        at com.xebialabs.xlrelease.script.ScriptService.executeScript(ScriptService.java:646)
        at com.xebialabs.xlrelease.script.ScriptService.executeScriptWithWriter(ScriptService.java:227)
        ... 93 common frames omitted
Caused by: org.python.core.PyException: null
        at org.python.core.Py.KeyError(Py.java:249)
        at org.python.core.PyObject.__getitem__(PyObject.java:738)
        at xldeploy.XLDeployClient$py.get_deployed_applications_for_environment$44(__pyclasspath__/xldeploy/XLDeployClient.py:382)
        at xldeploy.XLDeployClient$py.call_function(__pyclasspath__/xldeploy/XLDeployClient.py)

Deployment Task does not call 'validate' method before generating the deployment task

overrideDeployedProps bug

root.findall(key) returns a list. Per comment #32 (comment)

# deployed_properties must be a string, as the map_string_string type has a bug when putting the '=' in the key.
def override_deployed_properties(deployment_xml, deployed_properties):
    root = ET.fromstring(deployment_xml)
    if deployed_properties:
        deployeds_properties_dict = dict(ast.literal_eval(deployed_properties))
        for key in deployeds_properties_dict:
            pkey_xml = root.findall(key)
            pkey_xml.text = deployeds_properties_dict[key]
    return ET.tostring(root)

needs to be

# deployed_properties must be a string, as the map_string_string type has a bug when putting the '=' in the key.
def override_deployed_properties(deployment_xml, deployed_properties):
    root = ET.fromstring(deployment_xml)
    if deployed_properties:
        deployeds_properties_dict = dict(ast.literal_eval(deployed_properties))
        for key in deployeds_properties_dict:
            for pkey_xml in root.findall(key):
                pkey_xml.text = deployeds_properties_dict[key]
    return ET.tostring(root)

Error:

Creating initial deploy

Mapping all deployables

Exception during execution:
AttributeError: 'list' object has no attribute 'text' in <script> at line number 22

Undeploy Fails on 404 (Unavailable)

If I try and undeploy a package that isn't present, we get a 404 error and the task fails. On the one hand, that's correct, the task has not taken action. However, the environment state is correct. Should there be an option to "accept not found" as a success? So that if it's not deployed, I don't have to skip/pass the task, but it should just continue to the next step?

Task xldeploy.createRTD failure

I have xlr-xld plugin 2.2.0 and trying to create task xldeploy.createRTD.

Task is failing with following error...

Exception during execution: AttributeError: 'XLDeployClient' object has no attribute 'httpRequest' in <script> at line number 13.

Following are the customizations made to the community plugin

1. Added type xldeploy.createRTD to synthetic.xml.
2. Added createRTD.py to xldeploy directory of plugin.
   xlr-xldeploy-plugin-master.zip
3. Added def createRTDInstance and def addRTDToEnvironment to XLdeployclient.py in xldeploy directory of plugin.

Migrate Task failing

Hi,

I was hoping if someone could help me with why the Migrate Task keeps failing with the below.

https://XLD_Server_URL/deployit/internal/download/5568c4f3-4654-4ca0-9ca0-42d3082e902c
Exception during execution:
Exception: Failed to import package. Server return [500], with content [Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
Keystore was tampered with, or password was incorrect
Password verification failed] in <script> at line number 50

What are the prerequisites for migrating a deployment package between different instances of XLD? I have created truststores for both XLDs, added each other's .cer file in the respective stores, and on the XLR truststore as well. I get the same error if I reverse the servers too. The keystore and truststore passwords are definitely correct, and I am able to deploy packages to bot XLD serers via XLR too.

Any help will be much appreciated.
Thanks

Your CLI function is not working properly

When I am trying to create a task via xl deploy CLI. I am getting this error.

command used:
d = deployment.prepareInitial(package.id, environment.id)
d = deployment.prepareAutoDeployeds(d)
d = validate(d)
deployment.createDeployTask(d)

javax.ws.rs.ProcessingException: com.thoughtworks.xstream.converters.ConversionException:
---- Debugging information ----
cause-exception     : java.lang.NullPointerException
cause-message       : Name is null
class               : com.xebialabs.deployit.engine.api.execution.SerializableTask
required-type       : com.xebialabs.deployit.engine.api.execution.SerializableTask
converter-type      : com.xebialabs.deployit.booter.remote.xml.TaskConverterSelector
path                : /task
line number         : 1
version             : not available

There is something wrong with the cli package.

UnicodeEncodeError when parsing the xml data for tasks

need to encode('utf-8','ignore') http://stackoverflow.com/questions/1652904/unicodeencodeerror-ascii-codec-cant-encode-character?rq=1
improvement : new property to display or not the step logs...

Update CI property does not support list_of_string property values