zendesk / zendesk_app_migrator

CLI tool to assist in migrating v1 App Framework apps to v2

Zendesk App Migration Helper

What is it?

The App Migration Helper is CLI tool for assisting with the migration of v1 App Framework apps to v2

How does it work?

The migrator executes a series of tasks to:

• Reorganise files and folders
  • Copies templates
  • Copies translations
  • Copies stylesheets
  • Copies and rewrites Common JS modules
• Update the manifest file
  • Change to "frameworkVersion": "2.0"
  • Rewrite locations to v2 hash syntax
• Rewrite JavaScript code from v1 app.js
• Creates an HTML file from template that imports all the necessary deps., including v1 shims/helpers

App Scaffold

Please refer to App Migration Scaffold for detailed documentation on developing, testing, building and deploying applications migrated to V2.

Usage

Options

Option	Default	Required	Description
-p --path		Yes	The location of the v1 app
-r --replace-v1	false	No	Whether to backup v1 files, and replace with v2 during migration. Backed up files will be moved to a v1 folder, alongside the new v2 files.
-a --auto	false	No	Enables more end-to-end transformations of JavaScript, and CSS code. See the expanded Auto option section below for more details.

Auto option

Please note that auto transforms may not work as expected. Use with caution, and always test extensively after migration. For a better understanding of how auto transforms are expected to behave, look at tests for the migrate_app_js step tests. Transforms currently available:

• Rewrites synchronous v1 API calls to be asynchronous. This works by adding a shim for the ZAF SDK APIs. The shim will be combined with the use of async/awaits.
• Injecting JavaScript and CSS assets to support the use of the zdSelectMenu API available in v1.

For development...

Source files are under src, test files under src/test. The codebase currently makes use of ES6 features by way of the Typescript compiler. To that end, all of the source code is written in Typescript. An added bonus is that the package will ship with type declarations alongside the JavaScript.

Setting up your dev environment

You will need:

• Yarn
• NodeJS

brew install yarn && yarn install

Dependencies

Dependencies are declared via yarn in the package.json file.

Testing the CLI

Optionally install ts-node and typescript globally, like npm install -g ts-node typescript.

Run ts-node src/index.ts migrate --path ~/path/to/v1/app/source

If not installing ts-node and typescript globally, reference the local versions of those packages like ./node_modules/.bin/ts-node src/index.ts migrate --path ~/path/to/v1/app/source

After running yarn build, it is possible to run the down-level version of the Migrator like node ./lib/index.js migrate --path ~/path/to/v1/app/source. Similarly, the down-level tests can be run like ./node_modules/.bin/mocha ./lib/test/**/*.test.js. Doing so effectively just demonstrates that the output from the Typescript compiler is valid JavaScript for NodeJS.

Running tests

yarn test

Tests are run using the mocha test runner. chai is the assertion library. We make use of chai-as-promised to provide more elegant assertions against async behaviour.

Other scripts

• yarn test-watch
• yarn build
• yarn build-watch

Contributing

We welcome contributions. First, please read the Pull Request Guidelines and Style Guide pages on our wiki.

Deployment

The App Migrator will be deployed as a node package, and/or as a dependency of other projects.

zendesk_app_migrator's Issues

Add telemetry mechanism

It would be very helpful if we could have the migrator "call home" periodically. This could be an opt-in feature, i.e. we'd ask the user whether they would allow us to send anonymous data from time to time, so as to improve the tool.

If the user accepted then we could persist the choice by saving an artefact to the file system in the package, i.e. .migratorc.json.

To begin with, it would be good to collect:

• Number of times the migrator was run overall
• Number of times the migrator was run per-app
• Number of times the migrator ran successfully
• Number of times the migrator ran per-app
• Number of times the migrator failed
• Step at which the migrator failed (possibly including error message)

@pmgrove I'm interested in your opinion on this one, from a Product perspective

Allow developers to decide whether to overwrite (and backup) v1 source

Output a migration artefact

When the migrator runs, we should leave a migration artefact, so that later we can identify when an app that was migrated has been updated in Cactus.

The artefact could be something as simple as a key in the manifest.json generated by the migrator.

Perhaps "migrated": "1502418553265" would suffice?

Dry run shows all files that have been loaded into the virtual file system

When running the migration helper with the --dryrun flag, the output shows all of the files that were loaded into the virtual file system in the list of files that would have been saved.

While this behaviour isn't incorrect (it is showing all of the files in the virtual FS), it could be a bit confusing to the user. The speed of the commit call might also be improved if the commit only needed to run on the files that will be exported to the tmp directory.

Incorrect migration output when using colon-delimiting and invoke

The generated/converted code on the critical line is:
await wrapZafClient(this.zafClient, "ticketFields");

...when it should have maybe been something like:
await wrapZafClient(this.zafClient, "ticketFields:sharedWith.hide");

However, even the second call above doesn't work (I tried).

Command app_migration doesn't work

The following command is failing and gives the following output

Commad:
app_migrator --path --auto ./integrations.zendesk/

Output:

/zendesk_app_migrator/lib/index.js:18
    throw new Error(cmd + " not implemented yet");
    ^

Error: undefined not implemented yet
    at Object.<anonymous> (/Volumes/work/zendesk_app_migrator/lib/index.js:18:11)
    at Module._compile (module.js:635:30)
    at Object.Module._extensions..js (module.js:646:10)
    at Module.load (module.js:554:32)
    at tryModuleLoad (module.js:497:12)
    at Function.Module._load (module.js:489:3)
    at Function.Module.runMain (module.js:676:10)
    at startup (bootstrap_node.js:187:16)
    at bootstrap_node.js:608:3

Installed using:
npm i -g zendesk_app_migrator

Os:
Mac os High Sierra 10.13.6 (17G65)

replace-v1 flag breaks migration

Hello.
Recently I tried to migrate our app to framework v2 and get an error when tried to use --replace-v1 flag.
Here's the full output:

sandro@Sandros-MacBook $ zat migrate --replace-v1
App will be migrated from: /Users/sandro/Work/Projects/devmate/devmate-zendesk-app
App will be migrated to: /Users/sandro/Work/Projects/devmate/devmate-zendesk-app/v2/
? Are you sure you want to replace v1 files? Yes
v1 App will be backed up to: /Users/sandro/Work/Projects/devmate/devmate-zendesk-app/v1/
Warning: `parser` with value "postcss" is deprecated. Use "css", "less" or "scss" instead.
 [==================[22[1m=--] 12/13 steps completemv: cannot move to self: /Users/sandro/Work/Projects/devmate/devmate-zendesk-app/v1
 [====================[22[1m=] 13/13 steps complete
Installing dependencies
yarn install v1.3.2
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning "karma-webpack > [email protected]" has incorrect peer dependency "webpack@1 || ^2.1.0-beta".
[4/4] Building fresh packages...
Done in 45.27s.
Building v2 app
yarn run v1.3.2
$ webpack -p
Hash: a55139b8c2439aa4b3cf
Version: webpack 3.5.1
Time: 2710ms
     Asset       Size  Chunks                    Chunk Names
   main.js     315 kB       0  [emitted]  [big]  app
  main.css    0 bytes       0  [emitted]         app
index.html  745 bytes          [emitted]         
  [85] (webpack)/buildin/global.js 509 bytes {0} [built]
 [114] multi babel-polyfill ./src/javascripts/index.js ./src/stylesheets/app.scss 52 bytes {0} [built]
 [298] ./src/javascripts/index.js 1.11 kB {0} [built] [failed] [1 error]
 [299] ./src/stylesheets/app.scss 41 bytes {0} [built]
 [300] ./node_modules/css-loader?sourceMap&root=/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/dist/assets!./node_modules/sass-loader/lib/loader.js?sourceMap!./src/stylesheets/app.scss 248 bytes [built]
    + 299 hidden modules

ERROR in ./src/javascripts/index.js
Module build failed: Error: No ESLint configuration found.
    at Config.getLocalConfigHierarchy (/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/node_modules/eslint/lib/config.js:256:39)
    at Config.getConfigHierarchy (/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/node_modules/eslint/lib/config.js:182:43)
    at Config.getConfigVector (/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/node_modules/eslint/lib/config.js:287:21)
    at Config.getConfig (/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/node_modules/eslint/lib/config.js:330:29)
    at processText (/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/node_modules/eslint/lib/cli-engine.js:162:33)
    at CLIEngine.executeOnText (/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/node_modules/eslint/lib/cli-engine.js:668:26)
    at lint (/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/node_modules/eslint-loader/index.js:218:17)
    at Object.module.exports (/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/node_modules/eslint-loader/index.js:213:21)
 @ multi babel-polyfill ./src/javascripts/index.js ./src/stylesheets/app.scss
Child html-webpack-plugin for "index.html":
     1 asset
       [4] ./node_modules/handlebars-loader!./lib/templates/layout.hdbs 2.27 kB {0} [built]
        + 7 hidden modules
Child extract-text-webpack-plugin node_modules/extract-text-webpack-plugin/dist node_modules/css-loader/index.js?sourceMap&root=/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/dist/assets!node_modules/sass-loader/lib/loader.js?sourceMap!src/stylesheets/app.scss:
       [0] ./node_modules/css-loader?sourceMap&root=/Users/sandro/Work/Projects/devmate/devmate-zendesk-app/dist/assets!./node_modules/sass-loader/lib/loader.js?sourceMap!./src/stylesheets/app.scss 248 bytes {0} [built]
        + 1 hidden module
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
undefined
 [====================[22[1m=] 13/13 steps completeError: Promise rejected with value: 'error Command failed with exit code 2.\n'
    at /Users/sandro/.nvm/versions/node/v8.9.1/lib/node_modules/zendesk_app_migrator/node_modules/loud-rejection/index.js:27:12
    at Array.forEach (<anonymous>)
    at EventEmitter.<anonymous> (/Users/sandro/.nvm/versions/node/v8.9.1/lib/node_modules/zendesk_app_migrator/node_modules/loud-rejection/index.js:23:24)
    at emitTwo (events.js:126:13)
    at EventEmitter.emit (events.js:214:7)
    at emit (/Users/sandro/.nvm/versions/node/v8.9.1/lib/node_modules/zendesk_app_migrator/node_modules/signal-exit/index.js:77:11)
    at processEmit [as emit] (/Users/sandro/.nvm/versions/node/v8.9.1/lib/node_modules/zendesk_app_migrator/node_modules/signal-exit/index.js:150:5)

As you can see there's an error Module build failed: Error: No ESLint configuration found.
I noticed that none of "dot" files (.babelrc, .eslintrc, .node-version, .travis.yml) were copied to root directory of the project.

I think I've found the reason for this error and can submit a PR with the fix.

Add a sensible .gitignore for Node

For example, https://github.com/github/gitignore/blob/master/Node.gitignore

Generate build output automatically

I think we should adopt an approach like https://github.com/typestyle/typestyle/blob/master/package.json#L14-L15 where creating a new version via yarn version runs the preversion and postversion scripts (if configured), so that the build output (lib directories) is generated. This way, if anyone were consuming the package via the npm registry they would install a version that had JavaScript files alongside Typescript

Add PR template

PR templates are neat, they make things consistent. This project needs one.

Allow relative --path

As it currently works, the migrator requires an absolute path to be provided as the value of the --path option. There shouldn't be any reason we can't resolve relative paths too, i.e. ts-node src/index.ts migrate --path ../my_app

Convert the Migration Helper into a monorepo using Lerna

There is helper code being injected into the migration output. It might be better to be able to put this helper code into independently versioned micro-packages under the main package, that could be deployed using something like Lerna.

https://github.com/lerna/lerna

Dry run still tries to run scripts and creates tmp dir

When using the --dryrun flag, the runScripts function is still called and the export directory is still created.

These should not be run or created.

Add integration testing via Cucumber

It might be nice to have some BDD style features that tested the migrator E2E. We could add some fixture apps that tested common scenarios.

Cucumber.js would likely be the most conventional way to do this.

Incorrect migration output when code outside v1 return block

https://travis-ci.com/zendesk/zendesk_app_migrator/jobs/91853106

Show changed/diff summary as part of migration process

A summary of what were changed or moved during migration process would be helpful for app developers, especially if -a --auto flag was used.

Example of such summary:

* Migrate translations *
Original: /path/to/app/translations/*
Copied to: /path/to/app/v2/src/translations/*

* Migrate CSS *
Original: /path/to/app/app.css
Created: /path/to/app/v2/src/stylesheets/app.scss
Created: /path/to/app/v2/src/stylesheets/(diff|changelog).(md|txt)

*Migrate JS *
Original: /path/to/app/app.js
Created: /path/to/app/v2/src/javascripts/legacy_app.js
Created: /path/to/app/v2/src/javascripts/index.js
Created: /path/to/app/v2/src/javascripts/(diff|changelog).(md|txt)

Example of (diff|changelog).(md|txt) file:

--- /path/to/app/app.css
+++ /path/to/app/v2/src/stylesheets/app.scss
@@ -1,4 +1,6 @@
-h4 {
-  font-size: 16px;
-  margin-top: 20px;
+* {
+  h4 {
+    font-size: 16px;
+    margin-top: 20px;
+  }
 }

Other than creating (diff|changelog).(md|txt) files, Migration Helper might give app developers an Y/N option at the end of the process to print the diff to the terminal as well. Note: this might not be practical because the diff can be gigantic for non-trivial apps.

Transforming v1 APIs to v2

The aspirational end-goal for the Migration Helper.

An initial list of problems to solve. These can be split out into discrete Issues later.

• When a v1 API call is saved off to a scope declaration, then that declaration has wrapper methods called on it, i.e. var collabs = this.ticket().collaborators(); collabs.forEach(c => console.log(c.role())). The c.role() is difficult/complicated to replace.
• ticketFields and customFields APIs need to transformed into "special" colon-delimited paths
• Use of /zd(ComboSelect|Select)?Menu/ jQuery plugin needs to be shimmed, or included
• When an app function is changed to be async to support the use of await with a v1 -> v2 API, other bindings to that app function need to also be updated to have async/await
• For nav bar apps, references to "pane.activated" should be changed to "app.activated"

Migration fails on Windows 8.1 when parsing Common JS module paths

​When attempting to migrate an app on Windows 8.1, the Migrator chokes when building a dynamic RegExp from a String. The trailing (OS-specific) path separator, a backslash, is interpreted as an escape char.

Use ZAT to validate generated app package

Once the migrator has generated a v2 package, we should be able to use ZAT to validate the correctness of the package.

There is some duplication here though in that the migrator already runs some pre-migration validations to ensure the app can be migrated. Perhaps there is an opportunity to consolidate the validation phase somehow, or at least use the same validation mechanism?

Migrator.js breakage

Trying to run the migration (as root) I get a JS error from the migrator.js file:

thomas-b zendesk_widget # /home/tb/.rvm/gems/ruby-2.4.4@rails51/bin/zat migrate --replace-v1
The Zendesk App Migration Helper isn't installed yet. Would you like to try installing now? (y)
/usr/lib/node_modules/zendesk_app_migrator/lib/migrator.js:16
Symbol.asyncIterator =
^
TypeError: Cannot assign to read only property 'asyncIterator' of function 'function Symbol() { [native code] }'
at Object. (/usr/lib/node_modules/zendesk_app_migrator/lib/migrator.js:16:22)
at Module._compile (internal/modules/cjs/loader.js:678:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:689:10)
at Module.load (internal/modules/cjs/loader.js:589:32)
at tryModuleLoad (internal/modules/cjs/loader.js:528:12)
at Function.Module._load (internal/modules/cjs/loader.js:520:3)
at Module.require (internal/modules/cjs/loader.js:626:17)
at require (internal/modules/cjs/helpers.js:20:18)
at Object. (/usr/lib/node_modules/zendesk_app_migrator/lib/index.js:7:18)
at Module._compile (internal/modules/cjs/loader.js:678:30)
Unable to install the Zendesk App Migration Helper Please follow the installation instructions at https://github.com/zendesk/zendesk_app_migrator before running this command again

Any clues as to why I get this exception?

My env:
nodejs v10.1.0
git v1.9.1
Linux Mint 17.2 Rafaela
Ruby 2.4.4

App Migration "Service"

This issue is a placeholder for a potential Tech Review post.

I've been considering the possibility of standing up a new service, potentially as a serverless node app (Express.js?) running in AWS Lambda that can talk to S3. The service would provide a simple API by which an app could be migrated from v1 -> v2. Input might potentially be a reference to a v1 app upload in S3, output could be a reference to a new app upload in S3. If not input/output as just described, the service could accept a zip (v1), and return a new zip (v2).

Other thoughts:

• The service would be accessible within our network only
• The service could potentially be stateless
• Access to the service would be provided via Cactus, the App Developer Console. This would mean we could proactively message developers with v1 Public apps, and motivate them to upgrade to v2

Potential challenges:

• The migrator output will likely never be 100% production-ready, i.e. developers would need to do some manual migration before updating via Cactus.
• The number of Public v1 apps might not warrant standing up a completely new service.
• Once Public apps were migrated to v2, the service might no longer be useful.
• The cost in AWS could be prohibitive.
• There may be some infrastructure-related constraints with using AWS in this way

@danielbreves @pmgrove appreciate your input on this idea

Use Babel + Webpack as the build pipeline

To make the most of ES* features, like async/awaits we could use Babel to downlevel those features. We could additionally use Webpack to build the final output into a distributable bundle

Add extra validation steps

Before actually starting the migration process, Migration Helper can perform a number of validation as prerequisite checks:

• zat validate --path /path/to/v1_app should pass. The original app should be a valid V1 app.
• frameworkVersion in manifest.json must be less than 2.0.
• marketingOnly in manifest.json must not be true.
• Original app should not be requirements-only app.
• Any other specific conditions required for migrating V1 app to V2 app.

requirements.json is not copied over by app migrator

For v1 app with requirements, App Migrator should copy requirements.json as part of migration process.