metal / metal-tools-soy

Tool that can be used to compile metal soy files.

metal-tools-soy

Tool that can be used to compile metal soy files.

CLI

Install

$ npm install --global metal-tools-soy

Use

$ metalsoy

Options

You can see information about the available options by typing $ metalsoy --help in the command line.

dest

$ metalsoy --dest folderName

The directory where the compiled files will be stored.

externalMsgFormat

$ metalsoy --externalMsgFormat format

Pattern that should be used to format the extracted external messages from compiled files.

help

$ metalsoy --help

Shows help information for all options, including default values.

outputDir

$ metalsoy --outputDir folderName

Temp directoy used to compile soy files.

Note: this option does not determine where the final .soy.js files are placed, see --dest option.

skipMetalGeneration

$ metalsoy --skipMetalGeneration

Passing this will cause soy files to be just compiled, without the addition of metal generated code (like the component class).

soyDeps

$ metalsoy --soyDeps node_modules/metal*/src/**/*.soy

Soy files that the main source files depend on, but that shouldn't be compiled. The soy compiler needs these files.

src

$ metalsoy --src src/**/*.soy

The path globs to the soy files to be compiled.

version

$ metalsoy --version

Displays current version of metal-tools-soy.

SoyToIncrementalDomSrcCompiler

This project uses the SoyToIncrementalDomSrcCompiler to compile the soy files to metal using Incremental DOM. Since the compiler is not independently released, the process to update it in this project is as follows:

1. Clone the https://github.com/google/closure-templates repository
2. Update the <version> value inside pom.xml to the date of the latest commit that is going to get released using yyyy-mm-dd as the date format
3. Run mvn install on the root folder
4. Copy the generated file from ~/.m2/repository/com/google/template/soy/{version}/soy-{version}-SoyToIncrementalDomSrcCompiler.jar to the jar folder in this project

metal-tools-soy's Issues

Doesn’t work correctly with files with windows-style line-endings (CRLF)

We found the issue developing a Liferay tag(lib) with Soy. You can replicate the issue in a Liferay module and forcing the line-endings to be CRLF:

$ unix2dos src/main/resources/META-INF/resources/MyTemplate.soy
$ gradle clean build
... ... ...
> Building 67% > :modules:my-project:transpileJS
Running 'build'...
Compiling soy

stream.js:74
      throw er; // Unhandled stream error in pipe.
      ^
Error: Compile error:
errors during Soy compilation
In file drop_area/DropArea.soy:1:23: parse error at '
': expected eof, {alias, {deltemplate, or {template
{namespace MyTemplate}
                      ^


:modules:tools-osgi:adr-frontend-taglib:transpileJS FAILED
... ... ...
BUILD FAILED

Total time: 17.777 secs

And obviously you can replicate the fix:

$ dos2unix src/main/resources/META-INF/resources/MyTemplate.soy
$ gradle clean build
... ... ...
BUILD SUCCESSFUL

Total time: 18.755 secs

Improve error message from soy compiler when java < 8:

Today the error just says something like Exception in thread "main" java.lang.UnsupportedClassVersionError: com/google/template/soy/SoyToIncrementalDomSrcCompiler : Unsupported major.minor version 52.0, which is hard to understand, specially for js developers.

Let's try to detect this error and show a better message instead.

Params declared with certain types will not trigger updates on change.

I noticed that when I add a param with a record type:

{@param item: [label: string, id: number]}

...My component won't update if item changes.

I know we have a default shouldUpdate implementation for metal-soycomponents (see here), and I think this is returning false because item is not being parsed as being a @param to the template.

Looking at the regex used by soyparser (see here), it looks like the regex will not match that kind of type declaration. I'm guessing there might be a few other corner cases related to type declarations that might also lead to this problem.

[metal-soy-critic] Allow for use as a Library

From Allow for use as a Library by @mthadley:

We should export some functions that allows using metal-soy-critic as a library. We will probably need to update some of the return types so that they make sense for more generic use cases.

We can also consider adding a JSON output option. This would be useful for integration with tools like linters, continuous integration, etc.

Error in plugin 'metal-tools-soy': Parsing failed at 1:22

I setup a metaljs project and try to build the soy templates with metalsoy.
It results in the following error. Its hard to find out if its an error in the template or a bug.

node_modules/.bin/metalsoy
Compiling soy
Error in plugin 'metal-tools-soy'
Message:
Parsing failed at 1:22. Expecting:
'.'
'_'
'}'
a digit
a letter
Finished compiling soy

Thanks in advance

[metal-tools-soy] Create an API to exclude modules

It would be interesting having an API to exclude modules, something like

metalsoy --soyDeps ../node_modules/metal-*/src/**/*.soy --soyExcludes clay-dropdown

[metal-tools-soy]goog.require regex on replaceTemplateRequires isn't working correctly with dotted namespaces

Try execute the regex described at this line on this closure-templates generated file: https://gist.github.com/diegonvs/432de495b3a5318bd68bffc86364ff69

If you remove the dots on namespace the regex will catch the goog.require lines.

[soyparser] Conflicting compiler and parser rules

From Conflicting compiler and parser rules by @drewbrokke

Soy compiler recommends using double curly braces if attribute value contains single curly braces, but soyparser does not allow double curly braces for params (or anything else?).

Example:

{{param code:'YUI().use(
  \'aui-carousel\',
  function(Y) {
    // code goes here
  }
);'}}

[metal-tools-soy] Node process dies after trying to parse more than a specific amount of files in a subfolder

I'm trying to build a Soy project, but I noticed when I try to compile more than 82 files placed in a structure like src/subfolder, the metal-tools-soy package somehow makes Node process exit prematurely.

The function I used to handle the soy call is like this:

const soy = require('metal-tools-soy');

const soyDeps = ['node_modules/metal*/src/**/*.soy'];
const src = ['./src/**/*.soy'];
const dest = ['./soy'];

const buildSoyFiles = (src, dest, soyDeps) => 
  new Promise((resolve, reject) => {
    const handleError = error => reject(error);
    soy({src, dest, soyDeps, handleError}).on('end', () => resolve());
  });

This bug is specifically related to trying to compile more than 82 files in a subfolder and using src/**/*.soy glob pattern as src on soy options parameter.

If I use src/subfolder/*.soy, it works, and if I put all the files in src (not in a subfolder), it works too, but in a real project we use many folders (each one with its own subfolders and so on), even if I include each path to the Soy config to compile, the output can't handle the structure properly.

I've made a project to reproduce this behavior with ease. I created 2 npm scripts:

• npm run success, the script create.sh creates 82 files in src/child and index.js calls Soy in a promise, and it resolves successfully.
• npm run fail, the script will create 83 files in src/child, just as the first one, but index.js will not be able to resolve the promise anymore, and it wont reject as well.

I used a flag to verify if the promise either rejected or resolved, any of those would change the flag, but at the end of the fail case, the flag is still unchanged.
The program simply dies, and kill the whole node process, with no error.

---

PS:

• I used just elementary templates, with the bare minimum content to be a template;
• I cleaned up all the source and compiled files before the tests;
• Tested on Ubuntu and Mac OS.

Passing multiple glob patterns to soyDeps can easily result in duplicated templates

When passing multiple glob patterns to soyDeps, if two glob patterns match the same file, the file will be included twice in the soy compilation and result in an error.

Template 'Component.render' already defined at metal-some-component/src/Component.soy

Obviously this was easy to fix by changing my glob patterns, but it would be nice if we deduped files so they don't get included twice. I think part of the issues is that one of my patterns was a resolved absolute file path to a specific dependency, while the other was just a relative file path to node_modules.

Improve externalMsgFormat Support

We have some naive support to internationalize messages by using externalMsgFormat.

In Liferay Portal, we handle this from the ReplaceSoyTranslationCommand class.

We need a way to support that directly in metal-tools-soy so we can get rid of that additional processing step.

Using flags-taglib as a starting point, modify the following template (Flags.soy):

{namespace Flags}

/**
 * Prints the Flags component.
 */
{template .render}
	{msg desc=""}you-are-about-to-report-a-violation-of-our-{$termsOfUse}.-all-reports-are-strictly-confidential{/msg}
{/template}

This, in portal, will generate basically the following code (Flags.soy.js)

function $render(opt_data, opt_ijData, opt_ijData_deprecated) {
  opt_ijData = opt_ijData_deprecated || opt_ijData;
  var termsOfUse__soy4 = function termsOfUse__soy4() {
    incrementalDom.elementOpenStart('a');
    incrementalDom.attr('href', '#');
    incrementalDom.attr('target', '_blank');
    incrementalDom.elementOpenEnd();
    /** @desc  */
    var MSG_EXTERNAL_2898360495953199567 = Liferay.Language.get('terms-of-use');
MSG_EXTERNAL_2898360495953199567 = MSG_EXTERNAL_2898360495953199567.replace(/{(\d+)}/g, '\x01$1\x01')
    incrementalDom.text(goog.string.unescapeEntities(MSG_EXTERNAL_2898360495953199567));
    incrementalDom.elementClose('a');
  };
  /** @desc  */
  var MSG_EXTERNAL_6047645258311882210 = Liferay.Language.get('you-are-about-to-report-a-violation-of-our-x.-all-reports-are-strictly-confidential');
MSG_EXTERNAL_6047645258311882210 = MSG_EXTERNAL_6047645258311882210.replace(/{(\d+)}/g, '\x01$1\x01')
  var lastIndex_11 = 0,
      partRe_11 = /\x01\d+\x01/g,
      match_11;
  do {
    match_11 = partRe_11.exec(MSG_EXTERNAL_6047645258311882210) || undefined;
    incrementalDom.text(goog.string.unescapeEntities(MSG_EXTERNAL_6047645258311882210.substring(lastIndex_11, match_11 && match_11.index)));
    lastIndex_11 = partRe_11.lastIndex;
    switch (match_11 && match_11[0]) {
      case '\x010\x01':
        termsOfUse__soy4();
        break;
    }
  } while (match_11);
}

If we disable the replaceSoyTranslation task in the build.gradle file:

replaceSoyTranslation {
	enabled = false
}

And configure metal-tools-soy "properly" like:

"build": "metalsoy --externalMsgFormat \"Liferay.Language.get('\\$2')\" --soyDeps \"node_modules/+(clay-button|clay-icon|com.liferay.frontend.js.web)/**/*.soy\" && cross-env NODE_ENV=production babel --source-maps -d classes/META-INF/resources src/main/resources/META-INF/resources && liferay-npm-bundler && npm run cleanSoy",

The resulting file (Flags.soy.js) looks like:

    function $render(opt_data, opt_ijData, opt_ijData_deprecated) {
      opt_ijData = opt_ijData_deprecated || opt_ijData;
      var termsOfUse__soy4 = function termsOfUse__soy4() {
        incrementalDom.elementOpenStart('a');
        incrementalDom.attr('href', '#');
        incrementalDom.attr('target', '_blank');
        incrementalDom.elementOpenEnd();
        /** @desc  */
        var MSG_EXTERNAL_2898360495953199567 = Liferay.Language.get('terms-of-use');
        incrementalDom.text(goog.string.unescapeEntities(MSG_EXTERNAL_2898360495953199567));
        incrementalDom.elementClose('a');
      };
      /** @desc  */
      var MSG_EXTERNAL_6047645258311882210 = goog.getMsg('you-are-about-to-report-a-violation-of-our-{$termsOfUse}.-all-reports-are-strictly-confidential', { 'termsOfUse': '\x010\x01' });
      var lastIndex_11 = 0,
          partRe_11 = /\x01\d+\x01/g,
          match_11;
      do {
        match_11 = partRe_11.exec(MSG_EXTERNAL_6047645258311882210) || undefined;
        incrementalDom.text(goog.string.unescapeEntities(MSG_EXTERNAL_6047645258311882210.substring(lastIndex_11, match_11 && match_11.index)));
        lastIndex_11 = partRe_11.lastIndex;
        switch (match_11 && match_11[0]) {
          case '\x010\x01':
            termsOfUse__soy4();
            break;
        }
      } while (match_11);
    }

/cc @4lejandrito

[metal-soy-critic] Recursive configuration

From Recursive configuration by myself:

Currently, mcritic stops at the first occurrence of a config file. This makes it impossible to override default settings in specific areas of the code. Given the following layout:

• root
  • .soycriticrc.json
  • src
  • test
    • .soycriticrc.json

Soy files within the test folder will be linted using the root configuration rather than by the mix of both.

[soyparser] Add a top-level only parse option

From Add a top-level only parse option by @mthadley:

Add a top level parse option. This means instead of trying to fully parse the source file, we will just parse the templates and their params, and ignore any template content. For certain use cases, this should both make the faster and more robust.

So calling parse might look something like this:

import parse from 'soyparser';

parse(input, {
  type: 'top'
});

What's the relationship between this repo and `soyparser`?

https://www.npmjs.com/package/soyparser points to

• https://github.com/metal/soyparser which redirects to
• https://github.com/deprecate/soyparser (which is archived and marked as read-only), and which includes the following note:

Note: the soyparser project has been moved to the metal-tools-soy repo.

To simplify development and maintenance and handle all tools-soy projects, this project has been moved to the soyparser under the umbrella of the metal-tools-soy monorepo.

Thus, we would expect to find the package definition and source files for soyparser in this repo (https://github.com/metal/metal-tools-soy). However, this repo actually has a dependency on the soyparser package:

metal-tools-soy/package.json

Line 47 in dcd8845

"soyparser": "^1.0.3",

This suggests that the source for soyparser wasn't actually migrated, and thus https://www.npmjs.com/package/soyparser package is no longer being maintained, as its source code is no longer tracked anywhere.

Is the source code for soyparser still pending a migration into this repo? It was deleted with a note that it will be migrated in Sep 2018.

Or is soyparser deprecated, along with this repo?

Snapshot Testing

In this project we are applying several transformations on top of what the SoyToIncrementalDomSrcCompiler does.

Examples of such changes are:

• Wrap the templates in a component definition so we can easily import/export them and use them in a metal-component way
• Extract template parameters
• Reformat goog.msg external messages
• ...

Some of our current issues are:

• Using regular replace operations which fail when the output changes and the replace string is no longer found
• Not really using a full Closure-Compiler build makes our usage rely on some globals or other transformations which are hard to test

In order to be able to more confidently update the compiler, we need at least to be able to spot all the output changes that happen so we can adapt accordingly and slowly cover all the features we are actually using.

For this, I'd suggest using https://facebook.github.io/jest/ so we can have a clear understanding of how the different template output is changing as the tool changes.

@bryceosterhaus, @mthadley, you guys might have some more experience with this... what do you think? Would any of you have some time to give this a try?

Soy files with underscores don't work

That's because of this line.

Errors are not being propagated

Currently, soy errors are not signaled properly from metal-cli.

I'm not 100% sure, but it looks like https://github.com/metal/metal-tools-soy/blob/master/lib/pipelines/compileSoy.js#L219 should be:

callback(code);

That seems to fix a higher-level test I've added in metal-cli to assert the output code of a process where soy compilation fails, but I'm not sure how it should be tested, or if it makes sense in this context.

What do you thiknk, @eduardolundgren, @mairatma, @brunobasto, @Robert-Frampton?

Append information about failed soy parsing

Currently, we're simply emitting an error when a soy parsing fails, but we're not saying where did it fail in some cases. We should always append that information. For example:

Running 'build'...
Compiling soy
Error in plugin 'metal-tools-soy'
Message:
    Parsing failed at 5:4. Expecting:
' '
'@param'
a character matching function (ch) {
      return str.indexOf(ch) < 0;
    }

Aggregate Soy Tooling in this repo

We should try to convert this to a lerna monorepo to aggregate all related soy tooling for a simpler tracking and management.

Current Soy Tooling we own includes:

• metal-soy-critic
• soyparser
• gulp-soynode - consider removal?
• metal-tools-soy
• metal-soy-loader
• generate-soy-source-maps

Unsupported major.minor version 52.0

I was trying to build metal-modal and got the error

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/google/template/soy/SoyToIncrementalDomSrcCompiler : Unsupported major.minor version 52.0

Java Version 8 Update 45 (build 1.8.0_45-b14)

`tmp/metal-tools-soy/` folder is always created at the root of the drive

When I build Liferay Portal master, a tmp/metal-tools-soy/ folder is always created at the root of the drive.

I assume this is an issue of metal-tools-soy rather than Portal.

[metal-tools-soy] Add compilation cache for Soy files

I was wondering if make sense to cache compiled files to bump compilation performance.

One rough idea could be to write a cache file that contains the hashes of the files.

.cache

/a.soy 9e107d9d372bb6826bd81d3542a419d6
/b.soy e4d909c290d0fb1ca068ffaddf22cbd03
/c.soy 2bb8209c290d0fb1ca068ffaddf22cbd0

In the next build, before compiling all files, check the cache file and remove from the compilation file list the ones that the hash is still the same, then update the .cache file.

Dependencies

In case a.soy hasn't changed but it imports b.soy, if b.soy has changed a.soy also need to be recompiled. For that, we might use @mthadley tool to parse the files and calculate the dependencies.

Do you guys think it is doable?