[Bug] Packwerk.yml `exclude` option does not work about packwerk

[wildmaples]

@evaldasg Can you post the package.yml file that contains the keys?

The issue you are seeing doesn't seem to be related to Packwerk reading excluded files. It sounds like you didn't have your package.yml file set up properly, and the issue is caught by the validation check.

More specifically, packwerk seems to be picking up the keys [name", "version", "description", "main", "repository", "devDependencies", "license"] all of which are not valid keys for a package.yml file.

[exterm]

ugh, OK! That seems like a problem.

[exterm]

# this always will differ
package_manifests("**/") - package_manifests(some_value_from_packwerk_yml)

That's not true. The intention of the validation is that the package_paths option, which is an optional optimization to speed up analysis, is not set to a value that misses some packwerk packages.

However, I think we may be able to find a way to respect the ignore config that makes sense in this case.

[exterm]

Just tested it and running

      manifests.reject do |manifest_path|
        @configuration.exclude.any? do |exclude_pattern|
          File.fnmatch(File.expand_path(exclude_pattern), manifest_path, File::FNM_EXTGLOB)
        end
      end

Takes virtually no time on our largest app. Now, this app has only 67 packages (most packages are pretty large), but I still think it's a good indication that there won't be a noticeable performance impact from this.

I'd actually be curious if anyone reading this has an app with more packages, and if so how many!

Summarized - let's explore a PR for this. It looks like integrating into PackageSet is the way to go, unless someone has a differing opinion.

[exterm]

Fixed by @jordanstephens in #155 - Thank you Jordan!

[exterm]

@evaldasg are you still looking into this? If not, I would close this issue.

[exterm]

Actually, looking at the example (thank you for building it!) I can see the problem

[exterm]

exclude only influences which source files we check, not where we look for pacakges. This may be a bit confusing; it stems from the time when packwerk was still based on rubocop.

What you want is to influence where packwerk looks for package manifests, or packages in general, which is the package_paths option.

Example:

# Patterns to find package configuration files
package_paths: "./{,components/*/,components/*/*/,eagerlib,eagerlib/*/,test/support/helpers/}"

Note that this is an include list, so you have to state where packwerk should expect packages to live.

Please comment or reopen if this solution doesn't work for you.

[arminas]

@exterm, if I may to jump in, after adding the following line to packwerk.yml:

# Patterns to find package configuration files
package_paths: "./"

Then:

$ bin/packwerk validate

Validation failed ❗
Expected package paths for all package.ymls to be specified, but paths were missing for the following manifests:

node_modules/yaml-js/src/package.yml

It seems like there is a check, that requires all package.ymls to be included in package_paths at https://github.com/Shopify/packwerk/blob/main/lib/packwerk/application_validator.rb#L218.

Maybe this check is excessive and can be removed? Or warn level message, that some package.yml files are ignored, is more appropriate there?

[evaldasg]

As @arminas has pointed there is an issue with quite excessive validation:

  def check_package_manifest_paths
      all_package_manifests = package_manifests("**/")

      package_paths_package_manifests = package_manifests(package_glob)

      difference = all_package_manifests - package_paths_package_manifests

      if difference.empty?
        Result.new(true)
      else
        Result.new(
          false,
          <<~EOS
            Expected package paths for all package.ymls to be specified, but paths were missing for the following manifests:

            #{relative_paths(difference).join("\n")}
          EOS
        )
      end
    end

If the project is configured in packwerk.yml for package_paths different than "**/" it will always fail calling check_package_manifest_paths because of hardcoded global value (package_manifests("**/")) which does not care how packwerk is configured:

# this always will differ
package_manifests("**/") - package_manifests(some_value_from_packwerk_yml)

Example of output from check_package_manifest_paths method's variables in demo app (https://github.com/evaldasg/packwerk-issue):

all_package_manifests 
["/Users/e/projects/packwerk-issue/node_modules/yaml-js/src/package.yml", "/Users/evaldas/projects/packwerk-issue/package.yml"]
package_paths_package_manifests
["/Users/e/projects/packwerk-issue/package.yml"]
difference
["/Users/e/projects/packwerk-issue/node_modules/yaml-js/src/package.yml"]

[evaldasg]

🤷 well, it looks that packwerk.yml "intentions" are hard to understand for me.

Key	Default value	Description
package_paths	**/	a single pattern or a list of patterns to find package configuration files, see: Defining packages

What I expect by setting this option (package_paths) is that packwer won't fail by finding accidentally some package.yml file outside the defined package_paths value.

Even this option:

exclude	{bin,node_modules,script,tmp,vendor}/**/*	list of patterns for folder paths to exclude

is a mystery. I would never expect that packwer would fail with this (since node_modules should be excluded path 🤷 ):

$ bin/packwerk validate

Validation failed ❗
Expected package paths for all package.ymls to be specified, but paths were missing for the following manifests:

node_modules/yaml-js/src/package.yml

[exterm]

I agree, this is a usability issue. We should think about a solution. Maybe just improving the docs could help.

[jordanstephens]

Hi there, just want to add one more voice to this thread because I would like to be able to put bin/packwerk validate in our CI process, but I can't right now because of this issue.

Using the following packwerk.yml file:

include:
- "**/*.{rb,rake,erb}"
exclude:
- "{bin,db,node_modules,script,tmp,vendor}/**/*"
package_paths: "{.,engines/**,packages/**}"

I see the following output:

% bin/packwerk validate
📦 Packwerk is running validation...

Validation failed ❗
Expected package paths for all package.ymls to be specified, but paths were missing for the following manifests:

node_modules/art/lib/slick/package.yml

It looks like there are no configuration options available to convince bin/packwerk validate to return a successful exit code from what I believe to be a completely valid packwerk setup in this workspace.

---

@exterm mentioned above

However, I think we may be able to find a way to respect the ignore config that makes sense in this case.

The most straightforward way to do this that I can think of would be to continue to glob over the whole working directory looking for package.yml files, then ignore any that match the exclude glob patterns in the configuration. Would you accept a PR with something along the lines of the following change?

diff --git a/lib/packwerk/application_validator.rb b/lib/packwerk/application_validator.rb
index e127e1a..949aae9 100644
--- a/lib/packwerk/application_validator.rb
+++ b/lib/packwerk/application_validator.rb
@@ -198,7 +198,11 @@ module Packwerk
     end

     def check_package_manifest_paths
-      all_package_manifests = package_manifests("**/")
+      all_package_manifests = package_manifests("**/").reject do |manifest_path|
+        @configuration.exclude.any? do |exclude_pattern|
+          File.fnmatch(File.expand_path(exclude_pattern), manifest_path, File::FNM_EXTGLOB)
+        end
+      end
       package_paths_package_manifests = package_manifests(package_glob)

       difference = all_package_manifests - package_paths_package_manifests

I know it feels a little messy, but this accomplishes what I believe to be the expected behavior. It's difficult because you can't easily affirmatively exclude things with globs, you can only add things to the matched set.

An alternative might be to enable configuration of the glob ("**/") that goes into that first call to package_manifests. Maybe you could set it to something like ".,packages/**/* or whatever to capture the base package along with some others, but it seems like there is already some confusion about how to use package_paths alongside include and exclude so adding yet another glob pattern to define might make the API even more obtuse.

If there is any appetite for a change along the lines of the above suggestion, I'll be more than happy to open a formal PR and write up some tests for it.

[exterm]

🤔

I think it may be cleaner for check_package_manifest_paths to use PackageSet.package_paths, and for that method to respect the excludes.

PackageSet needs to be fast though (globbing for all ruby files to scan takes a few seconds on our largest code base), so we should be careful not to slow it down too much. Your idea of using fnmatch looks to be much faster than globbing for all excluded files, so it looks promising - we should make sure to test the performance of this on a very large code base though.

I'll accept a PR along those lines if:

• we can show that it doesn't considerably slow things down
• we are reasonably sure it doesn't break existing behavior (will have to try it out)

[jordanstephens]

Excellent! Thanks for investigating this further. I'll aim to submit a PR as soon as I have time—probably within the next week or so.