Comments (15)
@evaldasg Can you post the package.yml
file that contains the keys?
The issue you are seeing doesn't seem to be related to Packwerk reading excluded files. It sounds like you didn't have your package.yml
file set up properly, and the issue is caught by the validation check.
More specifically, packwerk seems to be picking up the keys [name", "version", "description", "main", "repository", "devDependencies", "license"]
all of which are not valid keys for a package.yml
file.
from packwerk.
ugh, OK! That seems like a problem.
from packwerk.
# this always will differ
package_manifests("**/") - package_manifests(some_value_from_packwerk_yml)
That's not true. The intention of the validation is that the package_paths
option, which is an optional optimization to speed up analysis, is not set to a value that misses some packwerk packages.
However, I think we may be able to find a way to respect the ignore
config that makes sense in this case.
from packwerk.
Just tested it and running
manifests.reject do |manifest_path|
@configuration.exclude.any? do |exclude_pattern|
File.fnmatch(File.expand_path(exclude_pattern), manifest_path, File::FNM_EXTGLOB)
end
end
Takes virtually no time on our largest app. Now, this app has only 67 packages (most packages are pretty large), but I still think it's a good indication that there won't be a noticeable performance impact from this.
I'd actually be curious if anyone reading this has an app with more packages, and if so how many!
Summarized - let's explore a PR for this. It looks like integrating into PackageSet
is the way to go, unless someone has a differing opinion.
from packwerk.
Fixed by @jordanstephens in #155 - Thank you Jordan!
from packwerk.
@evaldasg are you still looking into this? If not, I would close this issue.
from packwerk.
Actually, looking at the example (thank you for building it!) I can see the problem
from packwerk.
exclude
only influences which source files we check, not where we look for pacakges. This may be a bit confusing; it stems from the time when packwerk was still based on rubocop.
What you want is to influence where packwerk looks for package manifests, or packages in general, which is the package_paths
option.
Example:
# Patterns to find package configuration files
package_paths: "./{,components/*/,components/*/*/,eagerlib,eagerlib/*/,test/support/helpers/}"
Note that this is an include list, so you have to state where packwerk should expect packages to live.
Please comment or reopen if this solution doesn't work for you.
from packwerk.
@exterm, if I may to jump in, after adding the following line to packwerk.yml
:
# Patterns to find package configuration files
package_paths: "./"
Then:
$ bin/packwerk validate
Validation failed β
Expected package paths for all package.ymls to be specified, but paths were missing for the following manifests:
node_modules/yaml-js/src/package.yml
It seems like there is a check, that requires all package.ymls to be included in package_paths at https://github.com/Shopify/packwerk/blob/main/lib/packwerk/application_validator.rb#L218.
Maybe this check is excessive and can be removed? Or warn
level message, that some package.yml files are ignored, is more appropriate there?
from packwerk.
As @arminas has pointed there is an issue with quite excessive validation:
def check_package_manifest_paths
all_package_manifests = package_manifests("**/")
package_paths_package_manifests = package_manifests(package_glob)
difference = all_package_manifests - package_paths_package_manifests
if difference.empty?
Result.new(true)
else
Result.new(
false,
<<~EOS
Expected package paths for all package.ymls to be specified, but paths were missing for the following manifests:
#{relative_paths(difference).join("\n")}
EOS
)
end
end
If the project is configured in packwerk.yml
for package_paths
different than "**/"
it will always fail calling check_package_manifest_paths
because of hardcoded global value (package_manifests("**/")
) which does not care how packwerk
is configured:
# this always will differ
package_manifests("**/") - package_manifests(some_value_from_packwerk_yml)
Example of output from check_package_manifest_paths
method's variables in demo app (https://github.com/evaldasg/packwerk-issue):
all_package_manifests
["/Users/e/projects/packwerk-issue/node_modules/yaml-js/src/package.yml", "/Users/evaldas/projects/packwerk-issue/package.yml"]
package_paths_package_manifests
["/Users/e/projects/packwerk-issue/package.yml"]
difference
["/Users/e/projects/packwerk-issue/node_modules/yaml-js/src/package.yml"]
from packwerk.
packwerk.yml
"intentions" are hard to understand for me.
Key | Default value | Description |
---|---|---|
package_paths | **/ | a single pattern or a list of patterns to find package configuration files, see: Defining packages |
What I expect by setting this option (package_paths
) is that packwer
won't fail by finding accidentally some package.yml
file outside the defined package_paths
value.
Even this option:
exclude | {bin,node_modules,script,tmp,vendor}/**/* | list of patterns for folder paths to exclude |
---|
is a mystery. I would never expect that packwer
would fail with this (since node_modules
should be excluded path
$ bin/packwerk validate
Validation failed β
Expected package paths for all package.ymls to be specified, but paths were missing for the following manifests:
node_modules/yaml-js/src/package.yml
from packwerk.
I agree, this is a usability issue. We should think about a solution. Maybe just improving the docs could help.
from packwerk.
Hi there, just want to add one more voice to this thread because I would like to be able to put bin/packwerk validate
in our CI process, but I can't right now because of this issue.
Using the following packwerk.yml
file:
include:
- "**/*.{rb,rake,erb}"
exclude:
- "{bin,db,node_modules,script,tmp,vendor}/**/*"
package_paths: "{.,engines/**,packages/**}"
I see the following output:
% bin/packwerk validate
π¦ Packwerk is running validation...
Validation failed β
Expected package paths for all package.ymls to be specified, but paths were missing for the following manifests:
node_modules/art/lib/slick/package.yml
It looks like there are no configuration options available to convince bin/packwerk validate
to return a successful exit code from what I believe to be a completely valid packwerk setup in this workspace.
@exterm mentioned above
However, I think we may be able to find a way to respect the ignore config that makes sense in this case.
The most straightforward way to do this that I can think of would be to continue to glob over the whole working directory looking for package.yml
files, then ignore any that match the exclude
glob patterns in the configuration. Would you accept a PR with something along the lines of the following change?
diff --git a/lib/packwerk/application_validator.rb b/lib/packwerk/application_validator.rb
index e127e1a..949aae9 100644
--- a/lib/packwerk/application_validator.rb
+++ b/lib/packwerk/application_validator.rb
@@ -198,7 +198,11 @@ module Packwerk
end
def check_package_manifest_paths
- all_package_manifests = package_manifests("**/")
+ all_package_manifests = package_manifests("**/").reject do |manifest_path|
+ @configuration.exclude.any? do |exclude_pattern|
+ File.fnmatch(File.expand_path(exclude_pattern), manifest_path, File::FNM_EXTGLOB)
+ end
+ end
package_paths_package_manifests = package_manifests(package_glob)
difference = all_package_manifests - package_paths_package_manifests
I know it feels a little messy, but this accomplishes what I believe to be the expected behavior. It's difficult because you can't easily affirmatively exclude things with globs, you can only add things to the matched set.
An alternative might be to enable configuration of the glob ("**/"
) that goes into that first call to package_manifests
. Maybe you could set it to something like ".,packages/**/*
or whatever to capture the base package along with some others, but it seems like there is already some confusion about how to use package_paths
alongside include
and exclude
so adding yet another glob pattern to define might make the API even more obtuse.
If there is any appetite for a change along the lines of the above suggestion, I'll be more than happy to open a formal PR and write up some tests for it.
from packwerk.
I think it may be cleaner for check_package_manifest_paths
to use PackageSet.package_paths
, and for that method to respect the excludes.
PackageSet
needs to be fast though (globbing for all ruby files to scan takes a few seconds on our largest code base), so we should be careful not to slow it down too much. Your idea of using fnmatch
looks to be much faster than globbing for all excluded files, so it looks promising - we should make sure to test the performance of this on a very large code base though.
I'll accept a PR along those lines if:
- we can show that it doesn't considerably slow things down
- we are reasonably sure it doesn't break existing behavior (will have to try it out)
from packwerk.
Excellent! Thanks for investigating this further. I'll aim to submit a PR as soon as I have timeβprobably within the next week or so.
from packwerk.
Related Issues (20)
- Transitive Dependencies
- [Packwerk 3.0] Update UPGRADING.md with directions to upgrade to 3.0 HOT 1
- [Packwerk 3.0] Ensure API boundaries are up-to-date HOT 1
- [Bug Report] Syntax error when rendering two partials HOT 1
- [Bug Report] Runtime Error after running packwerk check HOT 1
- Packwerk thinks that `app/helpers/foobar_fizz.rb` is in the package `app/helpers/foobar` and not `app/helpers`
- [Bug Report] Stale violations on deleted files HOT 1
- [Bug Report] - Packwerk::NodeHelpers::TypeError HOT 3
- [Bug Report] deprecated_references.yml files erroneously removed if running `packwerk update-deprecations` with the `--package` argument
- [Bug Report] Behaviour of enforce_dependencies: key missing HOT 1
- [Bug Report] Improve strict mode HOT 2
- [Bug Report] Insufficient release notes HOT 2
- [Bug Report] packwerk is disabling Sorbet runtime checks, globally on Spring
- [Bug Report] Disable sorbet when using spring
- [Feature Request] Detect stale todos HOT 1
- [Bug Report] Exception when loading signatures
- [Bug Report] Default require behaviour
- [Bug Report] Dependency violation not detected when using public folders or in non-public folder with dependencies specified HOT 1
- [Feature Request] Allow configuration of `package_todo.yml` violation destinations HOT 4
- [Feature Request] Separate run-time and test-time dependencies HOT 11
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from packwerk.