shopify / packwerk Goto Github PK
View Code? Open in Web Editor NEWGood things come in small packages.
License: MIT License
Good things come in small packages.
License: MIT License
Description
bundle exec packwerk
and bin/packwerk
produces different result. bin/packwerk
loads files while bundle exec
does not.
To Reproduce
Try bundle exec packwerk validate
and bin/packwerk validate
packwerk validate
requires files to be loaded, so it will fail on the bundle exec
command.
Expected Behaviour
packwerk validate
requires files to be loaded, so it will fail on the bundle exec
command.
Screenshots
If applicable, add screenshots to help explain your problem.
Version Information
Additional Context
The solution is to move the file loading logic from bin/packwerk
script into ApplicationLoadPath
. ApplicationLoadPath
is only used in packwerk init
and packwerk validate
commands, the two commands that require Rails to be booted.
For additional context, see: #77 (comment)
It'd be nice to be able to include slim files, just like erb files are supported. Slim is actually built on top of Temple which is a generic templates parser/compiler. Maybe it would even be worth it to replace the existing erb parser with a temple parse as it includes erb built-in and you'd be getting support for many others all at once.
[Enhancement]
We are just beginning to use Packwerk and loving its power! However, it doesn't allow us to follow a common pattern in our existing packaged code. I would like this feature and wanted to see if you'd accept the PR or if we should think about forking.
We commonly follow this directory structure
app/
domains/
identity_domain/
user.rb
package.yml
identity_domain.rb
package.yml
In this structure, the root module sharing its name with the package serves as the entry point to the domain:
# app/domains/identity_domain.rb
module IdentityDomain
def self.get_user(id)
IdentityDomain::User.find(id)
end
end
This seems to conflict with Packwerk, because AFAICS there is no way to tell it to treat the root module as public. The best we can come up with is moving the methods to e.g. IdentityDomain::Public::API
but this feels quite verbose. I envision solving this with a per package opt-in like:
# app/domains/identity_domain/package.yml
enforce_privacy: true
public_root: true
This seems to overlap with the conversation in #98 but with a much narrower focus. I'm not talking about allowing arbitrary public entrypoints, but rather the option to treat the root module as a public constant.
As I say, would you consider a PR for this? Do you consider this an anti-pattern perhaps?
[EXPLORE]
Hi all!
I have been spinning the wheels on packwerk and love it so far!
Regarding privacy - when listing privacy constraints by constant, I would like to list the public files in the config so any new constants are private by default. Right now I can only use enforce_privacy which will make new constants public by default. With the new name I propose, enforce_api_boundary, the behavior of true could stay the same.
Is this something you would accept a PR for?
@exterm brought up that packwerk update
should actually be called packwerk update-deprecations
and I totally agree. packwerk update
can be confusing.
We should add a packwerk update-deprecations
command and add a deprecation warning for the packwerk update
command but still allow people to use it.
When you want to debug Packwerk for a specific context (file + constant), it's quite difficult to do without running a debugger for all contexts. What if we dropped into a debugger whenever we see an AST node that is a call, for a method named debugger, and no arguments?
These are generally not found in code that gets committed, so it should be non-intrusive, but best to scope it behind a flag/subcommand to be safe.
Description
In a componentised monolith, where should specs live to avoid causing privacy errors?
To Reproduce
Create a new Rails project, create a component with a private file like components/finance/app/models/account.rb
.
Write a spec for that model, where would you put it?
Expected Behaviour
Documentation should explain how to deal with tests as this is a very common issue.
Version Information
Additional Context
Two approaches comes immediately to mind, but none of them is straightforward:
components
subdirectory in your spec
folder and put the model test there (/spec/components/finance/models/account_spec.rb
). This won't work because the model above is private, so directly referencing it from spec/components/...
causes a privacy violation that needs to be ignored.spec
subdirectory of your component (components/finance/spec/models/account_spec.rb
). This makes packwerk check
pass, but RSpec won't look for tests in that folder by default. I managed to tweak RSpec
to do so locally (with --default-path
and --pattern
options), but this solution is hacky and our CI, that relies on Knapsack
to run parallel tests, doesn't work as expected.How do you do this in your packwerk-powered projects?
Description
This file will trip up packwerk: https://github.com/activeadmin/activeadmin/blob/master/spec/unit/belongs_to_spec.rb
I suspect this stems from the belongs_to association helper.
If you run into an issue like this, I recommend excluding the file via packwerk.yml.
To Reproduce
Add this file to your repo and packwerk will crash.
Expected Behaviour
I think it is fine for packwerk to not be able to parse a file as unusual as this one.
Description
Memory usage in packwerk v2.0 is significantly higher than it was in v1.4—so much higher that we will need to increase the resource class of our CI nodes in order for us to run packwerk v2.0. It looks like the max RSS for the whole process tree has increased from ~850 MB to about ~2.5GB for our use-case.
I ran gnu time -v
on bin/packwerk check
in our application working directory using both versions of packwerk and saw the following results:
Version 1.4 | Version 2.0 |
|
|
To Reproduce
Run packwerk while watching memory usage, compare between v1.4 and v2.0
Expected Behaviour
I expected memory usage to not increase so dramatically between v1.4 and v2.0.
Screenshots
The GNU time output only captures the max RSS of the root process, so I grabbed a couple screenshots from activity monitor to demonstrate that the memory usage is much higher in each of the child processes as well.
Version Information
Additional Context
I haven't looked into the changes yet, but I assume this is because packwerk now loads our rails application in each of its processes? I still wouldn't expect each child process's max RSS to balloon up so much.
Do you all have any idea why this might be? And secondly, any idea if it would be possible and reasonable to bring this number back down?
Thanks!
This is a copy-paste of @tomstuart's bug report, originally posted to https://github.com/Shopify/packwerk-old/issues/289
Describe the bug
When Packwerk::ConstNodeInspector
finds a constant in a class or module definition, it uses the lexical nesting to fully-qualify that constant’s name. This is not always correct because a namespace may refer to a constant that has already been defined elsewhere, e.g. in another package.
This bug is a variation on #234: this time it’s a reference to an existing constant elsewhere in the enclosing namespace, rather than to the root namespace, which causes the problem.
To Reproduce
Say we have a pair of components, one
and two
, which define the modules A::B
and A::C
respectively:
% tree components
components
├── one
│ ├── a
│ │ └── b.rb
│ └── package.yml
└── two
├── a
│ └── c.rb
└── package.yml
Both package.yml
files have enforce_dependencies
and enforce_privacy
set to true
and no dependencies
defined, so Packwerk should not allow either component to refer to the other.
In addition to defining A::B
, the file one/a/b.rb
also defines a nested class A::B::D
:
module A
module B
class D
def info
'defined inside ::A::B by component one'
end
end
end
end
And in addition to defining A::C
, the file two/a/c.rb
reopens that class and overwrites one of its methods:
module A
module C
class B::D
def info
'defined inside ::A::C by component two'
end
end
end
end
It might not be immediately obvious that two
is redefining A::B::D#info
here, but it is, because the reference to B
in class B::D
gets resolved to the existing constant A::B
, regardless of the definition occurring lexically inside module C
. (This is what #234 is about.)
So when these two components are loaded, two
refers to (and modifies) the class A::B::D
from one
:
% irb -Icomponents/one -Icomponents/two
>> require 'a/b'
=> true
>> require 'a/c'
=> true
>> A::B::D.new.info
=> "defined inside ::A::C by component two"
=> nil
packwerk check
doesn’t detect this violation because it doesn’t realise that class B::D
is a reference to A::B::D
. It assumes it must refer to A::C::B::D
because it appears inside module A; module C; …; end end
, but Ruby constant resolution is more complex.
This doesn’t present an immediate problem within Shopify because we don’t use compact constant nesting (class B::D
) in class definitions, but Packwerk may give incorrect results on external codebases.
Expected behavior
packwerk check
should give an error like this:
Dependency violation: ::A::B::D belongs to 'components/one', but 'components/two' does not specify a dependency on 'components/one'.
Are we missing an abstraction?
Is the code making the reference, and the referenced constant, in the right packages?
Inference details: 'B::D' refers to ::A::B::D which seems to be defined in components/one/a/b.rb.
Version information
Description
When adding Packwerk to an existing project an error was thrown during packwerk check
.
This seems to be due to a namespaced constant which is assigned a lambda expression, like Foo::Bar = -> {}
.
A mitigation is to exclude the offending file in packwerk.yml
so that the rest of the codebase can be analysed as usual. However it's not obvious which file was being processed when the error occurred.
To Reproduce
Add a namespaced constant which is assigned a lambda expression to any file which Packwerk parses, for example:
# foo.rb
Foo::Bar = -> {}
Then run bundle exec packwerk check
as usual.
Expected Behaviour
Packwerk should parse and analyse the lambda expression like any other code block.
Screenshots
/Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:224:in `type_of': undefined method `type' for nil:NilClass (NoMethodError)
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:101:in `constant?'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:267:in `module_creation?'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:163:in `module_name_from_definition'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/const_node_inspector.rb:43:in `constant_in_module_or_class_definition?'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:90:in `block in create_validator_slow'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/const_node_inspector.rb:23:in `constant_name_from_node'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:90:in `block in create_validator_slow'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/reference_extractor.rb:33:in `block in reference_from_node'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/reference_extractor.rb:32:in `each'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/reference_extractor.rb:32:in `reference_from_node'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_processor.rb:28:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation_2_6.rb:764:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation_2_6.rb:764:in `block in create_validator_method_medium2'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:12:in `visit'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:17:in `block in visit'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:63:in `block in each_child'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:62:in `each'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:62:in `each_child'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:16:in `visit'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:17:in `block in visit'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:63:in `block in each_child'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:62:in `each'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:62:in `each_child'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:16:in `visit'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/file_processor.rb:49:in `references_from_ast'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/file_processor.rb:37:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation_2_6.rb:703:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation_2_6.rb:703:in `block in create_validator_method_medium1'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/run_context.rb:56:in `process_file'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:90:in `block in create_validator_slow'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/parse_run.rb:66:in `block in find_offenses'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:515:in `call_with_index'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:485:in `process_incoming_jobs'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:465:in `block in worker'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:456:in `fork'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:456:in `worker'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:447:in `block in create_workers'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:446:in `each'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:446:in `each_with_index'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:446:in `create_workers'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:386:in `work_in_processes'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:289:in `map'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:302:in `flat_map'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/parse_run.rb:74:in `block in find_offenses'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/parse_run.rb:72:in `find_offenses'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/parse_run.rb:45:in `check'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/cli.rb:53:in `execute_command'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/cli.rb:40:in `run'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/exe/packwerk:12:in `<top (required)>'
from /Users/me/.asdf/installs/ruby/2.6.9/bin/packwerk:23:in `load'
from /Users/me/.asdf/installs/ruby/2.6.9/bin/packwerk:23:in `<top (required)>'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli/exec.rb:58:in `load'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli/exec.rb:58:in `kernel_load'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli/exec.rb:23:in `run'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli.rb:478:in `exec'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli.rb:31:in `dispatch'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli.rb:25:in `start'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/exe/bundle:49:in `block in <top (required)>'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/exe/bundle:37:in `<top (required)>'
from /Users/me/.asdf/installs/ruby/2.6.9/bin/bundle:23:in `load'
from /Users/me/.asdf/installs/ruby/2.6.9/bin/bundle:23:in `<main>'
Version Information
Additional Context
A constant which isn't namespaced, like Foo = -> {}
seems to be fine. Using Foo:Bar = Proc.new {}
seems to be fine.
It would be helpful if a parsing error included the name of the file being parsed.
In Zeitwerk, the load path is list of directories that it calls "root directories". They are "root" because any files that reside in them are expected to represent modules that are part of the root namespace. If an object is nested in some more specific namespace, it needs to be placed in a subdirectory of one of these roots. For example, in the following, lib
is a "root directory" with the following structure and the ruby files have the noted expected constants:
.
├── lib
│ ├── my_namespace
│ │ └── my_file.rb # MyNamespace::MyFile
│ └── my_other_file.rb # MyOtherFile
The thing is, Zeitwerk also supports setting a "default namespace" to each root directory via push_dir
which is called when the Zeitwerk loader is being configured. So in the above, it was assumed lib
was added to the loader via loader.push_dir("lib")
but it could also be added with loader.push_dir("lib", MyCompany)
which means the constants inside would be expected to be declared as MyCompany::MyNamespace::MyFile
and MyCompany::MyOtherFile
.
At GitHub we rely on this feature to work around some extremely legacy directories in our load path that do not conform to Zeitwerk's expectations. Up until now, we have worked around them in Packwerk by skipping some and "hacking" at others. At this point, we'd like to just add support for custom namespaces to Packwerk so we can be confident in Packwerk's operation.
While I haven't dived into the Packwerk's source for this yet, I suspect this would mean moving Packwerk::ApplicationLoadPaths
from config.autoload_paths
, eager_load_paths
, and autoload_once_paths
to Rails.autoloaders
to get our list of root directories as a hash of paths to modules. By default, these modules would all just be Object
as they are in Zeitwerk, but it would allow us to override that default as needed for certain paths.
I realize this would also necessitate a change to ConstantResolver
as well and have a WIP branch here. This branch uses strings instead of module instances for the namespaces in order to keep us as disentangled from a dependency standpoint as possible.
While I realize this is a change that is probably not needed for most codebases, I think that improving Packwerk's support for Zeitwerk's configuration is overall a positive.
So at this point, before I go ahead with a PR for Packwerk, what do folks think about adding such support?
To simplify our lives, let's assume that files are Ruby if they aren't recognized by any other parser (for now, that's just .erb). This will eliminate encountering "unknown file type" errors.
If assuming Ruby fails, we'll have to encourage people to use the include / exclude globs in their configs to handle other types of files.
We could assume a file without an extension is an executable, check to see if it looks like a Ruby executable, and ignore it if it's not. @thegedge prefers being explicit, but can understand that a larger repo may have built up a lot of executable files, many of which may not be Ruby.
Description
When packwerk update
is run (without any loaded files) from the command line, Packwerk fails to interprets some constant association.
To Reproduce
uncountable:
- 'crazy_details'
has_many :crazy_details
Run packwerk update
to get Packwerk to update the shitlist with a violation. You will see that Packwerk does not recognize the association.
Run bin/packwerk update
to get Packwerk to update. You will see that the violation is captured because the files are loaded through bin/packwerk
script.
Expected Behaviour
See 4 above.
Screenshots
If applicable, add screenshots to help explain your problem.
Version Information
Additional Context
Potential solutions:
packwerk update
runsPrivacy violations in migrations seem to be caught by Packwerk.
We suspect the t.belongs_to(:order)
is what is setting off the violation, as Packwerk thinks it is an active record association.
To get rid of these type of violations, users have been putting this violation in to the deprecated_references.yml
list.
There may be two solutions:
While trying to fix a parsing error shown on Packwerk, we had to run rubocop locally since Packwerk did not display any specifics of the error (line number, for example).
RuboCop::AST::Node#const_name
doesn’t reveal whether a constant is fully qualified (e.g. ::HELLO
) or not:
>> RuboCop::ProcessedSource.new('::HELLO', 2.6).ast.const_name
=> "HELLO"
>> RuboCop::ProcessedSource.new('HELLO', 2.6).ast.const_name
=> "HELLO"
As a result, the implementation of ParsedConstantDefinitions#collect_local_definitions
is unaware of whether a constant is fully qualified, so it’s unable to put it in the correct namespace:
>> definitions = ParsedConstantDefinitions.new(
root_node: parse_code('module Sales; ::HELLO = "World"; end')
)
=> #<Packwerk::ParsedConstantDefinitions
@local_definitions=
{"::Sales"=>#<Parser::Source::Range (string) 7...12>,
"::Sales::HELLO"=>#<Parser::Source::Range (string) 16...21>}>
>> definitions.local_reference?('HELLO')
=> false # should be true
>> definitions.local_reference?('Sales::HELLO')
=> true # should be false
I’m not going to fix this right now because I’m trying to make progress on a different task, but I think it’s a bug that’s going to affect users.
cc: @tomstuart
My CI vendor
folder contents is slightly different from my local environment. This produces different load_paths
value in packwerk.yml
causing this failure "Load path cache in #{@config_file_path} incorrect!"
Is there any way to get around this cache issue? I see the exclude
option does not affect the load paths at all. I'm not sure that it makes sense for it to, although I'm never going to define anything in vendor
as a package.
I also have issues with ETA: Actually, these are valid violations since the classes are defined twice.check
because it causes the ERROR: 'X' could refer to any of
error since some model classes are defined in a gem and also the Rails application. I do not have these issues locally since vendor
is not in the load_paths
.
I should note that I'm using Rails 5.2.4
Description
When I run packwerk init
or bundle exec packwerk init
I receive an uninitialized constant error:
29: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/bin/bundle:23:in `<main>'
28: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/bin/bundle:23:in `load'
27: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.6/exe/bundle:37:in `<top (required)>'
26: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/friendly_errors.rb:130:in `with_friendly_errors'
25: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.6/exe/bundle:49:in `block in <top (required)>'
24: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:24:in `start'
23: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
22: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:30:in `dispatch'
21: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
20: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
19: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
18: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:494:in `exec'
17: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:28:in `run'
16: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:63:in `kernel_load'
15: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:63:in `load'
14: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/bin/packwerk:23:in `<top (required)>'
13: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/bin/packwerk:23:in `load'
12: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/exe/packwerk:6:in `<top (required)>'
11: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/_methods.rb:222:in `block in _on_method_added'
10: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/call_validation.rb:126:in `validate_call'
9: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/call_validation.rb:126:in `call'
8: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/cli.rb:50:in `run'
7: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/_methods.rb:222:in `block in _on_method_added'
6: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/call_validation.rb:126:in `validate_call'
5: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/call_validation.rb:126:in `call'
4: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/cli.rb:59:in `execute_command'
3: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/cli.rb:96:in `init'
2: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/generators/application_validation.rb:9:in `generate'
1: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/generators/application_validation.rb:21:in `generate'
/Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/generators/application_validation.rb:31:in `generate_packwerk_validate_script': uninitialized constant Packwerk::Generators::ApplicationValidation::FileUtils (NameError)
The relevant line of code is here. I noticed there wasn't a require 'fileutils'
anywhere in the codebase, so adding one to line 4 of exe/packwerk
resolved this for me. I feel like there should be a require somewhere inside the gem, but I don't know why this doesn't break inside Docker w/ the same ruby version.
Things I tried as workarounds that reproduced/exhibited same behavior:
gem install packwerk
and adding it to top level of Gemfile
and using bundle exec
spring stop
in all instances (and DISABLE_SPRING=1
)Things look fine/not able to reproduce with:
To Reproduce
Ruby 2.6.6 (p146)/macos (intel)/packwerk 1.1.2/Rails 6.0.3.5:
rails new foo_bar && cd foo_bar && packwerk init
Expected Behaviour
Init should complete successfully
Screenshots
N/A
Version Information
Additional Context
I do have a workaround for this, so not urgent. Behavior is strange so appreciate any confirmations of this behavior. Would normally do a quick PR for this but the fact that I can't reproduce it except on my machine has me wanting to confirm the behavior first. I feel like I'm missing something...
Description
Packwerk can't parse a this file:
class Foobar
self::SOMETHING = 'foo'
end
To Reproduce
Create a ruby file with the contents above and have Packwerk check it.
Expected Behaviour
Not error
Version Information
Description
Currently Packwerk can only check constant references that are being autoloaded. This excludes any dependencies and in-repo gems that could exist in a codebase. Especially for in-repo gems we have a strong need for boundary checking.
To Reproduce
Add a package.yml with enforced privacy or dependency violations and run packwerk update
or packwerk check
, neither of these will work for non-autoloaded packages.
Expected Behaviour
Dependencies and other packages that are not autoloaded should still have boundaries enforced.
Version Information
Since we're migrating to our own executable, instead of Rubocop, it would make a lot more sense for the "check" subcommand to simply do this. If it adds too much runtime cost to check
, we can also do it as a separate subcommand (e.g., validate
).
Description
The cache periodically (rarely) fails in our CI system with the error shown in the screenshots section.
We suspect that the packwerk cache has a race condition when forked processes are initializing the cache independently.
Namely -- after one process has bust the cache and created the cache directory, another process can then bust the cache, after which the original process may try to write the cache contents of inflections.rb
. This can result in the bug:
Errno::ENOENT: No such file or directory @ rb_sysopen - tmp/cache/packwerk/inflections
To Reproduce
We are not able to reliably reproduce this bug.
Expected Behaviour
We expect the cache to not raise an error. From an implementation perspective, we expect tmp/cache/packwerk
to already exist before attempting to write the cache file.
Version Information
Additional Context
Some potential solutions we're looking at to solve this issue:
Cache
within the parent process to ensure that the call to FileUtils.rm_fr
is only run once at the start. This is what we went with in this PR: #183rm_fr
entirely, and use a strategy that hashes based on the digest of the current inflections.Fail check if shitlisted violation is removed or not updated.
We have a fix for this bug on Shopify/shopify - https://github.com/Shopify/shopify/pull/250880. However, we need to introduce a permanent fix for other repositories.
Options:
Have packwerk update --exit-status
or maybe packwerk update --check-stale
, which returns an exit status: 0 if there are no stale violations; and 1 if there is a stale violation. This can be easily used on CI to ensure there are no stale violations. The flag will run packwerk update
and check to see if there is a difference in the shitlists. Though I am concerned that this check would also be flagging new violations as stale and encouraging users to just run packwerk update
to update the shitlist.
Another option is to have packwerk check
understand stale references. This would require more refactoring and unifying the idea of Shitlists
and UpdatingShitlists
. This option is preferred as the former would cost us some memory.
Description
Some gems (e.g., http://github.com/github/graphql-client) provide ERB extensions which add or modify the standard ERB syntax. To support apps which use those gems, Packwerk should allow for customizing the ERB implementation.
To Reproduce
Expected Behaviour
Packwerk should provide a way to customize the ERB implementation.
Screenshots
N/A
Version Information
Description
packwerk
does not exclude configured folders for exclusion.
To Reproduce
Demo rails
app: https://github.com/evaldasg/packwerk-issue
Install npm package:
yarn add yaml-js
packwerk configured to exclude node_modules
exclude:
- "{bin,node_modules,script,tmp,vendor}/**/*"
packwerk validation fails:
➜ packwerk-issue git:(main) bin/packwerk validate
📦 Packwerk is running validation...
Validation failed ❗
Unknown keys in /packwerk-issue/node_modules/yaml-js/src/package.yml: ["name", "version", "description", "main", "repository", "devDependencies", "license"]
If you think a key should be included in your package.yml, please open an issue in https://github.com/Shopify/packwerk
Expected Behaviour
If packwerk.yml
configured with exclude
option, it is expected that packwerk
will skip those paths.
Version Information
Additional Context
Suspected code: https://github.com/Shopify/packwerk/blob/main/lib/packwerk/package_set.rb#L37
def package_paths(root_path, package_pathspec)
bundle_path_match = Bundler.bundle_path.join("**").to_s
glob_patterns = Array(package_pathspec).map do |pathspec|
File.join(root_path, pathspec, PACKAGE_CONFIG_FILENAME)
end
Dir.glob(glob_patterns)
.map { |path| Pathname.new(path).cleanpath }
.reject { |path| path.realpath.fnmatch(bundle_path_match) }
end
This rejects only gems
directory but does not take @configuration.exclude
into account.
Description
Running bundle exec packwerk init
is failing while creating application_load_paths due to Sorbet error: Return value: Expected type T::Array[String], got T::Array[T.any(Pathname, String)]
To Reproduce
This may be repo dependent, but running bundle exec packwerk init
failed on the Shopify Flow repo.
Expected Behaviour
No error.
Screenshots
If applicable, add screenshots to help explain your problem.
Version Information
Additional Context
I found a workaround to get it working and was able to initialize, validate, and execute packwerk checks.
I changed
(engine.config.autoload_paths + engine.config.eager_load_paths + engine.config.autoload_once_paths).uniq
to:
(engine.config.autoload_paths + engine.config.eager_load_paths + engine.config.autoload_once_paths).map(&:to_s).uniq
I can submit a PR with this change if those with more packwerk context think this is the correct change to make.
Description
Attempting to initialize/setup packwerk, but hitting a relative_path_from
issue.
To Reproduce
Expected Behaviour
Version Information
Additional Context
➜ web git:(master) ✗ be packwerk init
📦 Initializing Packwerk...
📦 Generating application validator...
✅ Packwerk application validation bin script generated in /Users/jasonlor/Documents/GitHub/app/web/bin
Version: 2.1.1
Usage: spring COMMAND [ARGS]
Commands for Spring itself:
binstub Generate Spring based binstubs. Use --all to generate a binstub for all known commands. Use --remove to revert.
help Print available commands.
server Explicitly start a Spring server in the foreground
status Show current status.
stop Stop all Spring processes for this project.
Commands for your application:
rails Run a rails command. The following sub commands will use Spring: console, runner, generate, destroy, test.
rake Runs the rake command
Traceback (most recent call last):
9: from bin/packwerk:22:in `<main>'
8: from bin/packwerk:22:in `new'
7: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/cli.rb:23:in `initialize'
6: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:18:in `from_path'
5: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:18:in `new'
4: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:45:in `initialize'
3: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:61:in `all_application_autoload_paths'
2: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:61:in `map'
1: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:65:in `block in all_application_autoload_paths'
/Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/2.6.0/pathname.rb:522:in `relative_path_from': different prefix: "" and "/Users/jasonlor/Documents/GitHub/app/web" (ArgumentError)
In test files, when we see:
class MyTest < ActiveSupport::TestCase
setup do
@thing = things(:thing_fixture)
end
end
We want to extract the constant name ::Thing
from the call things(:thing_fixture)
.
Tests should also be appropriately packaged (in other words, no boundary violations), and fixtures are an easy way for packages to (currently) depend on other packages without packwerk realizing. By registering these constant references we have a better signal to help packwerkers have proper isolation for their tests.
I'm thinking this will be a few PRs:
ReferenceExtractor
to accept a list of instances of anything that implements the "constant name from node" interface in (1).ReferenceExtractor
(the node.const_node?
path).For (4), to reduce false positives, I'm thinking the criteria for a "fixtures function call" would be the following:
**/test/**/*_test.rb
or **/spec/**/*_spec.rb
:send
with one or more arguments that are all symbols, and_fixture.model_class
, if it exists, otherwise just use the inflector on the filename).cc: @thegedge
We currently extract + filter references all in one step, which can sometimes be a point of confusion.
In particular, we do a lot of filtering:
packwerk/lib/packwerk/reference_extractor.rb
Lines 50 to 71 in c6b06b8
You can see we remove
There could be a light refactor here so that we can extract all references, followed by another layer to do the filtering. It should lend itself to easier testing and a lighter set of test suites.
Description
When running bundle exec packwerk init
on a fresh install, a Sorbet TypeError was thrown when generating the configuration file. This is due to a missing load_paths
value, for which Sorbet expects to receive an Array.
To Reproduce
bundle exec packwerk init
in a project that doesn't yet have a packwerk config fileExpected Behaviour
All init
generators should run without failure.
Version Information
Additional Context
This problem seems to have been introduced in #68, specifically at https://github.com/Shopify/packwerk/pull/68/files#diff-e43184d1a960a3c3821da21b3eb72c83379c2dcc996f6bcfac9aadd6898cd374L45.
Error details:
Parameter 'load_paths': Expected type T::Array[String], got type NilClass (TypeError)
Caller: /Users/mikelkew/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6084/lib/types/private/methods/call_validation.rb:78
Definition: /Users/mikelkew/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/packwerk-1.0.2/lib/packwerk/generators/configuration_file.rb:21
Description
Is there interest in having support for custom zeitwerk inflections added? Is it just a matter of "writing the code" or is there known blockers for it? Considering submitting a PR but wanted to check on it first...
When all the violations are resolved from a package, packwerk update
does not delete the file, leaving the file stale. packwerk update
should somehow be able to remove stale files completely.
We have ApplicationValidatorTest
which are integration tests. However, we should find a way to unit test the individual checks.
We could make unit testing this file easier by refactoring it so a fake filesystem can be injected.
This would make it easier to test but would require a larger refactor. https://github.com/Shopify/packwerk-old/pull/262#discussion_r455956863
TO DO:
Description
I am relatively new to Packwerk and working on building out different modularized portions of our application. As I am slowly chipping away at the backlog of packages needed, there are a lot of dependencies from the original structure of our application.
For example, I am building out a package that references a specific Model in the original app/models
location.
Package:
app/packages/home_operations
app/packages/home_operations/services/cleaning.rb
Constant:
::Job
Expected Behaviour
Instead of declaring the entire package or an entire directory, I'd like to be able to simply say ::Job
is a dependency, creating a much more concise dependency declaration.
Version Information
Additional Context
When running packwerk update-deprecations
, this is the output added to my deprecated-references.yml
:
".":
"::Job":
violations:
- dependency
files:
- app/packages/home_operations/services/cleaning.rb
I don't see anything in the documentation\USAGE.md that would seem to point to the expected formatting of the dependencies
entry in package.yml
.
Any guidance would be greatly appreciated!
We would like to refactor how updating the shitlist work.
https://github.com/Shopify/packwerk-old/pull/181#pullrequestreview-417116089
Base of @thegedge's suggestion we have at least two options to explore:
Change the behaviour of RunContext
, and we returned a different node processor for updating versus checking.
File and node processor work together to yield references, and some other class (e.g., Cli) would then process the references differently based on whether we're checking or updating
We currently can dump a lot of information at the user:
/Some/path/foo.rb:136:8
Privacy violation: '::Foo::Bar' is private to 'foo' but referenced from 'path'.
Is there a public entrypoint in 'foo/app/public/' that you can use instead?
Inference details: 'Foo::Bar' refers to ::Foo::Bar which seems to be defined in foo/app/foo/models/bar.rb.
/Some/path/foo.rb:151:8
Privacy violation: '::Foo::Spam' is private to 'foo' but referenced from 'path'.
Is there a public entrypoint in 'foo/app/public/' that you can use instead?
Inference details: 'Foo::Spam' refers to ::Foo::Spam which seems to be defined in foo/app/models/foo/spam.rb.
/Some/path/foo.rb:161:8
Privacy violation: '::Foo::Eggs' is private to 'foo' but referenced from 'path'.
Is there a public entrypoint in 'foo/app/public/' that you can use instead?
Inference details: 'Foo::Eggs' refers to ::Foo::Eggs which seems to be defined in foo/app/models/foo/eggs.rb.
There's a huge amount of repetition in there, and that's mostly because we were trying to fit the rubocop mold. Let's investigate grouping these together in a more concise and informative way.
Here's one idea:
# Privacy violations
These references appear to reference a private constant in relevant packages. You should look for a publicly accessible constant (<link> to learn more).
- /Some/path/foo.rb:136:8 references `Foo::Bar` in package 'foo'
- /Some/path/foo.rb:151:8 references `Foo::Spam` in package 'foo'
- /Some/path/foo.rb:161:8 references 'Foo::Eggs' in package 'foo'
# Dependency violations
...
Description
Currently the public folder is hard-coded as app/public
, which may not work for all users of the gem. They may wish to define their own public folder, and if none is defined, default to app/public
.
Why?
Depending on the architecture of the application, app/public
might just not make sense, or at least not be cohesive with the rest of the folder structure. Consider a layered architecture for example, where you break up the layers of your application via technical partition (i.e. presentation layer, domain layer, infrastructure layer etc.). Your top-level folders in your package would typically be /presentation
, /domain
etc., so logically in that architecture you would want to select the layer that makes sense to expose the package from, and app/public
conflicts with that vision.
I've raised a PR here that I believe introduces that feature, and seems to work nicely.
As @doodzik mentioned in https://github.com/Shopify/packwerk/pull/61/files#r517545831, we should look into refactoring cli.rb
as it is growing to be a large class.
We should move all of the different commands we have in to compositional objects, so we can just do something like command.run
. Breaking this class up would make it easier to have unit tests.
I would also be curious to see how RuboCop implements their Cli class.
Currently, all nested packages are accessible by any other package. We may want to reconsider this model so that nested packages are not "exported" by default, unless explicitly requested by the parent package:
# tree
.
└── foo
├── package.yml
└── bar
└── package.yml
# cat foo/package.yml
enforce_privacy: true
enforce_dependencies: true
dependencies:
- "foo/bar"
export_packages:
- "foo/bar"
This would allow those who depend on foo
to be able to also access bar
, otherwise any access to or dependency on bar
would be a violation. If we want to export a subset of bar
to the rest of the world, we would use aliasing:
# cat foo/app/public/foo/stuff.rb
module Foo
Stuff = ::Foo::Bar::Stuff
end
We've briefly described a project where packages could put more constraints on incoming edges (in the dependency graph).
@thegedge sees two possibilities for those constraints: a whitelist option or a blacklist option (not both). In this case, we would whitelist the parent package in the nested package, which means any other package would be a violation.
Make packwerk
compatible with buildkite parallelism. We could add a --parallelism_count
and --parallelism_index
(or some other names) and have buildkite set up to pass the env vars into those flags. This means no need to create another binstub for CI, and also means we can work with any CI system.
Description
When I enforce privacy for specific constants, a dependency violation to a public constant in that package is correctly detected. As soon as I reconfigure the package and set enforce_privacy: true
and move the public constant to the package's public folder the dependency violation is not detected anymore.
I expect that the way how I declare public/private constants doesn't influence the dependency check/violations.
To Reproduce
git clone [email protected]:Enceradeira/rails_problem_packwerk.git
bundle install
git checkout working
packwerk check
app/models/accounting/accounts.rb:6:8
Dependency violation: ::Debtors::DebtorsService belongs to 'app/models/debtors', but 'app/models/accounting' does not specify a dependency on 'app/models/debtors'.
Are we missing an abstraction?
Is the code making the reference, and the referenced constant, in the right packages?
Inference details: this is a reference to ::Debtors::DebtorsService which seems to be defined in app/models/debtors/debtors_service.rb.
To receive help interpreting or resolving this error message, see: https://github.com/Shopify/packwerk/blob/main/TROUBLESHOOT.md#Troubleshooting-violations
1 offense detected
No stale violations detected
git checkout not_working
packwerk check
No offenses detected
No stale violations detected
app/models/debtors/package.yml
git diff working not_working | cat
diff --git a/app/models/debtors/package.yml b/app/models/debtors/package.yml
index 619e57e..958af2c 100644
--- a/app/models/debtors/package.yml
+++ b/app/models/debtors/package.yml
@@ -1,5 +1,5 @@
-enforce_privacy:
- - "::Debtors::Enquiries"
+enforce_privacy: true
+
enforce_dependencies: true
dependencies:
- app/models/accounting
\ No newline at end of file
diff --git a/app/models/debtors/debtors_service.rb b/app/models/debtors/public/debtors_service.rb
similarity index 100%
rename from app/models/debtors/debtors_service.rb
rename to app/models/debtors/public/debtors_service.rb
Expected Behaviour
Example 2 should produce the same result as example 1. The way how I define the privacy boundary should not influence which dependency violations are detected.
Version Information
Currently in our monolith we are running packwerk check
in our CI. This is great and helps us manage our technical in a controlled manner. However, if a team removes some deprecated references and forgets to run packwerk update-deprecations
there is no CI failure and the associated deprecated_references.yml
file just gets a bit out of date with reality since we don't run packwerk detect-stale-violations
in CI.
This only becomes a problem when other people start running packwerk update-deprecations
and suddenly multiple PRs have the same line removals in them.
I propose one of the following ideas and I'm looking for thoughts and feedback about which approach might be most palatable:
packwerk check
to also detect stale violationspackwerk check --include-stale
to optionally change its return code based on stale violationspackwerk full-check
or something similar that checks bothWhat do folks think? Is this something that others would find helpful?
To enforce a certain architecture, we need to place restrictions on the dependency graph. Two examples of such restrictions.
- Package X does not want to be depended on by Package Y
For example: the inventory team does not want the storefront package to depend on them.
Layered architecture (Persistence doesn't want to be depended on by Presentation)
- Acyclic dependency graph
This check is already included in packwerk, as we believe it's a critical component of architectural success.
We should investigate how we can express this in Packwerk, either via custom checkers, package configuration, or both.
packages.yml
to depends_on:
or does_not_depend_on:
(or something else?) to make it clearerDescription
Running packwerk check
should ideally show a warning/error when a package.yml file includes items in the dependencies:
block that are not actually dependencies in the code.
To Reproduce
Start with a codebase with multiple packages in it, and where packwerk check and packwerk validate are both passing as expected.
Modify one of the package's package.yml files (package_a) to add a dependency on a different package (package b). Ensure that you don't create any cyclic loops in this experiment. Don't make any ruby code changes - the intention is to just add an extraneous dependency to package_b
Run packwerk validate and packwerk check, and note that the output indicates that the configuration is valid.
Expected Behaviour
I'd expect a warning or error saying that there's an extraneous/unused dependency on package_b in package_a.
Screenshots
N/A
Version Information
Additional Context
As our developers change code, they sometimes inadvertently remove the code that previously triggered a dependency. Ideally, packwerk would let them know this, so that they can remove the obsolete line from the package.yml
Description
If explicitly private constants are specified without the ::
prefix, they will not be matched when checking for privacy violations.
Specifying them in this way just does not have any effect.
To Reproduce
In a package a
defining MyConstant
at the top level (equivalent to ::MyConstant
), set enforce_privacy: ["MyConstant"]
.
From another package b
, add a reference to MyConstant
or ::MyConstant
.
Execute packwerk check --packages=b
.
Expected Behaviour
Packwerk reports an error for the reference to the private constant.
Actual Behavior
Packwerk reports no error.
Version Information
Additional Context
It works just fine if enforce_privacy
is set to ["::MyConstant"]
instead.
I see two possible solutions:
ApplicationValidator
to reject explicitly private constants that are not explicitly top-level - or in other words, only accept values that start with ::
Description
Trying bundle install
on a fresh clone of the Packwerk repository on MacOS Monterey yields the following error:
$ bundle install
Unable to find a spec satisfying sorbet-static (= 0.5.6360) in the set. Perhaps the lockfile is corrupted? Found
sorbet-static (0.5.6360-x86_64-linux), sorbet-static (0.5.6360-x86_64-linux), sorbet-static
(0.5.6360-universal-darwin-20), sorbet-static (0.5.6360-universal-darwin-20), sorbet-static
(0.5.6360-universal-darwin-19), sorbet-static (0.5.6360-universal-darwin-19), sorbet-static
(0.5.6360-universal-darwin-18), sorbet-static (0.5.6360-universal-darwin-18), sorbet-static
(0.5.6360-universal-darwin-17), sorbet-static (0.5.6360-universal-darwin-17), sorbet-static
(0.5.6360-universal-darwin-16), sorbet-static (0.5.6360-universal-darwin-16), sorbet-static
(0.5.6360-universal-darwin-15), sorbet-static (0.5.6360-universal-darwin-15), sorbet-static
(0.5.6360-universal-darwin-14), sorbet-static (0.5.6360-universal-darwin-14) that did not match the current
platform.
It looks like sorbet is locked at v0.5.6360, which doesn't support MacOS Monterey.
To Reproduce
bundle install
.sorbet-static
.Expected Behaviour
This is just affecting anyone trying to contribute/fork Packwerk on MacOS Monterey, but I would still hope that Sorbet can be bumped to a version that supports newer versions of MacOS.
Screenshots
N/A
Version Information
Additional Context
Bumping sorbet
via bundle update sorbet
should do the trick, but it looks like some RBI files generated by the current version of Tapioca are still using the deprecated T.enum
type. Running bin/srb tc
with Sorbet upgraded to v0.5.9396 yields the following error:
$ bin/srb tc
sorbet/rbi/gems/[email protected]:565: T.enum has been renamed to T.deprecated_enum https://srb.help/5004
565 | sig { params(constant: T.untyped, type_variable_type: T.enum([:type_member, :type_template]), type_variable: T::Types::TypeVariable, fixed: T.untyped, lower: T.untyped, upper: T.untyped).void }
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Autocorrect: Use `-a` to autocorrect
sorbet/rbi/gems/[email protected]:565: Replace with T.deprecated_enum
565 | sig { params(constant: T.untyped, type_variable_type: T.enum([:type_member, :type_template]), type_variable: T::Types::TypeVariable, fixed: T.untyped, lower: T.untyped, upper: T.untyped).void }
^^^^^^
Errors: 1
I tried upgrading Tapioca as well using bundle update tapioca
, but I also had to manually edit sorbet/rbi/todo.rbi
to fix an issue with Tapioca and autoloading.
Happy to submit a PR if the maintainers are interested in supporting development on newer versions of MacOS.
Description
I this is a usability issue. I accidentally had a duplicate entry under the packwerk load_path configuration which resulted in a cryptic error.
To Reproduce
load_paths:
- file_a
- file_b
...
- file_a
📦 Packwerk is running validation...
Validation failed ❗
Load path cache in [Project]/packwerk.yml incorrect!
Paths missing from file:
Extraneous load paths in file:
Expected Behaviour
The two options I see are:
Version Information
Description
Packwerk now checks for both new violations and stale deprecations when running check
. However, deprecation lists are per package, not per file, so we can only determine which deprecations are valid when checking whole packages. When check
ing single source files, we won't find all the violations that are in the deprecation file for the package, and then packwerk (wrongly) outputs There were stale violations found, please run `packwerk update-deprecations`
To Reproduce
packwerk check some_file.rb
on just a single fileExpected Behaviour
Output
📦 Packwerk is inspecting 1 file
.
📦 Finished in 0.76 seconds
No offenses detected
Version Information
Additional Context
I can think of three possible solutions here:
some_file.rb
we would only compare the found violations to those entries in the deprecation file that are referring to the same source filecheck
, put into a different command or an option --check-stale
My favourite is the middle one, make the comparison more intelligent, because then it will always "just work". The same fix could then also be applied to update-deprecations
, which currently misbehaves in a similar way when run on a single file.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.