shopify / packwerk

Good things come in small packages.

Packwerk

"I know who you are and because of that I know what you do." This knowledge is a dependency that raises the cost of change.

-- Sandi Metz, Practical Object-Oriented Design in Ruby

Packwerk is a Ruby gem used to enforce boundaries and modularize Rails applications.

Packwerk can be used to:

• Combine groups of files into packages
• Define package-level constant visibility (i.e. have publicly accessible constants)
• Help existing codebases to become more modular without obstructing development

Prerequisites

Packwerk needs Zeitwerk enabled, which comes with Rails 6.

Packwerk supports MRI versions 2.7 and above.

Demo

Watch a 1-minute video demo on how Packwerk works.

Installation

1. Add this line to your application's Gemfile:

gem 'packwerk'

2. Install the gem

Execute:

$ bundle install

Or install it yourself as:

$ gem install packwerk

2. Run bundle binstub packwerk to generate the binstub
3. Run bin/packwerk init to generate the configuration files

Usage

Read USAGE.md for usage once Packwerk is installed on your project.

Pronunciation

"Packwerk" is pronounced [ˈpakvɛʁk].

Ecosystem

Various third parties have built tooling on top of packwerk. Here's a selection of some that might prove useful:

• https://github.com/bellroy/graphwerk draws a graph of your package dependencies
• https://github.com/rubyatscale/packwerk-vscode integrates packwerk into Visual Studio Code so you can see violations right in your editor
• https://github.com/vinted/packwerk-intellij integrates packwerk into RubyMine so you can see violations right in your editor
• https://github.com/rubyatscale/packs-rails sets up Rails autoloading, as well as rspec and FactoryBot integration, for packages arranged in a flat list. packs-rails is quite convenient, but for autoloading we recommend to use Rails::Engines instead.
• https://github.com/rubyatscale/danger-packwerk integrates packwerk with danger.systems to provide packwerk feedback as Github inline PR comments
• https://github.com/rubyatscale/packwerk-extensions contains extensions for packwerk, including a checker for packwerk that allows you to enforce public API boundaries. This was originally extracted from packwerk itself.
• https://github.com/alexevanczuk/packs is a Rust implementation of packwerk that has experimental support for non-Rails, non-Zeitwerk applications.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake test to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

Limitations

With Ruby being a very dynamic language, static analysis tools such as Packwerk are bound to have limitations. To reduce the impact of those limitations, Packwerk is designed to avoid false positives (reporting references as violations that are actually fine) at any cost, and we pay the cost by accepting a small number of false negatives (failing to report actual violations).

• Packwerk can only resolve references to constants that are defined in code loaded by the application's Zeitwerk autoloader. This is because we rely on Zeitwerk's conventions, and code that is loaded differently (like through an explicit require) often doesn't follow these conventions.
• Method calls and objects passed around the application are completely ignored. Packwerk only cares about static constant references. That said, if you want Packwerk to analyze parameters of a method, you can use Sorbet to define a type signature. Sorbet signatures are pure Ruby code and use constants to express types, and Packwerk understands that.
• Support for custom Zeitwerk configuration is limited. If custom ActiveSupport inflections are used, Packwerk will understand that and everything is fine. However, if Zeitwerk is directly configured with custom Zeitwerk inflections or to collapse directories, Packwerk will get confused and produce false positives.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/Shopify/packwerk.

Read and follow the guidelines in CONTRIBUTING.md.

License

The gem is available as open source under the terms of the MIT License.

packwerk's Issues

[Bug Report] Max RSS is Significantly Higher in Version 2.0

Description
Memory usage in packwerk v2.0 is significantly higher than it was in v1.4—so much higher that we will need to increase the resource class of our CI nodes in order for us to run packwerk v2.0. It looks like the max RSS for the whole process tree has increased from ~850 MB to about ~2.5GB for our use-case.

I ran gnu time -v on bin/packwerk check in our application working directory using both versions of packwerk and saw the following results:

Version 1.4

Version 2.0

Command being timed: "bin/packwerk check" User time (seconds): 79.86 System time (seconds): 12.22 Percent of CPU this job got: 719% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:12.79 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 83560 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 399579 Voluntary context switches: 10258 Involuntary context switches: 179007 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 36 Page size (bytes): 4096 Exit status: 0

Command being timed: "bin/packwerk check" User time (seconds): 81.06 System time (seconds): 11.77 Percent of CPU this job got: 381% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:24.34 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 334968 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 5862 Minor (reclaiming a frame) page faults: 1233804 Voluntary context switches: 35237 Involuntary context switches: 156020 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 16 Socket messages received: 16 Signals delivered: 40 Page size (bytes): 4096 Exit status: 0

To Reproduce
Run packwerk while watching memory usage, compare between v1.4 and v2.0

Expected Behaviour
I expected memory usage to not increase so dramatically between v1.4 and v2.0.

Screenshots

The GNU time output only captures the max RSS of the root process, so I grabbed a couple screenshots from activity monitor to demonstrate that the memory usage is much higher in each of the child processes as well.

Packwerk v1.4 Memory Usage

Packwerk v2.0 Memory Usage

Version Information

• Packwerk: v2.0
• Ruby v3.0.2

Additional Context

I haven't looked into the changes yet, but I assume this is because packwerk now loads our rails application in each of its processes? I still wouldn't expect each child process's max RSS to balloon up so much.

Do you all have any idea why this might be? And secondly, any idea if it would be possible and reasonable to bring this number back down?

Thanks!

Separate filtering from reference extraction

We currently extract + filter references all in one step, which can sometimes be a point of confusion.

In particular, we do a lot of filtering:

packwerk/lib/packwerk/reference_extractor.rb

Lines 50 to 71 in c6b06b8

	def reference_from_constant(constant_name, node:, ancestors:, file_path:)
	namespace_path = Node.enclosing_namespace_path(node, ancestors: ancestors)
	return if local_reference?(constant_name, Node.name_location(node), namespace_path)
	constant =
	@context_provider.context_for(
	constant_name,
	current_namespace_path: namespace_path
	)
	return if constant&.package.nil?
	relative_path =
	Pathname.new(file_path)
	.relative_path_from(@root_path).to_s
	source_package = @context_provider.package_from_path(relative_path)
	return if source_package == constant.package
	Reference.new(source_package, relative_path, constant)
	end

You can see we remove

• locally referenced constants,
• references to constants packwerk is unable to resolve, or unable to resolve to a package, and
• references to constants in the same package as the source package of the reference.

There could be a light refactor here so that we can extract all references, followed by another layer to do the filtering. It should lend itself to easier testing and a lighter set of test suites.

[Bug Report] `packwerk update` requires `inflections.yml` file to be loaded from Rails

Description
When packwerk update is run (without any loaded files) from the command line, Packwerk fails to interprets some constant association.

To Reproduce

1. Add a custom association to the inflection file:

uncountable:
- 'crazy_details'

2. Create a mock violation in a package

has_many :crazy_details

3. Run packwerk update to get Packwerk to update the shitlist with a violation. You will see that Packwerk does not recognize the association.

4. Run bin/packwerk update to get Packwerk to update. You will see that the violation is captured because the files are loaded through bin/packwerk script.

Expected Behaviour

See 4 above.

Screenshots
If applicable, add screenshots to help explain your problem.

Version Information

• Packwerk: [e.g. v0.1.7]
• Ruby [e.g. v1.7]

Additional Context

Potential solutions:

• Add a warning for packwerk update runs
• Update documentation to ensure that it is clear that files need to be loaded if there are custom associations used in the application.

Run validation checker as part of `packwerk check`

Since we're migrating to our own executable, instead of Rubocop, it would make a lot more sense for the "check" subcommand to simply do this. If it adds too much runtime cost to check, we can also do it as a separate subcommand (e.g., validate).

Make Packwerk work in in-repo gems and other dependencies

Description
Currently Packwerk can only check constant references that are being autoloaded. This excludes any dependencies and in-repo gems that could exist in a codebase. Especially for in-repo gems we have a strong need for boundary checking.

To Reproduce
Add a package.yml with enforced privacy or dependency violations and run packwerk update or packwerk check, neither of these will work for non-autoloaded packages.

Expected Behaviour
Dependencies and other packages that are not autoloaded should still have boundaries enforced.

Version Information

• Packwerk: v1.0.0
• Ruby: 2.7.1p83

Declare a specific constant as a Dependency

Description
I am relatively new to Packwerk and working on building out different modularized portions of our application. As I am slowly chipping away at the backlog of packages needed, there are a lot of dependencies from the original structure of our application.

For example, I am building out a package that references a specific Model in the original app/models location.

Package:

• File path: app/packages/home_operations
• Specific class in question: app/packages/home_operations/services/cleaning.rb

Constant:

• File path: 'app/models/job.rb`
• Constant: ::Job

Expected Behaviour
Instead of declaring the entire package or an entire directory, I'd like to be able to simply say ::Job is a dependency, creating a much more concise dependency declaration.

Version Information

• Packwerk: 2.1.1
• Ruby: 3.0
• Rails: 6.1.5

Additional Context
When running packwerk update-deprecations, this is the output added to my deprecated-references.yml:

".":
  "::Job":
    violations:
    - dependency
    files:
    - app/packages/home_operations/services/cleaning.rb

I don't see anything in the documentation\USAGE.md that would seem to point to the expected formatting of the dependencies entry in package.yml.

Any guidance would be greatly appreciated!

Create flag for buildkite parallelism for `packwerk check`

Make packwerk compatible with buildkite parallelism. We could add a --parallelism_count and --parallelism_index (or some other names) and have buildkite set up to pass the env vars into those flags. This means no need to create another binstub for CI, and also means we can work with any CI system.

ActiveAdmin's belongs_to_test trips up packwerk [DOCUMENTATION]

Description
This file will trip up packwerk: https://github.com/activeadmin/activeadmin/blob/master/spec/unit/belongs_to_spec.rb
I suspect this stems from the belongs_to association helper.

If you run into an issue like this, I recommend excluding the file via packwerk.yml.

To Reproduce
Add this file to your repo and packwerk will crash.

Expected Behaviour
I think it is fine for packwerk to not be able to parse a file as unusual as this one.

Refactor cli.rb to be smaller and testable

As @doodzik mentioned in https://github.com/Shopify/packwerk/pull/61/files#r517545831, we should look into refactoring cli.rb as it is growing to be a large class.

We should move all of the different commands we have in to compositional objects, so we can just do something like command.run. Breaking this class up would make it easier to have unit tests.

I would also be curious to see how RuboCop implements their Cli class.

Fail check if shitlisted violation is removed

Fail check if shitlisted violation is removed or not updated.

We have a fix for this bug on Shopify/shopify - https://github.com/Shopify/shopify/pull/250880. However, we need to introduce a permanent fix for other repositories.

Options:

• Have packwerk update --exit-status or maybe packwerk update --check-stale, which returns an exit status: 0 if there are no stale violations; and 1 if there is a stale violation. This can be easily used on CI to ensure there are no stale violations. The flag will run packwerk update and check to see if there is a difference in the shitlists. Though I am concerned that this check would also be flagging new violations as stale and encouraging users to just run packwerk update to update the shitlist.

• Another option is to have packwerk check understand stale references. This would require more refactoring and unifying the idea of Shitlists and UpdatingShitlists. This option is preferred as the former would cost us some memory.

[Bug Report] Better handling of duplicate entries under load_paths

Description
I this is a usability issue. I accidentally had a duplicate entry under the packwerk load_path configuration which resulted in a cryptic error.

To Reproduce

load_paths:
- file_a
- file_b
...
- file_a

📦 Packwerk is running validation...

Validation failed ❗
Load path cache in [Project]/packwerk.yml incorrect!
Paths missing from file:

Extraneous load paths in file:

Expected Behaviour

The two options I see are:

• Let the validation pass by ignoring duplicates and display a warning.
• Improve the error message.

Version Information

• Packwerk 1.0.0
• Ruby 2.6.5

[Bug Report] bundle exec packwerk init is throwing a Sorbet Type Error

Description
Running bundle exec packwerk init is failing while creating application_load_paths due to Sorbet error: Return value: Expected type T::Array[String], got T::Array[T.any(Pathname, String)]

To Reproduce
This may be repo dependent, but running bundle exec packwerk init failed on the Shopify Flow repo.

Expected Behaviour
No error.

Screenshots
If applicable, add screenshots to help explain your problem.

Version Information

• Packwerk: 1.0.0 & 1.0.1
• Ruby 2.6.5

Additional Context
I found a workaround to get it working and was able to initialize, validate, and execute packwerk checks.
I changed

packwerk/lib/packwerk/application_load_paths.rb

Line 31 in 499750f

(engine.config.autoload_paths + engine.config.eager_load_paths + engine.config.autoload_once_paths).uniq

from:

(engine.config.autoload_paths + engine.config.eager_load_paths + engine.config.autoload_once_paths).uniq

to:

(engine.config.autoload_paths + engine.config.eager_load_paths + engine.config.autoload_once_paths).map(&:to_s).uniq

I can submit a PR with this change if those with more packwerk context think this is the correct change to make.

[Bug Report] `bundle exec` and `bin/packwerk` produces different behaviour

Description
bundle exec packwerk and bin/packwerk produces different result. bin/packwerk loads files while bundle exec does not.

To Reproduce
Try bundle exec packwerk validate and bin/packwerk validate
packwerk validate requires files to be loaded, so it will fail on the bundle exec command.

Expected Behaviour
packwerk validate requires files to be loaded, so it will fail on the bundle exec command.

Screenshots
If applicable, add screenshots to help explain your problem.

Version Information

• Packwerk: v1.1.0

Additional Context
The solution is to move the file loading logic from bin/packwerk script into ApplicationLoadPath. ApplicationLoadPath is only used in packwerk init and packwerk validate commands, the two commands that require Rails to be booted.

For additional context, see: #77 (comment)

Dependency violation not detected when using public folders for privacy boundary

Description
When I enforce privacy for specific constants, a dependency violation to a public constant in that package is correctly detected. As soon as I reconfigure the package and set enforce_privacy: true and move the public constant to the package's public folder the dependency violation is not detected anymore.

I expect that the way how I declare public/private constants doesn't influence the dependency check/violations.

To Reproduce

1. Example 1 reports correctly a dependency violation (on branch working).

git clone  [email protected]:Enceradeira/rails_problem_packwerk.git
bundle install
git checkout working
packwerk check

app/models/accounting/accounts.rb:6:8
Dependency violation: ::Debtors::DebtorsService belongs to 'app/models/debtors', but 'app/models/accounting' does not specify a dependency on 'app/models/debtors'.
Are we missing an abstraction?
Is the code making the reference, and the referenced constant, in the right packages?

Inference details: this is a reference to ::Debtors::DebtorsService which seems to be defined in app/models/debtors/debtors_service.rb.
To receive help interpreting or resolving this error message, see: https://github.com/Shopify/packwerk/blob/main/TROUBLESHOOT.md#Troubleshooting-violations


1 offense detected

No stale violations detected

2. Example 2 reports wrongly no violation (on branch not_working)

git checkout not_working
packwerk check

No offenses detected
No stale violations detected

3. The difference between Example 1 and Example 2 is only the way how I declare the privacy boundary in app/models/debtors/package.yml

git diff working not_working | cat    

diff --git a/app/models/debtors/package.yml b/app/models/debtors/package.yml
index 619e57e..958af2c 100644
--- a/app/models/debtors/package.yml
+++ b/app/models/debtors/package.yml
@@ -1,5 +1,5 @@
-enforce_privacy: 
-  - "::Debtors::Enquiries"
+enforce_privacy: true
+
 enforce_dependencies: true
 dependencies:
   - app/models/accounting
\ No newline at end of file
diff --git a/app/models/debtors/debtors_service.rb b/app/models/debtors/public/debtors_service.rb
similarity index 100%
rename from app/models/debtors/debtors_service.rb
rename to app/models/debtors/public/debtors_service.rb

Expected Behaviour
Example 2 should produce the same result as example 1. The way how I define the privacy boundary should not influence which dependency violations are detected.

Version Information

• Packwerk: 2.1.1
• Ruby 2.7.6p203
• Zeitwerk: 2.5.4
• Rails: 7.0.2.3

Any way of warning when extraneous dependencies exist in a package.yml file?

Description

Running packwerk check should ideally show a warning/error when a package.yml file includes items in the dependencies: block that are not actually dependencies in the code.

To Reproduce

Start with a codebase with multiple packages in it, and where packwerk check and packwerk validate are both passing as expected.

Modify one of the package's package.yml files (package_a) to add a dependency on a different package (package b). Ensure that you don't create any cyclic loops in this experiment. Don't make any ruby code changes - the intention is to just add an extraneous dependency to package_b

Run packwerk validate and packwerk check, and note that the output indicates that the configuration is valid.

Expected Behaviour

I'd expect a warning or error saying that there's an extraneous/unused dependency on package_b in package_a.

Screenshots

N/A

Version Information

• Packwerk: v2.1.1
• Ruby 2.7.4p191

Additional Context

As our developers change code, they sometimes inadvertently remove the code that previously triggered a dependency. Ideally, packwerk would let them know this, so that they can remove the obsolete line from the package.yml

[Bug Report] Uninitialized constant (FileUtils)

Description

When I run packwerk init or bundle exec packwerk init I receive an uninitialized constant error:

        29: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/bin/bundle:23:in `<main>'
        28: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/bin/bundle:23:in `load'
        27: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.6/exe/bundle:37:in `<top (required)>'
        26: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/friendly_errors.rb:130:in `with_friendly_errors'
        25: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/bundler-2.2.6/exe/bundle:49:in `block in <top (required)>'
        24: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:24:in `start'
        23: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        22: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:30:in `dispatch'
        21: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
        20: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        19: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
        18: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:494:in `exec'
        17: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:28:in `run'
        16: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:63:in `kernel_load'
        15: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:63:in `load'
        14: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/bin/packwerk:23:in `<top (required)>'
        13: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/bin/packwerk:23:in `load'
        12: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/exe/packwerk:6:in `<top (required)>'
        11: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/_methods.rb:222:in `block in _on_method_added'
        10: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/call_validation.rb:126:in `validate_call'
         9: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/call_validation.rb:126:in `call'
         8: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/cli.rb:50:in `run'
         7: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/_methods.rb:222:in `block in _on_method_added'
         6: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/call_validation.rb:126:in `validate_call'
         5: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.6295/lib/types/private/methods/call_validation.rb:126:in `call'
         4: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/cli.rb:59:in `execute_command'
         3: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/cli.rb:96:in `init'
         2: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/generators/application_validation.rb:9:in `generate'
         1: from /Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/generators/application_validation.rb:21:in `generate'
/Users/jaydorsey/.asdf/installs/ruby/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.1.2/lib/packwerk/generators/application_validation.rb:31:in `generate_packwerk_validate_script': uninitialized constant Packwerk::Generators::ApplicationValidation::FileUtils (NameError)

The relevant line of code is here. I noticed there wasn't a require 'fileutils' anywhere in the codebase, so adding one to line 4 of exe/packwerk resolved this for me. I feel like there should be a require somewhere inside the gem, but I don't know why this doesn't break inside Docker w/ the same ruby version.

Things I tried as workarounds that reproduced/exhibited same behavior:

• I tried with a brand new rails app (same version), macos, was able to reproduce
• I tried with both gem install packwerk and adding it to top level of Gemfile and using bundle exec
• I made sure to spring stop in all instances (and DISABLE_SPRING=1)
• Generate a bin/packwerk binstub
• Upgrading to Rails 6.1.3; Ruby 2.7.1

Things look fine/not able to reproduce with:

• I tried this inside a ruby 2.6.6 docker container, same rails version, not able to reproduce the error
• I tried this inside an alpine linux container w/ asdf-installed ruby 2.6.6; not able to reproduce

To Reproduce

Ruby 2.6.6 (p146)/macos (intel)/packwerk 1.1.2/Rails 6.0.3.5:

rails new foo_bar && cd foo_bar && packwerk init

Expected Behaviour

Init should complete successfully

Screenshots

N/A

Version Information

• Packwerk: 1.1.2
• Ruby 2.6.6
• Rails 6.0.3.5
• Macos
• asdf version manager

Additional Context

I do have a workaround for this, so not urgent. Behavior is strange so appreciate any confirmations of this behavior. Would normally do a quick PR for this but the fact that I can't reproduce it except on my machine has me wanting to confirm the behavior first. I feel like I'm missing something...

Packwerk doesn't support custom root namespaces for root directories in the load path

In Zeitwerk, the load path is list of directories that it calls "root directories". They are "root" because any files that reside in them are expected to represent modules that are part of the root namespace. If an object is nested in some more specific namespace, it needs to be placed in a subdirectory of one of these roots. For example, in the following, lib is a "root directory" with the following structure and the ruby files have the noted expected constants:

.
├── lib
│   ├── my_namespace
│   │   └── my_file.rb            # MyNamespace::MyFile
│   └── my_other_file.rb          # MyOtherFile

The thing is, Zeitwerk also supports setting a "default namespace" to each root directory via push_dir which is called when the Zeitwerk loader is being configured. So in the above, it was assumed lib was added to the loader via loader.push_dir("lib") but it could also be added with loader.push_dir("lib", MyCompany) which means the constants inside would be expected to be declared as MyCompany::MyNamespace::MyFile and MyCompany::MyOtherFile.

At GitHub we rely on this feature to work around some extremely legacy directories in our load path that do not conform to Zeitwerk's expectations. Up until now, we have worked around them in Packwerk by skipping some and "hacking" at others. At this point, we'd like to just add support for custom namespaces to Packwerk so we can be confident in Packwerk's operation.

While I haven't dived into the Packwerk's source for this yet, I suspect this would mean moving Packwerk::ApplicationLoadPaths from config.autoload_paths, eager_load_paths, and autoload_once_paths to Rails.autoloaders to get our list of root directories as a hash of paths to modules. By default, these modules would all just be Object as they are in Zeitwerk, but it would allow us to override that default as needed for certain paths.

I realize this would also necessitate a change to ConstantResolver as well and have a WIP branch here. This branch uses strings instead of module instances for the namespaces in order to keep us as disentangled from a dependency standpoint as possible.

While I realize this is a change that is probably not needed for most codebases, I think that improving Packwerk's support for Zeitwerk's configuration is overall a positive.

So at this point, before I go ahead with a PR for Packwerk, what do folks think about adding such support?

Support parsing slim (or temple?) template files

It'd be nice to be able to include slim files, just like erb files are supported. Slim is actually built on top of Temple which is a generic templates parser/compiler. Maybe it would even be worth it to replace the existing erb parser with a temple parse as it includes erb built-in and you'd be getting support for many others all at once.

Automate the adding of new package file paths to the config file

We want to automate the process of adding new filepaths from packages in .rubocop.packages.yml config file. This process could be done when update command is ran.

Error when parsing `Foo::Bar = -> {}`

Description
When adding Packwerk to an existing project an error was thrown during packwerk check.

This seems to be due to a namespaced constant which is assigned a lambda expression, like Foo::Bar = -> {}.

A mitigation is to exclude the offending file in packwerk.yml so that the rest of the codebase can be analysed as usual. However it's not obvious which file was being processed when the error occurred.

To Reproduce
Add a namespaced constant which is assigned a lambda expression to any file which Packwerk parses, for example:

# foo.rb
Foo::Bar = -> {}

Then run bundle exec packwerk check as usual.

Expected Behaviour
Packwerk should parse and analyse the lambda expression like any other code block.

Screenshots

/Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:224:in `type_of': undefined method `type' for nil:NilClass (NoMethodError)
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:101:in `constant?'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:267:in `module_creation?'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:163:in `module_name_from_definition'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/const_node_inspector.rb:43:in `constant_in_module_or_class_definition?'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:90:in `block in create_validator_slow'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/const_node_inspector.rb:23:in `constant_name_from_node'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:90:in `block in create_validator_slow'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/reference_extractor.rb:33:in `block in reference_from_node'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/reference_extractor.rb:32:in `each'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/reference_extractor.rb:32:in `reference_from_node'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_processor.rb:28:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation_2_6.rb:764:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation_2_6.rb:764:in `block in create_validator_method_medium2'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:12:in `visit'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:17:in `block in visit'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:63:in `block in each_child'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:62:in `each'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:62:in `each_child'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:16:in `visit'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:17:in `block in visit'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:63:in `block in each_child'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:62:in `each'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node.rb:62:in `each_child'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/node_visitor.rb:16:in `visit'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/file_processor.rb:49:in `references_from_ast'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/file_processor.rb:37:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation_2_6.rb:703:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation_2_6.rb:703:in `block in create_validator_method_medium1'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/run_context.rb:56:in `process_file'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:90:in `block in create_validator_slow'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/parse_run.rb:66:in `block in find_offenses'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:515:in `call_with_index'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:485:in `process_incoming_jobs'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:465:in `block in worker'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:456:in `fork'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:456:in `worker'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:447:in `block in create_workers'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:446:in `each'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:446:in `each_with_index'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:446:in `create_workers'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:386:in `work_in_processes'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:289:in `map'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/parallel-1.21.0/lib/parallel.rb:302:in `flat_map'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/parse_run.rb:74:in `block in find_offenses'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/parse_run.rb:72:in `find_offenses'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/parse_run.rb:45:in `check'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/cli.rb:53:in `execute_command'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/lib/packwerk/cli.rb:40:in `run'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/call_validation.rb:161:in `validate_call'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/sorbet-runtime-0.5.9464/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/packwerk-2.0.0/exe/packwerk:12:in `<top (required)>'
	from /Users/me/.asdf/installs/ruby/2.6.9/bin/packwerk:23:in `load'
	from /Users/me/.asdf/installs/ruby/2.6.9/bin/packwerk:23:in `<top (required)>'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli/exec.rb:58:in `load'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli/exec.rb:23:in `run'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli.rb:478:in `exec'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli.rb:31:in `dispatch'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/cli.rb:25:in `start'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/exe/bundle:49:in `block in <top (required)>'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
	from /Users/me/.asdf/installs/ruby/2.6.9/lib/ruby/gems/2.6.0/gems/bundler-2.2.32/exe/bundle:37:in `<top (required)>'
	from /Users/me/.asdf/installs/ruby/2.6.9/bin/bundle:23:in `load'
	from /Users/me/.asdf/installs/ruby/2.6.9/bin/bundle:23:in `<main>'

Version Information

• Packwerk: v2.0.0
• Ruby: v2.6.9
• Bundler: v2.2.32
• Asdf: v0.8.1

Additional Context
A constant which isn't namespaced, like Foo = -> {} seems to be fine. Using Foo:Bar = Proc.new {} seems to be fine.

It would be helpful if a parsing error included the name of the file being parsed.

Update command should remove `deprecated_references.yml` file when all violations are removed

When all the violations are resolved from a package, packwerk update does not delete the file, leaving the file stale. packwerk update should somehow be able to remove stale files completely.

Packwerk can't parse `self::`

Description
Packwerk can't parse a this file:

class Foobar
  self::SOMETHING = 'foo'
end

To Reproduce
Create a ruby file with the contents above and have Packwerk check it.

Expected Behaviour
Not error

Version Information

• Packwerk: [v1.0.0]
• Ruby [v1.7]

Privacy violations in migrations

Problem

Privacy violations in migrations seem to be caught by Packwerk.

We suspect the t.belongs_to(:order) is what is setting off the violation, as Packwerk thinks it is an active record association.

To get rid of these type of violations, users have been putting this violation in to the deprecated_references.yml list.

Potential solutions

There may be two solutions:

1. change the associations inspector to ignore methods that are not called on an implicit self
2. ignore violations in migrations files

[Bug Report] `packwerk init` failing due to TypeError

Description
When running bundle exec packwerk init on a fresh install, a Sorbet TypeError was thrown when generating the configuration file. This is due to a missing load_paths value, for which Sorbet expects to receive an Array.

To Reproduce

1. Run bundle exec packwerk init in a project that doesn't yet have a packwerk config file

Expected Behaviour
All init generators should run without failure.

Version Information

• Packwerk: v1.0.2
• Ruby: v2.7.1

Additional Context
This problem seems to have been introduced in #68, specifically at https://github.com/Shopify/packwerk/pull/68/files#diff-e43184d1a960a3c3821da21b3eb72c83379c2dcc996f6bcfac9aadd6898cd374L45.

Error details:

Parameter 'load_paths': Expected type T::Array[String], got type NilClass (TypeError)
Caller: /Users/mikelkew/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6084/lib/types/private/methods/call_validation.rb:78
Definition: /Users/mikelkew/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/packwerk-1.0.2/lib/packwerk/generators/configuration_file.rb:21

Support for treating package root module as public

[Enhancement]

We are just beginning to use Packwerk and loving its power! However, it doesn't allow us to follow a common pattern in our existing packaged code. I would like this feature and wanted to see if you'd accept the PR or if we should think about forking.

We commonly follow this directory structure

app/
  domains/
    identity_domain/
      user.rb
      package.yml
    identity_domain.rb
package.yml

In this structure, the root module sharing its name with the package serves as the entry point to the domain:

# app/domains/identity_domain.rb
module IdentityDomain
  def self.get_user(id)
    IdentityDomain::User.find(id)
  end
end

This seems to conflict with Packwerk, because AFAICS there is no way to tell it to treat the root module as public. The best we can come up with is moving the methods to e.g. IdentityDomain::Public::API but this feels quite verbose. I envision solving this with a per package opt-in like:

# app/domains/identity_domain/package.yml
enforce_privacy: true
public_root: true

This seems to overlap with the conversation in #98 but with a much narrower focus. I'm not talking about allowing arbitrary public entrypoints, but rather the option to treat the root module as a public constant.

As I say, would you consider a PR for this? Do you consider this an anti-pattern perhaps?

[Bug Report] Unable to execute packwek init

Description
Attempting to initialize/setup packwerk, but hitting a relative_path_from issue.

To Reproduce

• Initialize a fresh packwerk installation

Expected Behaviour

• It should work.

Version Information

• ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-darwin19]

Additional Context

➜  web git:(master) ✗ be packwerk init
📦 Initializing Packwerk...
📦 Generating application validator...
✅ Packwerk application validation bin script generated in /Users/jasonlor/Documents/GitHub/app/web/bin
Version: 2.1.1

Usage: spring COMMAND [ARGS]

Commands for Spring itself:

  binstub         Generate Spring based binstubs. Use --all to generate a binstub for all known commands. Use --remove to revert.
  help            Print available commands.
  server          Explicitly start a Spring server in the foreground
  status          Show current status.
  stop            Stop all Spring processes for this project.

Commands for your application:

  rails           Run a rails command. The following sub commands will use Spring: console, runner, generate, destroy, test.
  rake            Runs the rake command
Traceback (most recent call last):
	9: from bin/packwerk:22:in `<main>'
	8: from bin/packwerk:22:in `new'
	7: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/cli.rb:23:in `initialize'
	6: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:18:in `from_path'
	5: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:18:in `new'
	4: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:45:in `initialize'
	3: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:61:in `all_application_autoload_paths'
	2: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:61:in `map'
	1: from /Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/packwerk-1.0.0/lib/packwerk/configuration.rb:65:in `block in all_application_autoload_paths'
/Users/jasonlor/.rbenv/versions/2.6.6/lib/ruby/2.6.0/pathname.rb:522:in `relative_path_from': different prefix: "" and "/Users/jasonlor/Documents/GitHub/app/web" (ArgumentError)

Custom Zeitwerk Inflections

Description
Is there interest in having support for custom zeitwerk inflections added? Is it just a matter of "writing the code" or is there known blockers for it? Considering submitting a PR but wanted to check on it first...

Refactor how node processing work

We would like to refactor how updating the shitlist work.

https://github.com/Shopify/packwerk-old/pull/181#pullrequestreview-417116089

Base of @thegedge's suggestion we have at least two options to explore:

1. Change the behaviour of RunContext, and we returned a different node processor for updating versus checking.

2. File and node processor work together to yield references, and some other class (e.g., Cli) would then process the references differently based on whether we're checking or updating

Flipping the meaning of `enforce_privacy`?

[EXPLORE]

Hi all!

I have been spinning the wheels on packwerk and love it so far!

Regarding privacy - when listing privacy constraints by constant, I would like to list the public files in the config so any new constants are private by default. Right now I can only use enforce_privacy which will make new constants public by default. With the new name I propose, enforce_api_boundary, the behavior of true could stay the same.

Is this something you would accept a PR for?

ConstNodeInspector doesn’t try to resolve module namespaces

This is a copy-paste of @tomstuart's bug report, originally posted to https://github.com/Shopify/packwerk-old/issues/289

Describe the bug

When Packwerk::ConstNodeInspector finds a constant in a class or module definition, it uses the lexical nesting to fully-qualify that constant’s name. This is not always correct because a namespace may refer to a constant that has already been defined elsewhere, e.g. in another package.

This bug is a variation on #234: this time it’s a reference to an existing constant elsewhere in the enclosing namespace, rather than to the root namespace, which causes the problem.

To Reproduce

Say we have a pair of components, one and two, which define the modules A::B and A::C respectively:

% tree components
components
├── one
│   ├── a
│   │   └── b.rb
│   └── package.yml
└── two
    ├── a
    │   └── c.rb
    └── package.yml

Both package.yml files have enforce_dependencies and enforce_privacy set to true and no dependencies defined, so Packwerk should not allow either component to refer to the other.

In addition to defining A::B, the file one/a/b.rb also defines a nested class A::B::D:

module A
  module B
    class D
      def info
        'defined inside ::A::B by component one'
      end
    end
  end
end

And in addition to defining A::C, the file two/a/c.rb reopens that class and overwrites one of its methods:

module A
  module C
    class B::D
      def info
        'defined inside ::A::C by component two'
      end
    end
  end
end

It might not be immediately obvious that two is redefining A::B::D#info here, but it is, because the reference to B in class B::D gets resolved to the existing constant A::B, regardless of the definition occurring lexically inside module C. (This is what #234 is about.)

So when these two components are loaded, two refers to (and modifies) the class A::B::D from one:

% irb -Icomponents/one -Icomponents/two
>> require 'a/b'
=> true
>> require 'a/c'
=> true
>> A::B::D.new.info
=> "defined inside ::A::C by component two"
=> nil

packwerk check doesn’t detect this violation because it doesn’t realise that class B::D is a reference to A::B::D. It assumes it must refer to A::C::B::D because it appears inside module A; module C; …; end end, but Ruby constant resolution is more complex.

This doesn’t present an immediate problem within Shopify because we don’t use compact constant nesting (class B::D) in class definitions, but Packwerk may give incorrect results on external codebases.

Expected behavior

packwerk check should give an error like this:

Dependency violation: ::A::B::D belongs to 'components/one', but 'components/two' does not specify a dependency on 'components/one'.
Are we missing an abstraction?
Is the code making the reference, and the referenced constant, in the right packages?

Inference details: 'B::D' refers to ::A::B::D which seems to be defined in components/one/a/b.rb.

Version information

• Packwerk: v0.1.10
• Ruby: v2.6.6p146

Extract constant names from fixtures

What?

In test files, when we see:

class MyTest < ActiveSupport::TestCase
  setup do
    @thing = things(:thing_fixture)
  end
end

We want to extract the constant name ::Thing from the call things(:thing_fixture).

Why?

Tests should also be appropriately packaged (in other words, no boundary violations), and fixtures are an easy way for packages to (currently) depend on other packages without packwerk realizing. By registering these constant references we have a better signal to help packwerkers have proper isolation for their tests.

How?

I'm thinking this will be a few PRs:

1. Introduce an interface for getting a constant name from a node.
2. Refactor ReferenceExtractor to accept a list of instances of anything that implements the "constant name from node" interface in (1).
3. Extract existing "constant name from node" from ReferenceExtractor (the node.const_node? path).
4. Add new class to get constant name from a node if the node looks like a fixtures call.

For (4), to reduce false positives, I'm thinking the criteria for a "fixtures function call" would be the following:

• Directory of the file being parsed matches **/test/**/*_test.rb or **/spec/**/*_spec.rb
• AST node is of type :send with one or more arguments that are all symbols, and
• Name of function maps to a file in (a configurable) fixtures path.
  • We can preload all fixtures files to get the model class name too (via _fixture.model_class, if it exists, otherwise just use the inflector on the filename).

cc: @thegedge

[Bug Report] Explicitly private constants without `::` are ignored

Description

If explicitly private constants are specified without the :: prefix, they will not be matched when checking for privacy violations.
Specifying them in this way just does not have any effect.

To Reproduce

In a package a defining MyConstant at the top level (equivalent to ::MyConstant), set enforce_privacy: ["MyConstant"].
From another package b, add a reference to MyConstant or ::MyConstant.
Execute packwerk check --packages=b.

Expected Behaviour
Packwerk reports an error for the reference to the private constant.

Actual Behavior
Packwerk reports no error.

Version Information

• Packwerk: 1.3.2
• Ruby 3.0.2p107

Additional Context
It works just fine if enforce_privacy is set to ["::MyConstant"] instead.

I see two possible solutions:

• Enhance the ApplicationValidator to reject explicitly private constants that are not explicitly top-level - or in other words, only accept values that start with ::
• Better: Change the comparison code to that it assumes all explicitly private constants are top level.

[Bug Report] `Errno::ENOENT: No such file or directory @ rb_sysopen - tmp/cache/packwerk/inflections` (issue with cache)

Description
The cache periodically (rarely) fails in our CI system with the error shown in the screenshots section.

We suspect that the packwerk cache has a race condition when forked processes are initializing the cache independently.
Namely -- after one process has bust the cache and created the cache directory, another process can then bust the cache, after which the original process may try to write the cache contents of inflections.rb. This can result in the bug:
Errno::ENOENT: No such file or directory @ rb_sysopen - tmp/cache/packwerk/inflections

To Reproduce
We are not able to reliably reproduce this bug.

Expected Behaviour
We expect the cache to not raise an error. From an implementation perspective, we expect tmp/cache/packwerk to already exist before attempting to write the cache file.

Screenshots

Version Information

• Packwerk: 2.1.0
• Ruby: 2.7.5

Additional Context
Some potential solutions we're looking at to solve this issue:

• We can initialize Cache within the parent process to ensure that the call to FileUtils.rm_fr is only run once at the start. This is what we went with in this PR: #183
• Alternatively, we can also drop the rm_fr entirely, and use a strategy that hashes based on the digest of the current inflections.

[Bug Report] Current version of sorbet-static doesn't support MacOS Monterey

Description

Trying bundle install on a fresh clone of the Packwerk repository on MacOS Monterey yields the following error:

$ bundle install
Unable to find a spec satisfying sorbet-static (= 0.5.6360) in the set. Perhaps the lockfile is corrupted? Found
sorbet-static (0.5.6360-x86_64-linux), sorbet-static (0.5.6360-x86_64-linux), sorbet-static
(0.5.6360-universal-darwin-20), sorbet-static (0.5.6360-universal-darwin-20), sorbet-static
(0.5.6360-universal-darwin-19), sorbet-static (0.5.6360-universal-darwin-19), sorbet-static
(0.5.6360-universal-darwin-18), sorbet-static (0.5.6360-universal-darwin-18), sorbet-static
(0.5.6360-universal-darwin-17), sorbet-static (0.5.6360-universal-darwin-17), sorbet-static
(0.5.6360-universal-darwin-16), sorbet-static (0.5.6360-universal-darwin-16), sorbet-static
(0.5.6360-universal-darwin-15), sorbet-static (0.5.6360-universal-darwin-15), sorbet-static
(0.5.6360-universal-darwin-14), sorbet-static (0.5.6360-universal-darwin-14) that did not match the current
platform.

It looks like sorbet is locked at v0.5.6360, which doesn't support MacOS Monterey.

To Reproduce

1. Clone repository from a computer running MacOS Monterey.
2. Run bundle install.
3. Confirm that Bundler can't satisfy sorbet-static.

Expected Behaviour

This is just affecting anyone trying to contribute/fork Packwerk on MacOS Monterey, but I would still hope that Sorbet can be bumped to a version that supports newer versions of MacOS.

Screenshots

N/A

Version Information

• Packwerk: 1.4 (main branch)
• Ruby 3.0.0
• Platform: x86_64-darwin-21

Additional Context

Bumping sorbet via bundle update sorbet should do the trick, but it looks like some RBI files generated by the current version of Tapioca are still using the deprecated T.enum type. Running bin/srb tc with Sorbet upgraded to v0.5.9396 yields the following error:

$ bin/srb tc
sorbet/rbi/gems/[email protected]:565: T.enum has been renamed to T.deprecated_enum https://srb.help/5004
     565 |    sig { params(constant: T.untyped, type_variable_type: T.enum([:type_member, :type_template]), type_variable: T::Types::TypeVariable, fixed: T.untyped, lower: T.untyped, upper: T.untyped).void }
                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Autocorrect: Use `-a` to autocorrect
    sorbet/rbi/gems/[email protected]:565: Replace with T.deprecated_enum
     565 |    sig { params(constant: T.untyped, type_variable_type: T.enum([:type_member, :type_template]), type_variable: T::Types::TypeVariable, fixed: T.untyped, lower: T.untyped, upper: T.untyped).void }
                                                                    ^^^^^^
Errors: 1

I tried upgrading Tapioca as well using bundle update tapioca, but I also had to manually edit sorbet/rbi/todo.rbi to fix an issue with Tapioca and autoloading.

Happy to submit a PR if the maintainers are interested in supporting development on newer versions of MacOS.

Group offenses together

We currently can dump a lot of information at the user:

/Some/path/foo.rb:136:8
Privacy violation: '::Foo::Bar' is private to 'foo' but referenced from 'path'.
Is there a public entrypoint in 'foo/app/public/' that you can use instead?
Inference details: 'Foo::Bar' refers to ::Foo::Bar which seems to be defined in foo/app/foo/models/bar.rb.

/Some/path/foo.rb:151:8
Privacy violation: '::Foo::Spam' is private to 'foo' but referenced from 'path'.
Is there a public entrypoint in 'foo/app/public/' that you can use instead?
Inference details: 'Foo::Spam' refers to ::Foo::Spam which seems to be defined in foo/app/models/foo/spam.rb.

/Some/path/foo.rb:161:8
Privacy violation: '::Foo::Eggs' is private to 'foo' but referenced from 'path'.
Is there a public entrypoint in 'foo/app/public/' that you can use instead?
Inference details: 'Foo::Eggs' refers to ::Foo::Eggs which seems to be defined in foo/app/models/foo/eggs.rb.

There's a huge amount of repetition in there, and that's mostly because we were trying to fit the rubocop mold. Let's investigate grouping these together in a more concise and informative way.

Here's one idea:

# Privacy violations

These references appear to reference a private constant in relevant packages. You should look for a publicly accessible constant (<link> to learn more).

- /Some/path/foo.rb:136:8 references `Foo::Bar` in package 'foo'
- /Some/path/foo.rb:151:8 references `Foo::Spam` in package 'foo'
- /Some/path/foo.rb:161:8 references 'Foo::Eggs' in package 'foo'

# Dependency violations

...

Add unit tests for ApplicationValidator checks

We have ApplicationValidatorTest which are integration tests. However, we should find a way to unit test the individual checks.

We could make unit testing this file easier by refactoring it so a fake filesystem can be injected.
This would make it easier to test but would require a larger refactor. https://github.com/Shopify/packwerk-old/pull/262#discussion_r455956863

TO DO:

• check_autoload_path_cache,
• check_package_manifests_for_privacy,
• check_package_manifest_syntax,
• check_application_structure,
• check_inflection_file,
• check_acyclic_graph,
• check_package_manifest_paths,
• check_valid_package_dependencies,
• check_root_package_exist,

How do you deal with specs?

Description
In a componentised monolith, where should specs live to avoid causing privacy errors?

To Reproduce
Create a new Rails project, create a component with a private file like components/finance/app/models/account.rb.
Write a spec for that model, where would you put it?

Expected Behaviour
Documentation should explain how to deal with tests as this is a very common issue.

Version Information

• Packwerk: v2.0.0
• Ruby: v2.7.5

Additional Context
Two approaches comes immediately to mind, but none of them is straightforward:

1. Create a components subdirectory in your spec folder and put the model test there (/spec/components/finance/models/account_spec.rb). This won't work because the model above is private, so directly referencing it from spec/components/... causes a privacy violation that needs to be ignored.
2. Put specs in a spec subdirectory of your component (components/finance/spec/models/account_spec.rb). This makes packwerk check pass, but RSpec won't look for tests in that folder by default. I managed to tweak RSpec to do so locally (with --default-path and --pattern options), but this solution is hacky and our CI, that relies on Knapsack to run parallel tests, doesn't work as expected.

How do you do this in your packwerk-powered projects?

[Bug Report] Packwerk suggests to run `update-deprecations` when it doesn't apply

Description
Packwerk now checks for both new violations and stale deprecations when running check. However, deprecation lists are per package, not per file, so we can only determine which deprecations are valid when checking whole packages. When checking single source files, we won't find all the violations that are in the deprecation file for the package, and then packwerk (wrongly) outputs There were stale violations found, please run `packwerk update-deprecations`

To Reproduce

1. Choose a package that has recorded deprecations from multiple files
2. run packwerk check some_file.rb on just a single file
3. witness the confusing message

Expected Behaviour
Output

📦 Packwerk is inspecting 1 file
.
📦 Finished in 0.76 seconds

No offenses detected

Version Information

• Packwerk: 1.3.2
• Ruby 3.0.1p64

Additional Context
I can think of three possible solutions here:

• omit stale violations check if we're checking partial packages
• compare stale violations per file instead of per package; when just checking some_file.rb we would only compare the found violations to those entries in the deprecation file that are referring to the same source file
• remove stale violation check from check, put into a different command or an option --check-stale

My favourite is the middle one, make the comparison more intelligent, because then it will always "just work". The same fix could then also be applied to update-deprecations, which currently misbehaves in a similar way when run on a single file.

ParsedConstantDefinitions doesn’t support fully qualified constants

RuboCop::AST::Node#const_name doesn’t reveal whether a constant is fully qualified (e.g. ::HELLO) or not:

>> RuboCop::ProcessedSource.new('::HELLO', 2.6).ast.const_name
=> "HELLO"

>> RuboCop::ProcessedSource.new('HELLO', 2.6).ast.const_name
=> "HELLO"

As a result, the implementation of ParsedConstantDefinitions#collect_local_definitions is unaware of whether a constant is fully qualified, so it’s unable to put it in the correct namespace:

>> definitions = ParsedConstantDefinitions.new(
     root_node: parse_code('module Sales; ::HELLO = "World"; end')
   )
=> #<Packwerk::ParsedConstantDefinitions
     @local_definitions=
      {"::Sales"=>#<Parser::Source::Range (string) 7...12>,
       "::Sales::HELLO"=>#<Parser::Source::Range (string) 16...21>}>

>> definitions.local_reference?('HELLO')
=> false # should be true

>> definitions.local_reference?('Sales::HELLO')
=> true # should be false

I’m not going to fix this right now because I’m trying to make progress on a different task, but I think it’s a bug that’s going to affect users.

cc: @tomstuart

Assume Ruby Files for parser

To simplify our lives, let's assume that files are Ruby if they aren't recognized by any other parser (for now, that's just .erb). This will eliminate encountering "unknown file type" errors.

If assuming Ruby fails, we'll have to encourage people to use the include / exclude globs in their configs to handle other types of files.

We could assume a file without an extension is an executable, check to see if it looks like a Ruby executable, and ignore it if it's not. @thegedge prefers being explicit, but can understand that a larger repo may have built up a lot of executable files, many of which may not be Ruby.

Introduce command that combines `check` and `detect-stale-violations`

Currently in our monolith we are running packwerk check in our CI. This is great and helps us manage our technical in a controlled manner. However, if a team removes some deprecated references and forgets to run packwerk update-deprecations there is no CI failure and the associated deprecated_references.yml file just gets a bit out of date with reality since we don't run packwerk detect-stale-violations in CI.

This only becomes a problem when other people start running packwerk update-deprecations and suddenly multiple PRs have the same line removals in them.

I propose one of the following ideas and I'm looking for thoughts and feedback about which approach might be most palatable:

• Update packwerk check to also detect stale violations
• Add packwerk check --include-stale to optionally change its return code based on stale violations
• Add a new command packwerk full-check or something similar that checks both

What do folks think? Is this something that others would find helpful?

Drop into a debugger when encountering a `debugger` call

When you want to debug Packwerk for a specific context (file + constant), it's quite difficult to do without running a debugger for all contexts. What if we dropped into a debugger whenever we see an AST node that is a call, for a method named debugger, and no arguments?

These are generally not found in code that gets committed, so it should be non-intrusive, but best to scope it behind a flag/subcommand to be safe.

Allow Packages To Have Custom Public Paths

Description

Currently the public folder is hard-coded as app/public, which may not work for all users of the gem. They may wish to define their own public folder, and if none is defined, default to app/public.

Why?

Depending on the architecture of the application, app/public might just not make sense, or at least not be cohesive with the rest of the folder structure. Consider a layered architecture for example, where you break up the layers of your application via technical partition (i.e. presentation layer, domain layer, infrastructure layer etc.). Your top-level folders in your package would typically be /presentation, /domain etc., so logically in that architecture you would want to select the layer that makes sense to expose the package from, and app/public conflicts with that vision.

I've raised a PR here that I believe introduces that feature, and seems to work nicely.

Change the name `packwerk update` to `packwerk update-deprecations`

@exterm brought up that packwerk update should actually be called packwerk update-deprecations and I totally agree. packwerk update can be confusing.

We should add a packwerk update-deprecations command and add a deprecation warning for the packwerk update command but still allow people to use it.

Exclude does not apply to `load_paths`

My CI vendor folder contents is slightly different from my local environment. This produces different load_paths value in packwerk.yml causing this failure "Load path cache in #{@config_file_path} incorrect!"

Is there any way to get around this cache issue? I see the exclude option does not affect the load paths at all. I'm not sure that it makes sense for it to, although I'm never going to define anything in vendor as a package.

I also have issues with check because it causes the ERROR: 'X' could refer to any of error since some model classes are defined in a gem and also the Rails application. I do not have these issues locally since vendor is not in the load_paths. ETA: Actually, these are valid violations since the classes are defined twice.

I should note that I'm using Rails 5.2.4

Support customizing the ERB implementation

Description
Some gems (e.g., http://github.com/github/graphql-client) provide ERB extensions which add or modify the standard ERB syntax. To support apps which use those gems, Packwerk should allow for customizing the ERB implementation.

To Reproduce

1. Try to run Packwerk on an app which uses http://github.com/github/graphql-client's GraphQL tags.
2. Packwerk fails to parse the ERBs since it doesn't know how to parse the GraphQL tags

Expected Behaviour

Packwerk should provide a way to customize the ERB implementation.

Screenshots
N/A

Version Information

• Packwerk: v1.0.1
• Ruby: 2.6.6

Nested packages private by default

Currently, all nested packages are accessible by any other package. We may want to reconsider this model so that nested packages are not "exported" by default, unless explicitly requested by the parent package:

# tree
.
└── foo
    ├── package.yml
    └── bar
        └── package.yml

# cat foo/package.yml
enforce_privacy: true
enforce_dependencies: true
dependencies:
  - "foo/bar"
export_packages:
  - "foo/bar"

This would allow those who depend on foo to be able to also access bar, otherwise any access to or dependency on bar would be a violation. If we want to export a subset of bar to the rest of the world, we would use aliasing:

# cat foo/app/public/foo/stuff.rb
module Foo
  Stuff = ::Foo::Bar::Stuff
end

We've briefly described a project where packages could put more constraints on incoming edges (in the dependency graph).

@thegedge sees two possibilities for those constraints: a whitelist option or a blacklist option (not both). In this case, we would whitelist the parent package in the nested package, which means any other package would be a violation.

[Bug] Packwerk.yml `exclude` option does not work

Description
packwerk does not exclude configured folders for exclusion.

To Reproduce

Demo rails app: https://github.com/evaldasg/packwerk-issue

Install npm package:

yarn add yaml-js

packwerk configured to exclude node_modules

exclude:
- "{bin,node_modules,script,tmp,vendor}/**/*"

packwerk validation fails:

➜  packwerk-issue git:(main) bin/packwerk validate
📦 Packwerk is running validation...

Validation failed ❗
Unknown keys in /packwerk-issue/node_modules/yaml-js/src/package.yml: ["name", "version", "description", "main", "repository", "devDependencies", "license"]
If you think a key should be included in your package.yml, please open an issue in https://github.com/Shopify/packwerk

Expected Behaviour
If packwerk.yml configured with exclude option, it is expected that packwerk will skip those paths.

Version Information

• Packwerk: [1.1.2]
• Ruby [2.7.2]
• Rails [6.0.3.5]

Additional Context
Suspected code: https://github.com/Shopify/packwerk/blob/main/lib/packwerk/package_set.rb#L37

def package_paths(root_path, package_pathspec)
  bundle_path_match = Bundler.bundle_path.join("**").to_s

  glob_patterns = Array(package_pathspec).map do |pathspec|
    File.join(root_path, pathspec, PACKAGE_CONFIG_FILENAME)
  end

  Dir.glob(glob_patterns)
    .map { |path| Pathname.new(path).cleanpath }
    .reject { |path| path.realpath.fnmatch(bundle_path_match) }
end

This rejects only gems directory but does not take @configuration.exclude into account.

Restrictions on the dependency graph

To enforce a certain architecture, we need to place restrictions on the dependency graph. Two examples of such restrictions.

1. Package X does not want to be depended on by Package Y
   For example: the inventory team does not want the storefront package to depend on them.
   Layered architecture (Persistence doesn't want to be depended on by Presentation)

2. Acyclic dependency graph
   This check is already included in packwerk, as we believe it's a critical component of architectural success.

We should investigate how we can express this in Packwerk, either via custom checkers, package configuration, or both.

To do

• Change keys in packages.yml to depends_on: or does_not_depend_on: (or something else?) to make it clearer
• Investigate ArchRuby, a gem that runs a static check on an application and determines if it is following a pre-determined architecture.

Improve message output for parsing errors

While trying to fix a parsing error shown on Packwerk, we had to run rubocop locally since Packwerk did not display any specifics of the error (line number, for example).