GithubHelp home page GithubHelp logo

gregnavis / active_record_doctor Goto Github PK

View Code? Open in Web Editor NEW
1.7K 16.0 56.0 379 KB

Identify database issues before they hit production.

License: MIT License

Ruby 99.12% Logos 0.88%
database rails activerecord performance ruby

active_record_doctor's People

Contributors

amckinnell avatar bedgerotto avatar chubchenko avatar dmitryzuev avatar duleorlovic avatar eitoball avatar erick-ol avatar fatkodima avatar fryguy avatar gregnavis avatar jdufresne avatar jstuessy avatar julianlires avatar krainboltgreene avatar maschwenk avatar mehlah avatar olleolleolle avatar onerinas avatar petergoldstein avatar pingortle avatar rhymes avatar shime avatar syndbg avatar tatsuyafw avatar tommotorefi avatar uplus avatar vinnyglennon avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

mediasota shishirmk fryguy vinothinibalakrishnan syndbg dave4u90 postingres tcopeland jayqui sespindola duleorlovic ssabrii ladas johnsmith1212 nguyenanh cetuslabs mtejedorwolox shime bilal-tx ogonki-vetochki eitoball olleolleolle maratgaliev uplus hacker0x01 dmitryzuev maschwenk shimada-toy-box cyberxander90 kopylovvlad tatsuyafw vanharen fatkodima mehlah amckinnell ckolkey jgrau pingortle bearerpipelinetest neatnerdprime mamata2507 ydah julianlires apaciuk bedgerotto rumel joshfng vinnyglennon krainboltgreene jstuessy cxz jdufresne violetta2105 gee-forr onerinas

active_record_doctor's Issues

Cannot load such file -- active_record_doctor/tasks/extraneous_indexes after upgrading from 1.7.1 => 1.8.0

Hi, thanks for the gem, we find it very useful!

Just to let you know that a minor-version upgrade from 1.7.1 => 1.8.0 is not backwards compatible, with errors such as
cannot load such file -- active_record_doctor/tasks/extraneous_indexes

As I couldn't see anything mentioning this in the changelog, I had a dig and it looks like this came from b192090 -
if this project aims to use semantic versioning, then this change should have been part of a major version change.

For the record, the fix was easy, we just renamed /tasks/ to /detectors/ in our require paths for active_record_doctor and the corresponding module name (ActiveRecordDoctor::Tasks => ActiveRecordDoctor::Detectors) where we explicitly referred to one.

Check for has_one backed by a unique index 

class User
  has_one :account
end

class Account
  belongs_to :user
end

has_one should be backed by a unique index (on accounts.user_id), because otherwise user can have multiple accounts in the database.

There is already a check MissingUniqueIndexes, checking unique validations without indexes. It is pretty generically named and seems like this new check would be added into there? Or a new check created?

@gregnavis, wdyt?

polymorphic associations bug

My model

# Table name: configs
#
#  configurable_id   :integer
#  configurable_type :string
#
# Indexes
#
#  index_configs_on_configurable_type_and_configurable_id  (configurable_type,configurable_id)

but tool do not see this association and try to add index to configurable_id

Support Rails 5.1.1

i'm on rails 5.1.1 (ruby 2.4.1) and it seems bundler resolves active_record_doctor to 1.0.1, which depends on rails 4.2.x:

active_record_doctor was resolved to 1.0.1, which depends on rails (~> 4.2)

if i track master, then it resolves to 1.3.0, but neither seem to support rails 5.1.x:

active_record_doctor was resolved to 1.3.0, which depends on rails (< 5.1, >= 4.2)

an update would be appreciated! 😊

ArgumentError: Polymorphic associations do not support computing the class.

I have a Polymorphic belongs_to like this:

belongs_to :subtype, polymorphic: true, dependent: :destroy, autosave: true

I get this error when running the rake task:

ArgumentError: Polymorphic associations do not support computing the class.
/usr/local/bundle/gems/activerecord-6.1.4.7/lib/active_record/reflection.rb:407:in `compute_class'
/usr/local/bundle/gems/activerecord-6.1.4.7/lib/active_record/reflection.rb:366:in `klass'
/usr/local/bundle/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/incorrect_dependent_option.rb:45:in `block (2 levels) in detect'
/usr/local/bundle/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/incorrect_dependent_option.rb:42:in `each'
/usr/local/bundle/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/incorrect_dependent_option.rb:42:in `block in detect'
/usr/local/bundle/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/incorrect_dependent_option.rb:35:in `each'
/usr/local/bundle/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/incorrect_dependent_option.rb:35:in `detect'

I guess a) it shouldn't error out and b) should give some advice on what to do.

Allow models backed by a DB view in undefined_table_references

We've introduced models backed by database views to improve some associations, and now the doctor is complaining that the model doesn't have a valid table.

Which fair point: it doesn't. But the model is valid and works well in production, so we've had to remove this task from our CI process as it was preventing the build from being successful.

undefined_table_references should both get the list of tables and views in the database, since ActiveRecord works well with views to back a (read only) model.

Don't know how to build task 'active_record_doctor:incorrect_length_validation'

It isn't currently possible to run the active_record_doctor:incorrect_length_validation task:

Don't know how to build task 'active_record_doctor:incorrect_length_validation' (See the list of available tasks with `rake --tasks`)
Did you mean?  active_record_doctor:incorrect_dependent_option
/usr/local/bundle/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'

Tested on:

Ruby ruby-2.5.3
  * active_record_doctor (1.9.0)
	Summary: Identify database issues before they hit production.
	Homepage: https://github.com/gregnavis/active_record_doctor
	Path: /usr/local/bundle/gems/active_record_doctor-1.9.0

False positive for extraneous Postgres trigram index

Here is a fragment from my schema.rb file that is flagged by rails active_record_doctor:extraneous_indexes as having an extraneous index that I believe is a false positive.

t.index ["code", "account_id"], name: "index_skus_on_code_and_account_id", unique: true
t.index ["code"], name: "index_skus_on_code_trgm", opclass: :gin_trgm_ops, using: :gin

The output from running rails active_record_doctor:extraneous_indexes is:

The following indexes are extraneous and can be removed:
  index_skus_on_code_trgm (can be handled by index_skus_on_code_and_account_id)

Thanks for making such a useful tool.

`MissingForeignKeys` detector assumptions

The checker makes an assumption that foreign-key like columns should be in the form of ..._id.

It can:

  1. lead to false positives, when some table have some column ending with _id which is not a reference to other table, but, for example, some id from the external resource (users.stripe_id).
  2. miss some custom named foreign keys: belongs_to :user, foreign_key: :user_key

I think, what should be really checked are models having non-polymorphic belongs_to associations without foreign keys.

Checking only models would make it easier to also ignore the case, when underlying models belong to different databases.

Use `SchemaCache` internally

We can use SchemaCache internally to not request the same information from the db over and over. I am sure, this will improve runtime on decent sizeable projects with many models/tables.

Unfortunately, I do not have a decent project under my hands right now to test the actual improvement.

Ignore or warn about models referencing non-existent internal Rails tables by default

On Rails 7 (and probably on Rails 6 too) with new app I will get following errors, when I run bundle exec rails active_record_doctor:undefined_table_references:

ActionMailbox::InboundEmail references a non-existent table or view named action_mailbox_inbound_emails
ActiveStorage::VariantRecord references a non-existent table or view named active_storage_variant_records
ActiveStorage::Blob references a non-existent table or view named active_storage_blobs
ActiveStorage::Attachment references a non-existent table or view named active_storage_attachments
ActionText::RichText references a non-existent table or view named action_text_rich_texts
ActionText::EncryptedRichText references a non-existent table or view named action_text_rich_texts

I think it is intended, but in many cases it should not be an error, because most new apps are not using those features a long time, if ever at all (e.g. InboundEmail).

The documentation says that those tables are ignored by default for other tasks.

Maybe it is too much to ignore them in this case, then it should be only a warning?!
Probably it is hard to detect, if those models are used and only in this case to produce an error?

Anonymous subclass of ActiveRecord::Base causes exception

As part of our test suite we run ActiveRecordDoctor and also create anonymous subclasses of ActiveRecord::Base - the latter causes an exception in the former

Traceback (most recent call last):
	9: from repro.rb:34:in `<main>'
	8: from /Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/runner.rb:14:in `run_one'
	7: from /Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/errors.rb:5:in `handle_exception'
	6: from /Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/runner.rb:15:in `block in run_one'
	5: from /Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/base.rb:11:in `run'
	4: from /Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/base.rb:40:in `run'
	3: from /Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/missing_unique_indexes.rb:32:in `detect'
	2: from /Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/base.rb:145:in `models'
	1: from /Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/base.rb:145:in `reject'
/Users/owenstephens/.rbenv/versions/2.7.4/lib/ruby/gems/2.7.0/gems/active_record_doctor-1.9.0/lib/active_record_doctor/detectors/base.rb:146:in `block in models': undefined method `start_with?' for nil:NilClass (NoMethodError)

here's a simple reproduction script:

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem "rails", "= 5.2.6.2"
  gem "active_record_doctor", "= 1.9.0"
end

require "active_record"
require "active_record_doctor"

some_anonymous_model = Class.new(ActiveRecord::Base)

runner = ActiveRecordDoctor::Runner.new(ActiveRecordDoctor.load_config_with_defaults(nil))
runner.run_one(:missing_unique_indexes)

Suggestions

  1. Currently, all tables are checked against some violations. But for me, it makes more sense to check only tables having a model within application. When the db schema is not kept in the cleanest state, it is possible to have ununsed tables and to get false positives on them.
  2. This gem does not support rails multiple databases feature.
  3. There are some steps in the README on how to create migrations, but it would be helpful to be able to generate migrations automatically (by providing some flag to the rake task), when it is appropriate

Would be happy to help with PRs, if this is something you are looking for.

NoMethodError: undefined method `include?' for #<Proc...>

I just installed active_record_doctor 1.10.0 and ran it for the first time and the following error was thrown:

% bundle exec rake active_record_doctor --trace
** Invoke active_record_doctor (first_time)
** Invoke active_record_doctor:setup (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute active_record_doctor:setup
** Execute active_record_doctor
rake aborted!
NoMethodError: undefined method `include?' for #<Proc:0x0000000113da8568 /Users/manuel/code/myapp/app/app/models/admin_users/admin_user.rb:46 (lambda)>

            !validator.options.fetch(:in, []).include?(nil)
                                             ^^^^^^^^^
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:57:in `block in inclusion_validator_present?'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:54:in `any?'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:54:in `inclusion_validator_present?'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:46:in `validator_present?'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:31:in `block (2 levels) in detect'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:29:in `each'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:29:in `block in detect'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:26:in `each'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/missing_presence_validation.rb:26:in `detect'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/base.rb:50:in `run'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/detectors/base.rb:17:in `run'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/runner.rb:15:in `block in run_one'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/errors.rb:5:in `handle_exception'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/runner.rb:14:in `run_one'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/runner.rb:26:in `block in run_all'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/runner.rb:25:in `each'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/runner.rb:25:in `run_all'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/active_record_doctor-1.10.0/lib/active_record_doctor/rake/task.rb:60:in `block in define'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/sentry-ruby-5.4.2/lib/sentry/rake.rb:26:in `execute'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/Users/manuel/.rbenv/versions/3.1.2/bin/rake:25:in `load'
/Users/manuel/.rbenv/versions/3.1.2/bin/rake:25:in `<top (required)>'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/cli/exec.rb:58:in `load'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/cli/exec.rb:58:in `kernel_load'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/cli/exec.rb:23:in `run'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/cli.rb:485:in `exec'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/cli.rb:31:in `dispatch'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/cli.rb:25:in `start'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/exe/bundle:48:in `block in <top (required)>'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/lib/bundler/friendly_errors.rb:120:in `with_friendly_errors'
/Users/manuel/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/bundler-2.3.20/exe/bundle:36:in `<top (required)>'
/Users/manuel/.rbenv/versions/3.1.2/bin/bundle:25:in `load'
/Users/manuel/.rbenv/versions/3.1.2/bin/bundle:25:in `<main>'
Tasks: TOP => active_record_doctor

The code at admin_user.rb:46 is:

validates :my_option, inclusion: { in:      -> { _1.role?(:superadmin, :admin) ?
                                                 [true, false] :
                                                 [false] },
                                   message: "can only be selected for admins with role superadmin or admin" }

Let me know if I can do anything else to debug!

Handling multiple schemas

We'd like to check multiple schemas in our DB. At present we are using the gem with great success in specs thus:

output = ActiveRecordDoctor::Tasks::UnindexedForeignKeys.new.run

However we can't see an easy way to move between schemas. Is that supported? Thanks!

Unhandled exception in missing_non_null_constraint

It looks like ARD crashes when some relations don't exist:

C:\Users\Lucius Riccio\Documents\GitHub\COVID-CO2-tracker\COVID_CO2_TRACKER>bundle exec rake active_record_doctor:missing_non_null_constraint --trace
** Invoke active_record_doctor:missing_non_null_constraint (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute active_record_doctor:missing_non_null_constraint
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "action_text_rich_texts" does not exist
LINE 8:  WHERE a.attrelid = '"action_text_rich_texts"'::regclass
                            ^
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `block (2 levels) in query'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:18:in `block in query'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:17:in `query'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:824:in `column_definitions'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:116:in `columns'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_non_null_constraint.rb:14:in `block in run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_non_null_constraint.rb:11:in `map'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_non_null_constraint.rb:11:in `run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/base.rb:7:in `run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/tasks/active_record_doctor.rake:17:in `block in mount'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `block in execute'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `each'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `execute'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `each'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:83:in `block in run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:80:in `run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
C:/Ruby30-x64/bin/rake:31:in `load'
C:/Ruby30-x64/bin/rake:31:in `<main>'

Caused by:
PG::UndefinedTable: ERROR:  relation "action_text_rich_texts" does not exist
LINE 8:  WHERE a.attrelid = '"action_text_rich_texts"'::regclass
                            ^
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `block (2 levels) in query'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:18:in `block in query'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activesupport-6.1.2.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:17:in `query'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/postgresql_adapter.rb:824:in `column_definitions'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/activerecord-6.1.2.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:116:in `columns'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_non_null_constraint.rb:14:in `block in run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_non_null_constraint.rb:11:in `map'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_non_null_constraint.rb:11:in `run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/base.rb:7:in `run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/active_record_doctor-1.7.1/lib/tasks/active_record_doctor.rake:17:in `block in mount'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `block in execute'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `each'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:281:in `execute'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `each'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:83:in `block in run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/lib/rake/application.rb:80:in `run'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
C:/Ruby30-x64/bin/rake:31:in `load'
C:/Ruby30-x64/bin/rake:31:in `<main>'
Tasks: TOP => active_record_doctor:missing_non_null_constraint

Migration names must be unique

I ran into an issue today when I was checking for missing indexes:

  • Generate missing index migrations & apply (bundle exec rake db:migrate) per the docs
  • Add a commit that modifies a table I just created missing index migrations for (the new commit is also missing an index)
  • Re-generate missing index migrations, and try to migrate again

This generates an error message Multiple migrations have the name IndexForeignKeysIn<Model>. The migration files themselves have different time stamps. but the migration names are the same inside those files.

It would be nice to support running the missing index migration multiple times, but by default it creates a generic name and it appears that Rails cares about the class name of the migration. I think it should be acceptable to append a timestamp into the name, so it's IndexForeignKeysIn<Model><timestamp>, so that might be an easy solution to implement.

This is more of a minor convenience issue; I envision using this tool periodically just to check and make sure we're got the right indexes and it would be nice to not have to change the class name.

I'm using Rails 5.2 along with the latest stable release of active_record_doctor.

I'm happy to submit a PR for this in the next few days unless someone else grabs it

:missing_unique_indexes task isn't registered

It isn't currently possible to run the active_record_doctor:missing_unique_indexes task:

% be rake active_record_doctor:missing_unique_indexes --trace
rake aborted!
Don't know how to build task 'active_record_doctor:missing_unique_indexes' (see --tasks)
Did you mean?  active_record_doctor:missing_foreign_keys
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/task_manager.rb:59:in `[]'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:159:in `invoke_task'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/Users/seandmr/.rbenv/versions/2.4.4/bin/rake:23:in `load'
/Users/seandmr/.rbenv/versions/2.4.4/bin/rake:23:in `<top (required)>'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/Users/seandmr/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>'
/Users/seandmr/.rbenv/versions/2.4.4/bin/bundle:23:in `load'
/Users/seandmr/.rbenv/versions/2.4.4/bin/bundle:23:in `<main>'

Running the undefined_table_references task loads Rails twice 

.../lib/group_state_manager.rb:32: warning: already initialized constant GroupStateManager::LOGGING_ENABLED
.../lib/group_state_manager.rb:32: warning: previous definition of LOGGING_ENABLED was here

I get duplicate initialization of files in my lib/ directory when running the undefined_table_references task (and only that one). See above for an example sanitized snippet.

Check constraint enforcing a column is NOT NULL does not satisfy missing_non_null_constraint

tl;dr: missing_non_null_constraint gives a false positive when a column has a validated check constraint enforcing that the column is NOT NULL.

We use Postgres, and have added a (validated) check constraint that ensures there are no NULL values in a particular column, but we're seeing a false positive from missing_non_null_constraint. We used a check constraint in place of a NOT NULL constraint since the table in question is large, and we didn't want to hold the ACCESS EXCLUSIVE lock taken by SET NOT NULL for the duration of scanning the whole table to assert that the column contains no NULL values. Adding a NOT VALID constraint is very quick (no lengthy table scan) and VALIDATE CONSTRAINT only takes a SHARE UPDATE EXCLUSIVE lock, which doesn't block concurrent read/writes to the table. Interestingly, it sounds like with Postgres 12 (which we're not yet using), if there is a validated CHECK CONSTRAINT enforcing NOT NULL on a column, SET NOT NULL can be performed without the expensive table scan as it can rely on the validated check constraint.

N.b. A check constraint can assert an arbitrary boolean expression, so it may be slightly tricky to determine if a given constraint always enforces NOT NULL, but I think for basic constraints it should be straightforward.

Here's a reproduction script, which assumes a PG DB has been created using createdb my_test_db and then the script is run as DB_NAME=my_test_db DB_USERNAME=... DB_PASSWORD=... ruby repro.rb:

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem "rails", "= 5.2.6.2"
  gem "pg", "= 1.3.2"
  gem "active_record_doctor", "= 1.9.0"
end

require "active_record"
require "active_record_doctor"

ActiveRecord::Base.establish_connection(
  adapter: "postgresql",
  database: ENV["DB_NAME"],
  username: ENV["DB_USERNAME"],
  password: ENV["DB_PASSWORD"]
)

# ActiveRecord::Base.logger = Logger.new(STDOUT)

ActiveRecord::Base.connection.execute(<<~SQL)
  DROP TABLE IF EXISTS things;

  CREATE TABLE things(name text, number integer NOT NULL);

  -- Ensure any newly created rows have a name
  ALTER TABLE things ADD CONSTRAINT name_not_null CHECK (name IS NOT NULL) NOT VALID;

  -- Ensure all existing rows conform to the constraint (typically performed separately due to the
  -- differing locks that are taken by both operations)
  ALTER TABLE things VALIDATE CONSTRAINT name_not_null;
SQL

class Thing < ActiveRecord::Base
  validates_presence_of :name
  validates_presence_of :number
end

runner = ActiveRecordDoctor::Runner.new(ActiveRecordDoctor.load_config_with_defaults(nil))
runner.run_one(:missing_non_null_constraint)

Thing.create!(name: "present", number: 1)

thing = Thing.new(name: nil, number: 2)

puts "Thing without name is valid?: #{thing.valid?}"

begin
  # Ignore the validation and save anyway - it should blow up
  thing.save!(validate: false)
rescue => e
  puts "Caught #{e.message}"
end

The relevant output:

add `NOT NULL` to things.name - models validates its presence but it's not non-NULL in the database
Thing without name is valid?: false
Caught PG::CheckViolation: ERROR:  new row for relation "things" violates check constraint "name_not_null"
DETAIL:  Failing row contains (null, 2).
: INSERT INTO "things" ("number") VALUES ($1)

Showing that there is a presence validation and that the DB prevents null values for name, but there is a false positive.

feature request: missing_presence_validation set flag to ignore if default set

Been using active_record_doctor:missing_presence_validation and find that over 75% of the columns listed have defaults defined in the db and in this case no active record validation is needed for these.

For me having a way to ignore any columns with defaults set would be a nice feature and filter out a lot of work not needed.

Great Gem and Thanks

Errors generate

Hey

I have this error:
/home/XXXX/.rvm/gems/ruby-2.3.3/gems/active_record_doctor-1.3.0/lib/generators/active_record_doctor/add_indexes/add_indexes_generator.rb:30:in content': undefined method camelize' for nil:NilClass (NoMethodError)

When I execute the command:
rails generate active_record_doctor:add_indexes unindexed_foreign_keys.txt

Thanks !

Document/Permit usage outside a rake task

Thanks a lot for this gem, really helpful 😄!

We added those to our test suite, ensuring we don't have new database problems in the future.

Here is an example of how we use it :

  describe 'Unindexed Foreign Keys' do
    before { Rails.application.load_tasks }

    let(:allowed) do
      {
        'users' => ['allowed_id'],
        'articles' => %w(other_id allowed_id),
      }
    end

    let(:missing) { ActiveRecordDoctor::Tasks::UnindexedForeignKeys.run.first }

    it do
      expect(allowed.keys).to match_array missing.keys

      allowed.each do |table, attributes|
        expect(attributes).to match_array missing[table]
      end
    end
  end

This works great, but I would prefer not having to load rake tasks to be able to do that (before { Rails.application.load_tasks }).
Also it would be nice to document this usage in the Readme!

Suggestion regarding indexes on boolean fields

Indexes on booleans are not as useful as people tend to think. On my laptop using postgres (default configuration) index was only used when <= 20% of rows satisfied the target value.

Most of the time, people add indexes for boolean columns like admin, deleted, banned, which have a distribution of 1% vs 99%, so the partial index would help.

So, I propose to suggest removing indexes on boolean fields or make them partial.

Check for missing length validations

It will be a good idea to require field length validations (validates :name, length: { maximum: 64 }) when a model has a field limit in the database (t.string :name, limit: 64).

This way we will not make a useless save and rollback to the database for invalid fields, but also have properly populated validation errors.

Default belongs_to validations don't trigger missing non-null constraint

When belongs_to relationships are required by default, as they are in Rails 5.0+ config defaults, i.e.

Rails.application.config.active_record.belongs_to_required_by_default = true

Then belongs_to relationships without optional: true have a validation attached that is not detected by MissingNonNullConstraint when that same column does not have a null: false constraint in the database.

I think these should be detected and reported?

it fetches sidekiq option as table and column

cat unindexed_foreign_keys.txt

Sidekiq concurrency: 25
membership_roles inherited_from_id
memberships member_id
projects parent_id

cat config/sidekiq.yml


---
:concurrency: <%= ENV['DB_POOL'] || ENV['RAILS_MAX_THREADS'] || 5 %>
(...omitted...)

undefined method `to_set' for String, when using a lowered index

My migration is:

add_index :users, 'lower(email)', unique: true

It translates in schema.rb as:

t.index "lower((email)::text)", name: "index_users_on_lower_email", unique: true

Note there are no surrounding brackets like we have for other fields:

add_index :users,  :auth_token, unique: true
t.index ["auth_token"], name: "index_users_on_auth_token", unique: true

The error is:

NoMethodError: undefined method `to_set' for "lower((email)::text)":String

          index.columns.to_set == columns.to_set && index.unique

in active_record_doctor-1.9.0/lib/active_record_doctor/detectors/missing_unique_indexes.rb:66:in block in unique_index?

This can be fixed by making sure index.columns is wrapped into an Array:

  def unique_index?(table_name, columns, scope)
    columns = (Array(scope) + columns).map(&:to_s)

    indexes(table_name).any? do |index|
      Array(index.columns).to_set == columns.to_set && index.unique
    end
  end

Error: ActiveRecord::StatementInvalid: Could not find table 'action_text_rich_texts' 

$ bundle exec rake active_record_doctor:missing_presence_validation

rake aborted!
ActiveRecord::StatementInvalid: Could not find table 'action_text_rich_texts'
~/.rvm/gems/ruby-2.6.6/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/sqlite3_adapter.rb:358:in `table_structure'
~/.rvm/gems/ruby-2.6.6/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:114:in `columns'
~/.rvm/gems/ruby-2.6.6/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_presence_validation.rb:14:in `block in run'
~/.rvm/gems/ruby-2.6.6/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_presence_validation.rb:11:in `map'
~/.rvm/gems/ruby-2.6.6/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/missing_presence_validation.rb:11:in `run'
~/.rvm/gems/ruby-2.6.6/gems/active_record_doctor-1.7.1/lib/active_record_doctor/tasks/base.rb:7:in `run'
~/.rvm/gems/ruby-2.6.6/gems/active_record_doctor-1.7.1/lib/tasks/active_record_doctor.rake:17:in `block in mount'
~/.rvm/gems/ruby-2.6.6/gems/airbrake-10.0.2/lib/airbrake/rake.rb:19:in `execute'
~/.rvm/gems/ruby-2.6.6/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
~/.rvm/gems/ruby-2.6.6/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
~/.rvm/gems/ruby-2.6.6/bin/bundle:23:in `load'
~/.rvm/gems/ruby-2.6.6/bin/bundle:23:in `<main>'
Tasks: TOP => active_record_doctor:missing_presence_validation
(See full trace by running task with --trace)

undefined method to_h for Array

Just tried on current project (ruby 2.0.0p598 / rails 4.2.5.1):
➜ ✗ rake active_record_doctor:unindexed_foreign_keys > unindexed_foreign_keys.txt

rake aborted!
NoMethodError: undefined method to_h for #Array:0x007faf6c25abc0
/Users/salf/.rvm/gems/ruby-2.0.0-p598/gems/active_record_doctor-1.0.2/lib/active_record_doctor/tasks/unindexed_foreign_keys.rb:32:in `unindexed_foreign_keys'

missing_presence_validation: unexpected error on HABTM join tables

Hi!

when running rake active_record_doctor:missing_presence_validation, the gem reports a missing validation error on the FK columns on join tables (used in HABTM relations).

Is this really the intended behavior of this gem? Since there is no model for the join table, it shouldn't report an error IMHO.

Example with schema:

  create_table "pets", force: :cascade do |t|
    t.string "name", null: false
    t.date "birth", null: false
  end

  create_table "pets_users", id: false, force: :cascade do |t|
    t.integer "users_id", null: false
    t.integer "pets_id", null: false
    t.index ["pets_id"], name: "index_pets_users_on_pets_id"
    t.index ["users_id"], name: "index_pets_users_on_users_id"
  end

  create_table "users", force: :cascade do |t|
    t.string "email"
  end

Pet:

  class Pet < ApplicationRecord
    has_and_belongs_to_many :users
    validates :name, presence: true
    validates :birth, presence: true
  end

User model

class User < ApplicationRecord
  has_and_belongs_to_many :pets
end

Output:

The following models and columns should have presence validations:
  HABTM_Users: users_id, pets_id
  HABTM_Pets: users_id, pets_id

Expected output: no error

  • gem version: active_record_doctor (1.8.0)

Task extraneous_indexes crashes on duplicate index

A table with two indexes both pointing to the same table column, crashes task extraneous_indexes. Example indexes shown below, including error message:

CREATE INDEX others_thing_fkey
  ON others
  USING btree
  (thing_id);

CREATE INDEX thing_fkey
  ON others
  USING btree
  (thing_id);
NoMethodError: undefined method `name' for nil:NilClass
/ruby-2.2.0/gems/active_record_doctor-1.1.1/lib/active_record_doctor/tasks/extraneous_indexes.rb:39:in `block (2 levels) in extraneous_indexes'
/ruby-2.2.0/gems/active_record_doctor-1.1.1/lib/active_record_doctor/tasks/extraneous_indexes.rb:34:in `map'
/ruby-2.2.0/gems/active_record_doctor-1.1.1/lib/active_record_doctor/tasks/extraneous_indexes.rb:34:in `block in extraneous_indexes'
/ruby-2.2.0/gems/active_record_doctor-1.1.1/lib/active_record_doctor/tasks/extraneous_indexes.rb:24:in `map'
/ruby-2.2.0/gems/active_record_doctor-1.1.1/lib/active_record_doctor/tasks/extraneous_indexes.rb:24:in `extraneous_indexes'
/ruby-2.2.0/gems/active_record_doctor-1.1.1/lib/active_record_doctor/tasks/extraneous_indexes.rb:15:in `run'
/ruby-2.2.0/gems/active_record_doctor-1.1.1/lib/active_record_doctor/tasks/extraneous_indexes.rb:7:in `run'
/ruby-2.2.0/gems/active_record_doctor-1.1.1/lib/tasks/active_record_doctor_tasks.rake:10:in `block (2 levels) in <top (required)>'

Great Gem! very helpful. Let me know if you require further details.
Thanks

Incorrect extraneous index calculations

Applicable readme section: https://github.com/gregnavis/active_record_doctor#removing-extraneous-indexes.

For the sake of this explanation, let's say we have a rooms table that has rooms.room_number and rooms.floor_number. We build a multi-column index with floor_number first, room_number second.

A multi-column index in SQL works by (essentially) concatenating the values of the two columns. Here's a simplified drawing of that:

floor_number_room_number_index
1;1
1;2
1;3
2;1
2;2
2;3

Indexes are fast in part because they group the values. In the example above, only the floor_number is grouped together. So this index will work for floor_number. But room_number is not grouped, and the database is smart enough to know that. It will not use this index for room_number, but will do a full table search instead.

The worst part of all this is that it is even more complicated than that. Check out this article for a more complete explanation. There's also a useful SO question on the topic.

Fix: Fixes readme at incorrect dependent option

At incorrect dependent option says that you can use the configurations enabled, ignore_models and ignore_columns, on that case instead of ignore_columns the configuration allows you to use ignore_associations

Support for rails5 

$ bundle install
Fetching gem metadata from https://rails-assets.org/...
Fetching version metadata from https://rails-assets.org/..
Fetching gem metadata from https://rails-assets.org/...
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rails-assets.org/..
Fetching version metadata from https://rubygems.org/
Fetching dependency metadata from https://rails-assets.org/..
Fetching dependency metadata from https://rubygems.org/
Resolving dependencies...
Bundler could not find compatible versions for gem "rails":
  In snapshot (Gemfile.lock):
    rails (= 5.0.0)

  In Gemfile:
    rails (~> 5.0.0)

    active_record_doctor was resolved to 1.0.1, which depends on
      rails (~> 4.2)

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

Unique and non-unique indexes are considered equivalent

Problem

A non-unique index should never be considered an equivalent of a unique index. If the two indexes are on the same columns then the unique one should be preferred as dropping it would change the semantics of the app. There are a few cases to consider.

Case 1: identical columns

With indexes like:

    add_index :users, :last_name
    add_index :users, :last_name, unique: true, name: :unique_index_on_users_last_name

active_record_doctor is likely to recommend dropping the first index which is wrong. The first index should be dropped.

Case 2: proper unique prefix

For example:

    add_index :users, [:last_name, :first_name]
    add_index :users, :last_name, unique: true

In this case no index should be dropped.

Case 3: proper prefix of an unique index

For example:

    add_index :users, [:last_name, :first_name], unique: true
    add_index :users, :last_name

In the situation above, the second index should be dropped. (Note that case 1 is a special case of case 3).

Display list of tasks at top of readme

Scrolling through the README to find the name of the task I want to run is a little cumbersome. What do you think about listing them at the top of the file with links to the full descriptions?

Error on missing_foreign_keys 

10343 % rake active_record_doctor:missing_foreign_keys
rake aborted!
NotImplementedError: foreign_keys is not implemented
<gems>/activerecord-5.0.3/lib/active_record/connection_adapters/abstract/schema_statements.rb:864:in `foreign_keys'
<gems>/active_record_doctor-1.4.1/lib/active_record_doctor/tasks/missing_foreign_keys.rb:48:in `foreign_key?'
<gems>/active_record_doctor-1.4.1/lib/active_record_doctor/tasks/missing_foreign_keys.rb:34:in `block (2 levels) in missing_foreign_keys'
<gems>/active_record_doctor-1.4.1/lib/active_record_doctor/tasks/missing_foreign_keys.rb:29:in `select'
<gems>/active_record_doctor-1.4.1/lib/active_record_doctor/tasks/missing_foreign_keys.rb:29:in `block in missing_foreign_keys'
<gems>/active_record_doctor-1.4.1/lib/active_record_doctor/tasks/missing_foreign_keys.rb:26:in `map'
<gems>/active_record_doctor-1.4.1/lib/active_record_doctor/tasks/missing_foreign_keys.rb:26:in `missing_foreign_keys'
<gems>/active_record_doctor-1.4.1/lib/active_record_doctor/tasks/missing_foreign_keys.rb:18:in `run'
<gems>/active_record_doctor-1.4.1/lib/active_record_doctor/tasks/missing_foreign_keys.rb:10:in `run'
<gems>/active_record_doctor-1.4.1/lib/tasks/active_record_doctor_tasks.rake:16:in `block (2 levels) in <top (required)>'
Tasks: TOP => active_record_doctor:missing_foreign_keys
(See full trace by running task with --trace)
10357 %

Improvements

I have some improvements in mind, but need to verify this is aligned with the project goals, before implementing.

  1. Currently, we can run one check at a time. Would be better to have some command (bin/active_record_doctor) to simply run all checks.
  2. There is no way to configure the behavior. Would be good to have some config file (.yml) to be able to setup, for example, which checks to skip or which table/columns to ignore.
  3. There is some code regarding loading/reloading - minitest-fork_executor gem, test/model_factory.rb, code in test/setup.rb. I can reimplement how things are tested and get rid of all of that.
  4. Gem depends on rails, railties, activerecord, activesupport, but should really depend on activerecord only
  5. Gem supports only postgresql. Would be great to support also mysql. For sqlite it seems useless.
  6. There is no way to easily run a test suite against all the Gemfile's - needs to use BUNDLE_GEMFILE=... bundle exec rake. May be a good idea to add a dev dependency on appraisals gem to do this.
  7. Make it easier to use on CI.
  8. Some checks can cause a lot of unnecessary db queries while checking (for example,

    active_record_doctor/lib/active_record_doctor/detectors/missing_foreign_keys.rb

    Lines 22 to 23 in a9ab257

    !foreign_key?(table, column) &&
    !polymorphic_foreign_key?(table, column)
    ), so to use some caching/memoization
  9. Be able to run it on non-rails applications

undefined_table_references - uninitialized constant ActiveRecord::ConnectionAdapters::PostgreSQLAdapter

When running bundle exec rake active_record_doctor:undefined_table_references on an app (Rails 6, FWIW) with MySQL, it throws a NameError: uninitialized constant ActiveRecord::ConnectionAdapters::PostgreSQLAdapter exception. Obviously, there's a DB type check missing somewhere. All others are functional. This isn't something I'm worried about at the moment or I'd try to form a PR, but just passing it on.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.