chrisfrank / rack-reducer

Declaratively filter data via URL params, in any Rack app, with any ORM.

Rack::Reducer

Declaratively filter data via URL params, in any Rack app, with any ORM.

Install

Add rack-reducer to your Gemfile:

gem 'rack-reducer', require: 'rack/reducer'

Rack::Reducer has no dependencies beyond Rack itself.

Use

If your app needs to render a list of database records, you probably want those records to be filterable via URL params, like so:

GET /artists => all artists
GET /artists?name=blake` => artists named 'blake'
GET /artists?genre=electronic&name=blake => electronic artists named 'blake'

Rack::Reducer can help. It applies incoming URL params to an array of filter functions you define, runs only the relevant filters, and returns your filtered data. Here’s how you might use it in a Rails controller:

# app/controllers/artists_controller.rb
class ArtistsController < ApplicationController

  # Step 1: Instantiate a reducer
  ArtistReducer = Rack::Reducer.new(
    Artist.all,
    ->(name:) { where('lower(name) like ?', "%#{name.downcase}%") },
    ->(genre:) { where(genre: genre) },
  )

  # Step 2: Apply the reducer to incoming requests
  def index
    @artists = ArtistReducer.apply(params)
    render json: @artists
  end
end

This example app would handle requests as follows:

# GET /artists => All artists:
[
  { "name": "Blake Mills", "genre": "alternative" },
  { "name": "Björk", "genre": "electronic" },
  { "name": "James Blake", "genre": "electronic" },
  { "name": "Janelle Monae", "genre": "alt-soul" },
  { "name": "SZA", "genre": "alt-soul" }
]

# GET /artists?name=blake => Artists named "blake":
[
  { "name": "Blake Mills", "genre": "alternative" },
  { "name": "James Blake", "genre": "electronic" }
]

# GET /artists?name=blake&genre=electronic => Electronic artists named "blake"
[{ "name": "James Blake", "genre": "electronic" }]

API Documentation

https://www.rubydoc.info/gems/rack-reducer

Framework-specific Examples

These examples apply Rack::Reducer in different frameworks and ORMs. The pairings of ORMs and frameworks are arbitrary, just to demonstrate a few possible stacks.

Sinatra/Sequel

This example uses Sinatra to handle requests, and Sequel as an ORM.

# config.ru
class SinatraExample < Sinatra::Base
  DB = Sequel.connect ENV['DATABASE_URL']

  # dataset is a Sequel::Dataset, so filters use Sequel query methods
  ArtistReducer = Rack::Reducer.new(
    DB[:artists],
    ->(genre:) { where(genre: genre) },
    ->(name:) { grep(:name, "%#{name}%", case_insensitive: true) },
  )

  get '/artists' do
    @artists = ArtistReducer.apply(params).all
    @artists.to_json
  end
end

Rack Middleware/Ruby Array

This example runs a raw Rack app with Rack::Reducer mounted as middleware. It doesn't use an ORM at all -- it just stores data in a ruby array.

# config.ru
require 'rack'
require 'rack/reducer'
require 'json'

ARTISTS = [
  { name: 'Blake Mills', genre: 'alternative' },
  { name: 'Björk', genre: 'electronic' },
  { name: 'James Blake', genre: 'electronic' },
  { name: 'Janelle Monae', genre: 'alt-soul' },
  { name: 'SZA', genre: 'alt-soul' },
]

app = Rack::Builder.new do
  # dataset is an Array, so filter functions use Array methods
  use Rack::Reducer::Middleware, dataset: ARTISTS, filters: [
    ->(genre:) { select { |item| item[:genre].match(/#{genre}/i) } },
    ->(name:) { select { |item| item[:name].match(/#{name}/i) } },
    ->(sort:) { sort_by { |item| item[sort.to_sym] } },
  ]
  run ->(env) { [200, {}, [env['rack.reduction'].to_json]] }
end

run app

When Rack::Reducer is mounted as middleware, it stores its filtered data in env['rack.reduction'], then calls the next app in the middleware stack. You can change the env key by passing a new name as option to use:

use Rack::Reducer::Midleware, key: 'custom.key', dataset: ARTISTS, filters: [
  # an array of lambdas
]

With Rails scopes

The Rails quickstart example created a reducer inside a controller, but if your filters use lots of ActiveRecord scopes, it might make more sense to keep your reducers in your models instead.

# app/models/artist.rb
class Artist < ApplicationRecord
  # filters get instance_exec'd against the dataset you provide -- in this case
  # it's `self.all` -- so filters can use query methods, scopes, etc
  Reducer = Rack::Reducer.new(
    self.all,
    ->(name:) { by_name(name) },
    ->(genre:) { where(genre: genre) },
    ->(sort:) { order(sort.to_sym) }
  )

  scope :by_name, lambda { |name|
    where('lower(name) like ?', "%#{name.downcase}%")
  }
end

# app/controllers/artists_controller.rb
class ArtistsController < ApplicationController
  def index
    @artists = Artist::Reducer.apply(params)
    render json: @artists
  end
end

Default filters

Most of the time it makes sense to use required keyword arguments for each filter, and skip running the filter altogether when the keyword argments aren't present.

But sometimes you'll want to run a filter with a default value, even when the required params are missing. The code below will order by params[:sort] when it exists, and by name otherwise.

class ArtistsController < ApplicationController
  ArtistReducer = Rack::Reducer.new(
    Artist.all,
    ->(genre:) { where(genre: genre) },
    ->(sort: 'name') { order(sort.to_sym) }
  )

  def index
    @artists = ArtistReducer.apply(params)
    render json: @artists
  end
end

Calling Rack::Reducer as a function

For a slight performance penalty (~5%), you can skip instantiating a reducer via ::new and just call Rack::Reducer as a function. This can be useful when prototyping, mostly because you don't need to think about naming anything.

# app/controllers/artists_controller.rb
class ArtistsController < ApplicationController
  # Step 1: there is no step 2
  def index
    @artists = Rack::Reducer.call(params, dataset: Artist.all, filters: [
      ->(name:) { where('lower(name) like ?', "%#{name.downcase}%") },
      ->(genre:) { where(genre: genre) },
    ])
    render json: @artists
  end
end

How Rack::Reducer Works

Rack::Reducer takes a dataset, an array of lambdas, and a params hash.

To return filtered data, it calls Enumerable#reduce on your array of lambdas, with the reduction's initial value set to dataset.

Each reduction looks for keys in the params hash that match the current lambda's keyword arguments. If the keys exist, it instance_execs the lambda against the dataset, passing just those keys as arguments, and finally passes the filtered dataset on to the next lambda.

Lambdas that don't find all their required keyword arguments in params don't execute at all, and just pass the unaltered dataset down the chain.

The reason Reducer works with any ORM is that you supply the dataset and filter functions. Reducer doesn't need to know anything about ActiveRecord, Sequel, Mongoid, etc -- it just instance_execs your own code against your own dataset.

Performance

For requests with empty params, Rack::Reducer has no measurable performance impact. For requests with populated params, Rack::Reducer is about 10% slower than a set of hand-coded conditionals, according to spec/benchmarks.rb.

 Conditionals (full)   530.000  i/100ms
      Reducer (full)   432.000  i/100ms
Conditionals (empty)   780.000  i/100ms
     Reducer (empty)   808.000  i/100ms
Calculating -------------------------------------
 Conditionals (full)      4.864k (± 2.3%) i/s -     24.380k in   5.015551s
      Reducer (full)      4.384k (± 1.3%) i/s -     22.032k in   5.026651s
Conditionals (empty)      7.889k (± 1.7%) i/s -     39.780k in   5.043797s
     Reducer (empty)      8.129k (± 1.7%) i/s -     41.208k in   5.070453s

Comparison:
     Reducer (empty):     8129.5 i/s
Conditionals (empty):     7889.3 i/s - same-ish: difference falls within error
 Conditionals (full):     4863.7 i/s - 1.67x  slower
      Reducer (full):     4383.8 i/s - 1.85x  slower

In Rails, note that params is never empty, so use request.query_parameters instead if you want to handle parameterless requests at top speed.

# app/controllers/artists_controller.rb
class ArtistController < ApplicationController
  # ArtistReducer = Rack::Reducer.new(...etc etc)

  def index
    @artists = ArtistReducer.apply(request.query_parameters)
    render json: @artists
  end
end

Alternatives

If you're working in Rails, Plataformatec's excellent HasScope has been solving this problem since 2009. I prefer keeping my request logic all in one place, though, instead of spreading it across my controllers and models.

Periscope, by Steve Richert, seems like another solid Rails option. It is Rails-only, but it supports more than just ActiveRecord.

For Sinatra, Simon Courtois has a Sinatra port of has_scope. It depends on ActiveRecord.

Contributors

Thank you @danielpuglisi, @nicolasleger, @jeremyshearer, and @shanecav84 for helping improve Rack::Reducer!

Contributing

Bugs

Please open an issue on Github.

Pull Requests

PRs are welcome, and I'll do my best to review them promptly.

License

MIT

Copyright 2018 Chris Frank

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

rack-reducer's Issues

Filter not working with more than one integer/enum filter?

Hello,

I am using Rack::Reducer to develop a Rails application where I want to filter books by various attributes. I have two attributes :status and :pubtypewhich are integers on the database level, but implemented as enums in my Book model. I have written both filters in exactly the same way:

# Code in Book model
Reducer = Rack::Reducer.new(
  self.all,
  # some more attribute filters here...   
  ->(status:) { status == "" ? where("status = IFNULL(@status, status) OR (status IS NULL)") : by_status(status) },
  ->(pubtype:) { pubtype == "" ? where("pubtype = IFNULL(@pubtype, pubtype) OR (pubtype IS NULL)") : by_pubtype(pubtype) },
)

With this code and a corresponding form and view, :status is correctly filtered and only books with the desired status are shown in my view. (It is also possible to combine the filter with others like :author, :genre etc.) Trying to filter by :pubtype, however, produces a view with no books.

I have been trying for several hours to get both filters to work. Cutting out all the "noise" (from the other filters etc.), it seems to boil down to this: Only the integer filter that comes first works. In the code above, only the :status filter works correctly. If I reverse the two lines, only the :pubdate filter works.

I do not experience this behavior with my string filters - I have lots of them and they all work just fine.

Tell me if you need more info and/or code snippets and I will see that I can provide them.

Looking forward to your answer!

Handling ranges with optional parameters

Hey Chris! Any tips on how to filter with optional min/max query parameters? Optional meaning users could query one, both, or none.

For example:

count_min=5 -> would return all results greater than 5
count_min=5?count_max=10 -> would return all results between 5 and 10 inclusive

Default filters not working when params are empty

Hey @chrisfrank

First of all, thanks for this amazing gem. Started using v1.0.1 at the beginning of the year and was able to clean up a lot of index actions 👍😉

Tried v1.1.0 in a new project today and noticed a slight change in how default filters are working (or not working) due to the addition in https://github.com/chrisfrank/rack-reducer/blob/master/lib/rack/reducer/reduction.rb#L22 which causes them to be skipped when params are empty. Not sure if this is intentional. I used them for defining default ordering so far.

RFC re: proposed changes in 2.0

I'd like to propose two API changes for 2.0, aimed at improving performance, simplifying the docs, and making it easier to integrate Rack::Reducer consistently across different Rack stacks.

1. Unify the functional and mixin-style APIs
2. Drop or move the middleware API

These changes could be mostly backward-compatible, but they'd be cleaner to implement as breaking changes. I'm eager for input before I make a decision.

1. Unify the functional and mixin APIs.

Rack::Reducer’s functional style is verbose, and it inefficiently encourages allocating a new array of filter functions on every request. How inefficient this is depends on your setup, but from rough benchmarks it's ~10% slower and ~30% less memory efficient than the mixin style.

The mixin style, on the other hand, is tightly coupled to Rails-ish models, and I don't like that it defines reduce on a class that in practice often returns an an Enumerable, which already has its own implementation of reduce.

I propose unifying the two APIs into one. It should be terse and efficient like the mixin style, and self-contained like the functional style. Here's what I have in mind:

# Proposed unified API for 2.0
class App < SinatraLike::Base
  class Artist < SomeORM::Model
  end

  # Instantiate a "reducer" once, on app boot
  ArtistsReducer = Rack::Reducer.new(
    Artist.all,
    ->(genre:) { where(genre: genre) },
    ->(sort:) { order(sort.to_sym) },
  )

  get '/artists' do
    # Call the reducer on each request
    @artists = ArtistsReducer.call(params)
    @artists.all.to_json
  end
end

# Current 1.0 functional style, for comparison
class FunctionalStyle < SinatraLike::Base
  class Artist < SomeORM::Model
  end

  get '/artists' do
    # this allocates a new array of filters on each request :(
    @artists = Rack::Reducer.call(params, dataset: Artist.all, filters: [
      ->(genre:) { where(genre: genre) },
      ->(sort:) { order(sort.to_sym) },
    ])
    @artists.all.to_json
  end
end

# Current 1.0 mixin style, for comparison
class MixinStyle < SinatraLike::Base
  class Artist < SomeORM::Model
    extend Rack::Reducer
    reduces self.all, filters: [
      ->(genre:) { where(genre: genre) },
      ->(sort:) { order(sort.to_sym) },
    ]
  end

  get '/artists' do
    @artists = Artist.reduce(params)
    @artists.all.to_json
  end
end

With this unified API in place, we could either leave the 1.0 APIs intact for backward compatibility, or drop them in pursuit of simplicity and enforcing fast defaults. I would prefer to drop them, but would be happy to be convinced otherwise.

2. Drop or move the Middleware API

Rack mounts middleware by calling ::new(app, options), but I want to use Rack::Reducer.new for the API outlined above.

I’m tempted to drop the middleware API entirely, because I've never needed to use it in a real app. Any I time I could have used middleware, it has been more practical to call Rack::Reducer as a function.

If you've found a useful case for Rack::Reducer as middleware, I'd be open to including it in 2.0 under a slightly different API:

# config.ru

# 1.0 (not supported in 2.0)
# use Rack::Reducer
# run MyApp

# 2.0 (proposed)
use Rack::Reducer::Middleware
run MyApp

Thanks in advance for your input.

ActionController::Parameters issue with nested parameters

Hi @chrisfrank

I just stumbled across an issue with Rails's ActionController::Parameters in filters with nested parameters. Example:

->(date:) { between(date[:from], date[:to]) }

This results in nil values for date[:from] and date[:to], because Rails is converting ActionController::Parameters to HashWithIndifferentAccess objects when calling #to_unsafe_h in https://github.com/chrisfrank/rack-reducer/blob/master/lib/rack/reducer.rb#L60. Which doesn't trigger the Hash#symbolize_keys from the refinements but the default implementation, causing the nested parameters to be converted back to strings.

pry(main)> ActionController::Parameters.new({ date: { 'from' => '2019-01-01' } }).to_unsafe_h.symbolize_keys
=> {:date=>{"from"=>"2019-01-01"}}

Not sure how to solve this yet, but I'm looking for a solution.

Reducer is not refreshed with latest data

In the README example (pasted below), ArtistReducer is not refreshed with latest Artist.all when a new artist is created. To reproduce the issue:

• visit index page with no parameters. Expect all artists to appear.
• create a new record
• revisit index page. Expect new record to not appear, when it should appear.

# app/controllers/artists_controller.rb
class ArtistsController < ApplicationController

  # Step 1: Instantiate a reducer
  ArtistReducer = Rack::Reducer.new(
    Artist.all,
    ->(name:) { where('lower(name) like ?', "%#{name.downcase}%") },
    ->(genre:) { where(genre: genre) },
  )

  # Step 2: Apply the reducer to incoming requests
  def index
    @artists = ArtistReducer.apply(params)
    render json: @artists
  end
end