lukearno / selector

WSGI request delegation. (AKA routing.)

Selector

WSGI request delegation. (AKA routing.)

$ pip install selector

Overview

This distribution provides WSGI middleware for "RESTful" dispatch of requests to WSGI applications by URL path and HTTP request method. Selector now also comes with components for environ-based dispatch and on-the-fly middleware composition. There is a very simple optional mini-language for path matching expressions. Alternately we can easily use regular expressions directly or even create our own mini-language. There is a simple "mapping file" format that can be used. There are no architecture specific features (to MVC or whatever). Neither are there any framework specific features.

Quick Start

import selector

app = selector.Selector()
app.add('/resource/{id}', GET=wsgi_handler)

If you have ever designed a REST protocol you have probably made a table that looks something like this:

/foos/{id}
POST	Create a new foo with id == {id}.
GET	Retrieve the foo with id == {id}.
PUT	Update the foo with id == {id}.
DELETE	Delete the foo with id == {id}.

Selector was designed to fit mappings of this kind.

Lets suppose that we are creating a very simple app. The only requirement is that http://example.com/myapp/hello/Guido responds with a simple page that says hello to Guido (where "Guido" can actually be any name at all). The interface of this extremely useful service looks like:

/myapp/hello/{name}
GET	Say hello to {name}.

Here's the code for myapp.py:

from selector import Selector

def say_hello(environ, start_response):
    args, kwargs = environ['wsgiorg.routing_args']
    start_response("200 OK", [('Content-type', 'text/plain')])
    return ["Hello, %s!" % kwargs['name']]
    
app = Selector()
app.add('/myapp/hello/{name}', GET=say_hello)

Run it with Green Unicorn:

$ gunicorn myapp:app

Of course, you can use Selector in any WSGI environment.

How It Works

When a route is added, the path expression is converted into a regular expression. (You can also use regexes directly.) When the Selector instance receives a request, it checks each regex until a match is found. If no match is found, the request is passed to Selector.status404. Otherwise, it modifies the environ to store some information about the match and looks up the dict of HTTP request methods associated with the regex. If the HTTP method is not found in the dict, the request is passed to Selector.status405. Otherwise, the request is passed to the WSGI handler associated with the HTTP method.

Path Expressions

As you probably noticed, you can capture named portions of the path into environ['wsgiorg.routing_args']. (They also get put into environ['selector.vars'], but that is deprecated in favor of a routing args standard.)

You can also capture things positionally:.

def show_tag(environ, start_response):
    args, kwargs = environ['wsgiorg.routing_args']
    user = kwargs['user']
    tag = args[0]
    # ...

s.add('/myapp/{user}/tags/{}', GET=show_tag)

Selector supports a number of datatypes for your routing args, specified like this: {VARNAME:DATATYPE} or just {:DATATYPE}.

type	regex
word	\w+
alpha	[a-zA-Z]+
digits	\d+
number	\d*.?\d+
chunk	[^/^.]+
segment	[^/]+
any	.+

These types work for both named and positional routing args:

s.add('/collection/{:digits}/{docname:chunk}.{filetype:chunk}', GET=foo)

(You can even add your own types with just a name and a regex, but we will get to that in a moment.)

Parts of the URL path can also be made optional using [square brackets.]

s.add("/required-part[/optional-part]", GET=any_wsgi)

Optional portions in path expressions can be nested.

s.add("/recent-articles[/{topic}[/{subtopic}]][/]", GET=recent_articles)

By default, selector does path consumption, which means the matched portion of the path information is moved from environ['PATH_INFO'] to environ['SCRIPT_NAME'] when routing a request. The matched portion of the path is also appended to a list found or created in environ['selector.matches'], where it is is available to upstack consumers. It's useful in conjunction with open ended path expressions (using the pipe character, |) for recursive dispatch:

def load_book(environ, start_response):
    args, kwargs = environ['wsgiorg.routing_args']
    # load book
    environ['com.example.book'] = db.get_book(kwargs['book_id'])
    return s(environ, start_response)

def load_chapter(environ, start_response):
    book = environ['com.example.book']
    args, kwargs = environ['wsgiorg.routing_args']
    chapter = book.chapters[kwargs['chapter_id'])
    # ... send some response

s.add("/book/{book_id}|", GET=load_book)
s.add("/chapter/{chapter_id}", GET=load_chapter)

Plain Regexes, Custom Types and Custom Parsers

You can create your own parser and your own path expression syntax, or use none at all. All you need is a callable that takes the path expression and returns a regex string.

s.parser = lambda x: x
s.add('^\/somepath\/$', GET=foo)

You can add a custom type to the default parser when you instantiate it or by modifying it in place.

parser = selector.SimpleParser(patterns={'mytype': 'MYREGEX'})
assert parser('/{foo:mytype}') == r'^\/(?P<foo>MYREGEX)$'

s.parser.patterns['othertype'] = 'OTHERREGEX'
assert parser('/{foo:othertype}') == r'^\/(?P<foo>OTHERREGEX)$'

Prefix and Wrap

Often you have some common prefix you would like appended to your path expressions automatically when you add routes. You can set that when instantiating selector and change it as you go.

# Add the same page under three prefixes:
s = Selector(prefix='/myapp')
s.add('/somepage', GET=get_page)
s.prefix = '/otherapp'
s.add('/somepage', GET=get_page)
s.add('/somepage', GET=get_page, prefix='/app3')

Selector can automatically wrap the callables you route to. I often use Yaro, which puts WSGI behind a pretty request object.

import selector, yaro  
  
def say_hello(req):  
    return "Hello, World!"  
  
s = selector.Selector(wrap=yaro.Yaro)  
s.add('/hello', GET=say_hello) 

Adding Routes

There are basically three ways to add routes.

One at a Time

So far we have been adding routes with .add()

foo_handlers = {'GET': get_foo, 'POST': create_foo}

s.add('/foo', method_dict=foo_handlers)
s.add('/bar', GET=bar_handler)
s.add('/read-only-foo',
      method_dict=foo_handlers,
      POST=sorry_charlie)
)

Notice how POST was overridden for /read-only-foo.

.add() also takes a prefix key word arg.

Slurping up a List

.slurp() will load mapping from a list of tuples, which turns out to be pretty ugly, so you would probably only do this if you were building the list programmatically. (... like, if parsing your own URL mapping file format, for instance.)

routes = [('/foo', {'GET': foo}),
          ('/bar', {'GET': bar})]
s = Selector(mappings=routes)
# or
s.slurp(routes)

.slurp() takes the keyword args prefix, parser and wrap...

Mapping Files

Selector supports a sweet URL mapping file format.

/foo/{id}[/]  
    GET somemodule:some_wsgi_app  
    POST pak.subpak.mod:other_wsgi_app  
  
@prefix /myapp  
@wrap yaro:Yaro

/path[/]  
    GET module:app  
    POST package.module:get_app('foo')  
    PUT package.module:FooApp('hello', resolve('module:setting'))  
  
@parser :lambda x: x  
@wrap :lambda x: x  

@prefix  
^/spam/eggs[/]$  
    GET mod:regex_mapped_app

This format is read line by line.

• Blank lines and lines starting with # as their first non-whitespace characters are ignored.
• Directives start with @ and modulate route adding behavior.
• Path expressions come on their own line and have no leading whitespace
• HTTP method -> handler mappings are indented

There are three directives: @prefix, @parser and @wrap, and they do what you think they do. The @parser and @wrap directives take resolver statements. Handlers are resolver statements too. HTTP method to handler mappings are applied to the preceding path expression.

Files of this format can be used in the following ways.

s = Selector(mapfile='map1.urls')
s.slurp_file('map2.urls')

Selector.slurp_file() supports optional prefix, parser and wrap keyword arguments, too.

Initializing a Selector

All the functionality is covered above, but, to summarize the init signature:

    def __init__(self,
                 mappings=None,
                 prefix="",
                 parser=None,
                 wrap=None,
                 mapfile=None,
                 consume_path=True):

Customizing 404s and 405s and Chain Dispatchers

You can replace Selector's 404 and 405 handlers. They're just WSGI.

s = Selector()
s.status404 = my_404_wsgi
s.status405 = my_405_wsgi

You could chain Selector instances together, or fall through to other types of dispachers or any handler at all really.

s1 = Selector(mapfile='map1.urls')
s2 = Selector(mapfile='map2.urls')
s1.status404 = s2

Environ Dispatcher

EnvironDispatcher routes a request based on the environ. It's instantiated with a list of (predicate, wsgi_app) pairs. Each predicate is a callable that takes one argument (environ) and returns True or False. When called, the instance iterates through the pairs until it finds a predicate that returns True and runs the app paired with it.

is_admin = lambda env: 'admin' in env['session']['roles'] 
is_user = lambda env: 'user' in env['session']['roles'] 
default = lambda env: True

rules = [(is_admin, admin_screen), (is_user, user_screen), (default,
access_denied)]

envdis = EnvironDispatcher(rules)

s = Selector()
s.add('/user-info/{username}[/]', GET=envdis)

Middleware Composer

Another WSGI middleware included in selector allows us compose middleware on the fly (compose as in function composition) in a similar way. MiddlewareComposer also is instantiated with a list of rules, only instead of WSGI apps you have WSGI middleware. When called, the instance applies all the middlewares whose predicates are true for environ in reverse order, and calls the resulting app.

lambda x: True; f = lambda x: False
rules = [(t, a), (f, b), (t, c), (f, d), (t, e)]
                
composed = MiddlewareComposer(app, rules)

s = Selector()
s.add('/endpoint[/]', GET=composed)

is equivalent to

a(c(e(app)))

Routing Args in Callable Signatures

There are some experimental, somewhat old decorators in Selector that facilitate putting your routing args into the signatures of your callables.

from selector import pliant, opliant  
  
@pliant  
def app(environ, start_response, arg1, arg2, foo='bar'):  
    ...  
  
class App(object):  
    @opliant  
    def __call__(self, environ, start_response, arg1, arg2, foo='bar'):  
        ...  

Exposing Callables

Selector now provides classes for naked object and HTTP method to object method based dispatch, for completeness.

from selector import expose, Naked, ByMethod  
  
class Nude(Naked):  
    # If this were True we would not need expose  
    _expose_all = False  
      
    @expose  
    list(self, environ, start_response):  
        ...  
  
class Methodical(ByMethod):  
    def GET(self, environ, start_response):  
        ...  
    def POST(self, environ, start_response):  
        ...

API Docs

Read Selector's API Docs on Read the Docs.

Tests

Selector has 100% unit test coverage, as well as some basic functional tests.

Here is output from a recent run.

luke$ fab test
[localhost] local: which python
Running unit tests with coverage...
[localhost] local: py.test -x --doctest-modules selector.py --cov selector
tests/
===============================================================================
test session starts
===============================================================================
platform darwin -- Python 2.6.1 -- pytest-2.1.3
collected 72 items 

selector.py .
tests/__init__.py .
tests/test_harness.py ...
tests/util.py .
tests/wsgiapps.py .
tests/functional/__init__.py .
tests/functional/conftest.py .
tests/functional/test_simple_routes.py .....
tests/unit/__init__.py .
tests/unit/mocks.py .
tests/unit/test_by_method.py ....
tests/unit/test_default_handlers.py ...
tests/unit/test_environ_dispatcher.py ..
tests/unit/test_middleware_composer.py ..
tests/unit/test_naked.py ............
tests/unit/test_pliant.py ...
tests/unit/test_selector_add.py ......
tests/unit/test_selector_call.py ..
tests/unit/test_selector_init.py ...
tests/unit/test_selector_mapping_format.py .......
tests/unit/test_selector_select.py .....
tests/unit/test_selector_slurp.py ...
tests/unit/test_simple_parser.py ....
----------------------------------------------------------------- coverage:
platform darwin, python 2.6.1-final-0
-----------------------------------------------------------------
Name       Stmts   Miss  Cover
------------------------------
selector     261      0   100%

============================================================================ 72
passed in 1.13 seconds
============================================================================
[localhost] local: which python
Running PEP8 checker
No PEP8 violations found! W00t!

Release Management Policy and Versioning

Selector is SemVer compliant.

Release management is codified in the fabfile.py in the release task.

Hack!

Fork it.

$ git clone http://github.com/lukearno/selector.git

Set yourself up in a virtualenv and list the fab tasks at your disposal. (Requires Virtualenv.)

$ . bootstrap

Run the tests.

(.virt/)$ fab test

Licenses

Use under MIT or GPL.

Copyright (c) 2006 Luke Arno, http://lukearno.com/

selector's Issues

not escaped dot in number parameter pattern

Hi. It seems like the dot in number pattern was accedentally left unescaped.

As is the pattern will not only match .9 or 00.12 but also 123!45, because unescaped dot matches any character.

More approrpiate pattern would be r'\d*\.?\d+'

API Docs Broken on RTD! - Get Whitelisted

Question 1 on the RTD FAQ: you need to get whitelisted to install you project in a virtualenv so autodoc will work.

http://read-the-docs.readthedocs.org/en/latest/faq.html#my-project-isn-t-building-with-autodoc

Routing design flaw

Hi,

I have been writing a JSON REST api, using selector and yaro, plus a middleware of my own (to jsonify the easy way).
The fact is that I made some decorators int Bottle/Flask way (@app.get(url)), and the add_rule called for the same url, different verbs behaves as it souldn't. Here's an example:

from selector import Selector

def get_test(environ, start_response):
    return None

def post_test(environ, start_response):
    return None

app = Selector()
app.add('/', GET=get_test, POST=post_test)
print app.mappings

app = Selector()
app.add('/', GET=get_test)
app.add('/', POST=post_test)
print app.mappings

This is the result of those prints (prettyprinted):

[
    (
        <_sre.SRE_Pattern object at 0xb709d440>,
        {
            'POST': <function post_test at 0xb7062a04>,
            'GET': <function get_test at 0xb7062aac>
        }
    )
]
[
    (
        <_sre.SRE_Pattern object at 0xb709d440>,
        {
            'GET': <function get_test at 0xb7062aac>
        }
    ),
    (
        <_sre.SRE_Pattern object at 0xb709d440>,
        {
            'POST': <function post_test at 0xb7062a04>
        }
    )
]

As you can see, the same url ("/") gets two entries in the second app, causing a 405 in one of the methods (obviously, gets a hit in the first rule, but finds no method for the verb; does not continue checking).

Please take a look, although is not a critical issue

Util Dir is a Code Smell

Util dir is a code smell. Move the CSV in under the tests and the python script into the fabfile.

Couple of Generated Artifacts not Ignored by Git

Directory create by fab html_coverageis not ignored by git.
Files and directories created by fab regenerate_api_docs.

Add htmlcov and docs/_build to .gitignore.

Docstring and Don't Output Virt Dir for _freshvirt()

Add docstring for _freshvirt() and remove that puts(virt).

Release Verification Test

Test installing to a fresh virt from pypi and importing.

Inline Regexs for Path Expressions and Types

Regexes for types and path expressions should usable inline.

Path expressions that start with ^ should be taken as raw regexes. IOW, this should work without specifying any custom parser.

s.add(r'^\/endpoint\/$', ...)

Regexes for types should be definable inline with perlish slashes like this:

s.add('/foo/{fooname:/ANYREGEX/}', ...

Either case would set construct=False WRT the proposed URL generation feature (so, no URL generation)

Add ReadTheDocs Post-Receive Hook

Just check the setting for RTD under Admin->Service Hooks.

Change Homepage to Github in setup.py

I updated PyPI manually last night, but I need to update setup.py too.

Generated Docs Out of Date

I guess these need to be rebuilt upon release, but I want it to be thoughtful and intentional. How do I detect out of date generated docs and complain when trying to release. ...I guess just generate them to a different temp folder and do some diffing?

For the moment, I will do it manually and open a ticket to add it to preflight checks.

Sign Tags and Builds with GPG Key

Preflight checks in fabfile should check that the gpg-agent is available and git config user.signingkey can be used to sign something. Git tag and src and egg distributions for pypi can then be signed.

missing VERSION file

It appears the latest release (v0.9.3, pulled from PyPI) is faulty. v0.8.11 is the last release to install without problems.

$ pip install selector
Downloading/unpacking selector
  Downloading selector-0.9.3.tar.gz
  Running setup.py egg_info for package selector
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "build/selector/setup.py", line 6, in <module>
        with open('VERSION', 'rb') as version_file:
    IOError: [Errno 2] No such file or directory: 'VERSION'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "build/selector/setup.py", line 6, in <module>

    with open('VERSION', 'rb') as version_file:

IOError: [Errno 2] No such file or directory: 'VERSION'

----------------------------------------
Command python setup.py egg_info failed with error code 1

Add README.md to docs/ Directory

This will avoid confusion when newcomers look in that directory on GitHub.

Add reviewth.is Post-Receive Hook

See http://reviewth.is/

Building API Docs on RTD Fails For Unit Tests

I am not sure that I can build these docs on RTD when I am autodoc-ing the tests unless I properly install them. I don't think I want to A) make the test suite it's own package with it's own setup.py or B) make the tests an installed part of Selector. If there is some way to install the tests selectively, or some other way to get RTD to build them, that would be cool.

I am going to back off of this, for now, and keep the issue open. I want to treat the tests as first class code and document them in this way, but, until I figure out how, I am going to be pragmatic and at least build the selector module's public API docs in the mean time.

...OTOH, maybe should the API docs just be limited to the Public API that which I want to consider Selector's public API, WRT SemVer. I may potentially just choose to close this and leave it that way...

Pushd, Popd Don't Work in Fab Tasks, Use lcd()

fab clean and fab html_coverage not working on Ubuntu. Fabric uses bash on OS X, I guess.

URL Generation

s = selector.Selector()
s.add("/foos/{fooid}", name='afoo')
assert s.url('afoo', fooid=11) == '/foo/'

Maybe parser can optionally return something like this instead of a regex string:

class EndPoint(object):
    def __init__(self, construct=True):
        self._construct = construct
    def match(self, path_info):
        # for simple parser, compile and use regex here
    def url(self, *a, **kw):
        if self._construct is True:
            # build url with string.Template
        else:
            # raise something

I think type tuples for the parser could have an optional third: ('digit', '\d+', '%d'). Defaulting to '%s', so ('word', '\w+') is the same as ('word', '\w+', '%s').

The parser can implement this by building up a template for string.Template as it goes, unless it hits a conditional portion ([] square brackets), in which case it produces an EndPoint(contruct=False)

Partials seem like an obvious bonus here:

s.add('/books/{book}/chapters/{chapter}', GET=show_chapter)
chapter_url = s.url_parial('chapter', book=2)
assert chapter_url(chapter=15) == "/books/2/chapters/15"

Also, how do we support this in the mapping file format?

story: /stories/{story}
    GET mymodule:show_story

might be the same as

s.add('/stories/{story}', name='story', GET=show_story)

... and if that is going to work, then I guess this wil

s.add('story: /stories/{story}', GET=show_story)

I don't think there is any existing reason to put whitespace in a path expression line in a mapping file, so I think len(pathexpr.strip().split()) should serve as a sufficient test for this case.