postgrest / postgrest

REST API for any Postgres database

PostgREST serves a fully RESTful API from any existing PostgreSQL database. It provides a cleaner, more standards-compliant, faster API than you are likely to write from scratch.

Sponsors

Big thanks to our sponsors! You can join them by supporting PostgREST on Patreon.

Usage

1. Download the binary (latest release) for your platform.

2. Invoke for help:

   postgrest --help

Documentation

Latest documentation is at postgrest.org. You can contribute to the docs in PostgREST/postgrest/docs.

Performance

TLDR; subsecond response times for up to 2000 requests/sec on Heroku free tier. If you're used to servers written in interpreted languages, prepare to be pleasantly surprised by PostgREST performance.

Three factors contribute to the speed. First the server is written in Haskell using the Warp HTTP server (aka a compiled language with lightweight threads). Next it delegates as much calculation as possible to the database including

• Serializing JSON responses directly in SQL
• Data validation
• Authorization
• Combined row counting and retrieval
• Data post in single command (returning *)

Finally it uses the database efficiently with the Hasql library by

• Keeping a pool of db connections
• Using the PostgreSQL binary protocol
• Being stateless to allow horizontal scaling

Security

PostgREST handles authentication (via JSON Web Tokens) and delegates authorization to the role information defined in the database. This ensures there is a single declarative source of truth for security. When dealing with the database the server assumes the identity of the currently authenticated user, and for the duration of the connection cannot do anything the user themselves couldn't. Other forms of authentication can be built on top of the JWT primitive. See the docs for more information.

Versioning

A robust long-lived API needs the freedom to exist in multiple versions. PostgREST does versioning through database schemas. This allows you to expose tables and views without making the app brittle. Underlying tables can be superseded and hidden behind public facing views.

Self-documentation

PostgREST uses the OpenAPI standard to generate up-to-date documentation for APIs. You can use a tool like Swagger-UI to render interactive documentation for demo requests against the live API server.

This project uses HTTP to communicate other metadata as well. For instance the number of rows returned by an endpoint is reported by - and limited with - range headers. More about that.

Data Integrity

Rather than relying on an Object Relational Mapper and custom imperative coding, this system requires you put declarative constraints directly into your database. Hence no application can corrupt your data (including your API server).

The PostgREST exposes HTTP interface with safeguards to prevent surprises, such as enforcing idempotent PUT requests.

See examples of PostgreSQL constraints and the API guide.

Supporting development

You can help PostgREST ongoing maintenance and development by:

• Making a regular donation through Patreon https://www.patreon.com/postgrest

• Alternatively, you can make a one-time donation via Paypal https://www.paypal.me/postgrest

Every donation will be spent on making PostgREST better for the whole community.

Thanks

The PostgREST organization is grateful to:

• The project sponsors and backers who support PostgREST's development.
• The project contributors who have improved PostgREST immensely with their code and good judgement. See more details in the changelog.

The cool logo came from Mikey Casalaina.

postgrest's Issues

Send proper json mime type

Limit responses with range headers

the ones that angular-paginate-anything sends

Replace literal parameters with placeholders

This will allow the db to cache and re-use query plans.

Insert where clause before offset and limit

Currently it selects a range from the table and then applies the where clause to that sub-range. It should apply the where clause to the whole table and then apply the range.

Return 404 for unknown table

Currently it gives a 400 with postgres error

execute: PGRES_FATAL_ERROR: ERROR:  relation "1.foo" does not exist
LINE 1: ...n(array_agg(row_to_json(t))) from ( select * from "1".foo  L...
                                                             ^

If we add an exception catching middleware we can detect the particular code for this error and do it there.

Use */0 rather than 0-0/0 for empty responses

The range 0-0 means that one results was returned, namely the zeroth row.

Port sql statements to quasiquotes

postgresql-simple provides a quoter

Provide clearer error message when bad json gets posted

It currently says

{"error":"Failed reading: satisfyElem"}

Show a better parse error to the user.

handle POST

POST /items should return 201 with location header pointing at the url of the created item (e.g. /items?primary_key=bar).

Query params are not permitted. To replace an existing resource use PUT. To update multiple, use PATCH. If params are provide the server should send a 400 response.

We'll handle 202 (accepted) delayed creation in a different issue.

Include standard headers in OPTIONS

Currently we include some nice schema info for a table in the OPTIONS payload, but we are lacking the real headers that HTTP expects.

Enable HSTS in production environments

https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security#HSTS_mechanism_overview

We shouldn't turn this on in development because our cert is self-signed and the browser will forbid site access.

Schema return value

The response currently looks like this:

{"pkey":["id"],"columns":{"lastname":{"precision":null,"updatable":true,"schema":"1","name":"lastname","type":"character varying","maxLen":255,"nullable":true,"position":2},"firstname":{"precision":null,"updatable":true,"schema":"1","name":"firstname","type":"character varying","maxLen":255,"nullable":true,"position":1}

It would be helpful for columns to return an array since I don't know the column names in advance.

Conditionally include created resource in POST response

Perhaps use some kind of accept header content negotiation in the request for this.

https://stackoverflow.com/questions/25475502/including-created-resource-in-http-post-response

Return http 200 for complete ranges

If the (lack of) range request causes no limiting to happen then return 200 rather than 206.

Default values listed in OPTIONS is coerced to string

For instance a column default value of false is reported as "false" which is incorrect.

Provide graphical entity-relation diagram

If the client requests / with the right accept headers, use graphviz to serve a graphical representation of the tables and key relations.

Combine populateSql call with query execution for speed

populateSql :: Connection -> QuotedSql -> IO String

is what was once called pgFormat. As you can see it returns a string of the sql ready to execute. We could make a populateAndExecSql function that executes it as an embedded query to save on db roundtrips.

Remove use of `head` in code

It is potentially unsafe

Honor database uri argument

Right now it is hard coded to use the database dbapi_test and schema base.

Return 404 for tables that the current role is not allowed to see

Right now the access problem raises a postgres exception which reveals that a table does indeed exist. We should be more sneaky.

Avoid parsing and unparsing json

Convert

selectAll :: T.Text -> Connection -> IO JSON.Value

to

selectAll :: T.Text -> Connection -> IO ByteStream

Serve cleaner json on errors

We can use wai middleware for this
http://hackage.haskell.org/package/wai-middleware-catch-0.3.6/docs/Network-Wai-Middleware-Catch.html

We need to detect specifically the postgres exceptions and include all the info such as the error code and description.

Output RAML for schema info

For the OPTIONS verb we should be outputting an industry standard to say the type of the response json as well as the parameters and versions accepted.

Fix orphan instance warnings

Provide column default values in OPTIONS /table

We need to identify columns that get their values from a postgres sequence. A GUI should not prompt for values of such columns.

Prevent duplicate query for pgFormat

Currently every query is executed more than once -- first to safely interpolate strings and then to run the generated query.

handle bulk POST

If the post contains an array of objects, write them to the db all at once.

Cache cabal sandbox in travis

We want quick travis builds to accelerate our pull-request workflow

Show `updatable` as false if user does not have permissions

In OPTIONS /table column view it gives false hope of being able to change the value for users who don't have permissions.

Use query params to filter

/:table?col=val&col2=val2

I was thinking that if { col1...coln } constitute a (possibly compound) primary key then the response would be a single object rather than an array.

Add hlint to continuous integration for this project

To enforce consistent style for would-be contributors, let's make hlint failures break the build. Could throw in a cabal build as well to make sure the code compiles.

Include foreign key information in schema dump

handle PUT

To /items?primary_key=bla

If the primary key does not yet exist this should create the record. All columns must be specified in the payload. If they are not, the server should return a 400 with a body explaining they attempted a non-idempotent action with a PUT.

Make a bespoke fixture sql

Right now test/fixtures/schema.sql is a postgres dump and includes commands that cause unnecessary code churn and merge conflicts as we work. We should rewrite it all with clean hand-written sql and continue to modify it that way.

Return 403 on insecure (http) requests

The app currently blocks the requests but returns a 200 which is inaccurate.

Add CORS headers to OPTIONS responses

Validate command line Postgres URI

Use Network.URI (isURI)

Set Content-Location header in response to GET requests with filter params

/items?c=eq:1&b=eq:2&a=eq:3 should specify a canonical version of itself with filters in alphabetical order.

Content-Location: /items?a=eq:3&b=eq:2&c=eq:1

This helps caches work.

Include valid values for enums in table options

The type of an enum is currently listed as USER-DEFINED and gives no clue to acceptable values.

Move table structure body out of the OPTIONS

Use rel="describedby" link header in the regular get response

https://www.iana.org/assignments/link-relations/link-relations.xml
https://www.mnot.net/blog/2012/10/29/NO_OPTIONS

Also include rel="describes" in the description url link headers.

Fix Location link in post to db lacking a primary key

It's possible to have such a table in postgres. I added a demo table called no_pk to the testing fixture which we can use to test this feature. Right now posts to this table work, but the Location header link replies with /no_pk? because the query params are generated from the (possibly compound) primary key. I am thinking the best solution is to include every single column as a constraint in the link. Even though it may return more than one row, it is the best we can do.

GET / with no db connection returns 400

It should be 500.

Silence the "data: end of file" message

It was introduced by 2f584be

Pass postgres errors forward to client

Determine a nice JSON format for it and include proper http status code

Enable all warnings in travis build and set them as errors

Use accept header versioning to select schema

The idea is that the client requests

application/vnd.you.com+format; version=n

and this makes n the schema used when accessing /:table

CORS middleware obliterates body for OPTIONS

@DrBoolean discovered that when his app requests OPTIONS /table with cors headers he doesn't get back the table structure JSON but instead gets a blank body. Looks like wai-cors is interfering. @adambaker is there a way to configure the middleware to pass the body through?

Allow running tests with configurable db and user

whether with command line options or a configuration file

Use json results in other queries

no need to define toRow and fromRow, just use json

Authentication and authorization

Authorization will use a designated schema, with a designated table to authenticate users. Eventually, we would like this table to be general enough to support most common authentication strategies. For this feature, we will support username/password authentication. This table will have:

1. identifier
2. encrypted password
3. salt
4. authorization role.

For authorization, we'll have each request authenticate, then before executing any action specified by the request, change roles to the role recovered from column 4 above, and then reset the role on the connection at the end of the request.