A scaffolding tool for projects using DataLoader, Flow and PostgreSQL.
Keeping database and codebase in sync is hard. Whenever changes are done to the database schema, these changes need to be reflected in the codebase's type declarations.
Most of the loaders are needed to perform simple PK look ups, e.g. UserByIdLoader. Writing this logic for every table is a mundane task.
PostLoader solves both of these problems by:
- Creating type declarations for all database tables.
- Creating loaders for the most common lookups.
If you are interested to learn more, I have written an article on the subject: I reduced GraphQL codebase size by 40% and increased type coverage to 90%+. Using code generation to create data loaders for all database resources..
- ORM is not going to give you strict types and code completion.
- ORM has runtime overhead for constructing the queries and formatting the results.
PostLoader is a CLI program (and a collection of utilities) used to generate code based on a PostgreSQL database schema.
The generated code consists of:
- Flow type declarations describing every table in the database.
- A factory function used to construct a collection of loaders.
A loader is created for every column in a unique index (unique indexes including multiple columns are not supported), e.g. UserByIdLoader.
A loader is created for every column that has name ending with _id.
A non-unique loader is used to return multiple rows per lookup, e.g. CitiesByCountryIdLoader. The underlying data in this example comes from a table named "city". PostLoader is using pluralize module to pluralize the table name.
A loader is created for every resource discoverable via a joining table.
- A joining table consists of at least 2 columns that have names ending
_id. - The table name is a concatenation of the column names (without
_idsuffix) (in alphabetical order, i.e.genre_movie, notmovie_genre).
Assume a many-to-many relationship of movies and genres:
CREATE TABLE movie (
id integer NOT NULL,
name text
);
CREATE TABLE venue (
id integer NOT NULL,
name text
);
CREATE TABLE genre_movie (
id integer NOT NULL,
genre_id integer NOT NULL,
movie_id integer NOT NULL
);
Provided the above schema, PostLoader will create two non-unique loaders:
MoviesByGenreIdLoaderGenresByMovieIdLoader
Type names are created from table names.
Table name is camel cased, the first letter is uppercased and suffixed with "RecordType", e.g. "movie_rating" becomes MovieRatingRecordType.
Property names of type declarations are derived from the respective table column names.
Column names are camel cased, e.g. "first_name" becomes firstName.
Loader names are created from table names and column names.
Table name is camel cased, the first letter is uppercased, suffixed with "By" constant, followed by the name of the property (camel cased, the first letter is uppercased) used to load the resource, followed by "Loader" constant, e.g. a record from "user" table with "id" column can be loaded using UserByIdLoader loader.
export POSTLOADER_DATABASE_CONNECTION_URI=postgres://postgres:[email protected]/test
export POSTLOADER_COLUMN_FILTER="return /* exclude tables that have a _view */ !columns.map(column => column.tableName).includes(tableName + '_view')"
export POSTLOADER_TABLE_NAME_MAPPER="return tableName.endsWith('_view') ? tableName.slice(0, -5) : tableName;"
export POSTLOADER_DATA_TYPE_MAP="{\"email\":\"text\"}"
postloader generate-loaders > ./PostLoader.js
This generates a file containing a factory function used to construct a DataLoader for every table in the database and Flow type declarations in the following format:
// @flow
import {
getByIds,
getByIdsUsingJoiningTable
} from 'postloader';
import DataLoader from 'dataloader';
import type {
DatabaseConnectionType
} from 'slonik';
export type UserRecordType = {|
+id: number,
+email: string,
+givenName: string | null,
+familyName: string | null,
+password: string,
+createdAt: string,
+updatedAt: string | null,
+pseudonym: string
|};
// [..]
export type LoadersType = {|
+UserByIdLoader: DataLoader<number, UserRecordType>,
+UsersByAffiliateIdLoader: DataLoader<number, $ReadOnlyArray<UserRecordType>>,
// [..]
|};
// [..]
export const createLoaders = (connection: DatabaseConnectionType) => {
const UserByIdLoader = new DataLoader((ids) => {
return getByIds(connection, 'user', ids, 'id', '"id", "email", "given_name" "givenName", "family_name" "familyName", "password", "created_at" "createdAt", "updated_at" "updatedAt", "pseudonym"', false);
});
const UsersByAffiliateIdLoader = new DataLoader((ids) => {
return getByIdsUsingJoiningTable(connection, 'affiliate_user', 'user', 'user', 'affiliate', 'r2."id", r2."email", r2."given_name" "givenName", r2."family_name" "familyName", r2."password", r2."created_at" "createdAt", r2."updated_at" "updatedAt", r2."pseudonym"', ids);
});
// [..]
return {
UserByIdLoader,
UsersByAffiliateIdLoader,
// [..]
};
};
Notice that the generated file depends on postloader package, i.e. you must install postloader as the main project dependency (as opposed to a development dependency).
- Dump the generated code to a file in your project tree, e.g.
/generated/PostLoader.js. - Create PostgreSQL connection resource using Slonik.
- Import
createLoadersfactory function from the generated file. - Create the loaders collections.
- Consume the loaders.
Example:
// @flow
import {
createPool
} from 'slonik';
import {
createLoaders
} from './generated/PostLoader';
import type {
UserRecordType
} from './generated/PostLoader';
const pool = createPool('postgres://');
const loaders = createLoaders(pool);
const user = await loaders.UserByIdLoader.load(1);
const updateUserPassword = (user: UserRecordType, newPassword: string) => {
// [..]
};You can optionally pass a second parameter to createLoaders – loader configuration map, e.g.
const loaders = createLoaders(connection, {
UserByIdLoader: {
cache: false
}
});Unfortunately, PostgreSQL does not describe materilized view columns as non-nullable even when you add a constraint that enforce this contract (see this Stack Overflow question).
For materialied views, you need to explicitly identify which collumns are non-nullable. This can be done by adding POSTLOAD_NOTNULL comment to the column, e.g.
COMMENT ON COLUMN user.id IS 'POSTLOAD_NOTNULL';
COMMENT ON COLUMN user.email IS 'POSTLOAD_NOTNULL';
COMMENT ON COLUMN user.password IS 'POSTLOAD_NOTNULL';
COMMENT ON COLUMN user.created_at IS 'POSTLOAD_NOTNULL';
COMMENT ON COLUMN user.pseudonym IS 'POSTLOAD_NOTNULL';
Alternatively, update the pg_attribute.attnotnull value of the target columns, e.g.
CREATE OR REPLACE FUNCTION set_attribute_not_null(view_name TEXT, column_names TEXT[])
RETURNS void AS
$$
BEGIN
UPDATE pg_catalog.pg_attribute
SET attnotnull = true
WHERE attrelid IN (
SELECT
pa1.attrelid
FROM pg_class pc1
INNER JOIN pg_namespace pn1 ON pn1.oid = pc1.relnamespace
INNER JOIN pg_attribute pa1 ON pa1.attrelid = pc1.oid AND pa1.attnum > 0 AND NOT pa1.attisdropped
WHERE
pn1.nspname = 'public' AND
pc1.relkind = 'm' AND
pc1.relname = view_name AND
pa1.attname = ANY(column_names)
);
END;
$$ language 'plpgsql';
set_attribute_not_null('person_view', ARRAY['id', 'imdb_id', 'tmdb_id', 'headshot_image_name', 'name']);
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent