Under active development. Everything is subject to change. Interested in talking to us? Join our public chat room.
Federalist is a unified interface for publishing static government websites. It automates common tasks for integrating GitHub, Prose, and Amazon Web Services to provide a simple way for developers to launch new websites or more easily manage existing ones.
To run the server, you'll need Node.js and Ruby installed. The setup process will automatically install Jekyll and its dependencies based on the github-pages
gem.
You'll also need nvm.
To build sites using Hugo, install Hugo and make sure it's available in your path.
We have a few environment variables that the application uses, here is a list of the environment variables that the application will use if present:
-
FEDERALIST_AWS_BUILD_KEY
- the AWS key for container builds -
FEDERALIST_AWS_BUILD_SECRET
- the AWS secret for container builds -
FEDERALIST_BUILD_CALLBACK
- the endpoint for build status, defaults to 'http://localhost:1337/build/status/'. This URL must include the trailing slash -
FEDERALIST_BUILD_ENGINE
- the build engine to use, defaults to 'buildengine' -
FEDERALIST_BUILD_TOKEN
- random token used to protect the build status endpoint -
FEDERALIST_CACHE_CONTROL
- 'max-age=60' -
FEDERALIST_PUBLISH_DIR
- where to publish files if not S3, defaults to './assets' -
FEDERALIST_S3_BUCKET
- bucket ID to push files to on S3 -
FEDERALIST_SQS_QUEUE
- the name of an SQS queue. If defined, Federalist will send build messages to this queue and expect an external build service -
FEDERALIST_TEMP_DIR
- where files will be temporarily built, defaults to './.tmp' -
FEDERALIST_TEST_ORG
- A github org to authorize the test user against -
FEDERALIST_TEST_PASSWORD
required for tests - A github user password to run the tests with -
FEDERALIST_TEST_USER
required for tests - A github user to run the tests with -
GITHUB_CLIENT_CALLBACK_URL
- for dev you'll probably want to use http://localhost:1337/auth/github/callback -
GITHUB_CLIENT_ID
required - get this when you register your app with Github -
GITHUB_CLIENT_SECRET
required - you'll also get this when you register your app -
GITHUB_TOKEN
- A GitHub oauth token that will be used in place of authentication -
GITHUB_WEBHOOK_SECRET
- random token used to protect webhook messages, defaults to 'testingSecret' -
GITHUB_WEBHOOK_URL
- should be full url, defaults to 'http://localhost:1337/webhook/github' -
NEW_RELIC_APP_NAME
- application name to report to New Relic -
NEW_RELIC_LICENSE_KEY
- license key for New Relic -
NODE_ENV
- Node.js environment setting, defaults to 'development' -
PORT
- Server port, defaults to 1337 -
SAILS_LOG_LEVEL
- Log level, defaults to 'info'
You'll notice that we talk about a /config/local.js
file below, particularly for setting up the Github app information. For local development either approach is fine, but for production environments you'll want to set these env vars instead of committing local.js
to your git history.
- Download or Clone this repository from Github either by using the command line or repo's website on Github. On the right side of the repo's page, there is a button that states "Clone in Desktop".
- Run
nvm use
to ensure you are using the correct version of node - Run
npm install
from the root(the directory that houses the projects files on your computer) of the repository to load modules and install Jekyll dependencies
Together these commands will looks something like the following:
$ git clone [email protected]:18F/federalist.git
$ cd federalist
$ nvm use
$ npm install
-
Copy
config/local.sample.js
toconfig/local.js
.$ cp config/local.sample.js config/local.js
-
Set up an application on GitHub. You'll want to use
http://localhost:1337/auth
as the "Authorization callback url". Once you have created the application, you'll see a Client ID and Client Secret. Add these values toconfig/local.js
.
passport: {
github: {
options: {
clientID: '<<use the value from Github here>>',
clientSecret: '<<use the value from Github here>>',
callbackURL: 'http://localhost:1337/auth/github/callback'
}
}
}
In the end, your local.js
file should look something like this:
module.exports = {
passport: {
github: {
options: {
clientID: 'abcdef123456',
clientSecret: 'aabbccddeeff112233445566',
callbackURL: 'http://localhost:1337/auth/github/callback'
}
}
}
};
- Federalist grants access according to the organizations a user is a part of. Find your organization's ID by visiting
https://api.github.com/orgs/<your-org-name>
and copying theid
into the whitelist oforganizations
inconfig/local.js
, for example:
organizations: [
6233994, // 18f
14109682, // federalist-users
14080592, // us-federal-sbst
99999999 // your org added here
]
- Run the server with
npm start
(You can usenpm run watch:server
for the server to restart when you save file changes andnpm run watch:client
to rebuild front-end files on each save) at the directory of the project on your local computer.
There are really two applications in one repo here. Right now we're OK with that because we're moving quick to get done with the prototypal phase of the project.
If you are working on the front-end of the application, the things you need to know are:
- It is a Backbone based application
- It is built with
browserify
and useswatchify
to build on changes - It lives in
/assets/app
You can use npm run watch
to get the project built and running. This will set up watchify
to run when front end files change and will set up the server to reload on any file change (front end included)
By default, the application should use local disk storage in place of a database. This is easier to get started and isn't a problem for local development. In production, the app uses Postgres as the data store. To use Postgres in your local dev environment:
- First, you'll need to install Postgres.
- Next, you'll have to create the
federalist
database for the application.$ createdb federalist
should do the trick - Add postgres to your
/config/local.js
file
connections: {
postgres: {
adapter: 'sails-postgresql',
database: 'federalist'
}
},
models: {
connection: 'postgres'
}
To run the integration tests you'll need:
- Selenium (and Java).
- chromedriver
We're going to use selenium-standalone
to make installing all of those tools
just a bit easier to manage and install and we have some npm
scripts to help
out.
$ npm run test:selenium:install
$ npm run test:selenium:start
Make sure to run both of these commands, and install Java if it asks you to do so. It is required for selenium. After the second command is run successfully you should see something like
09:47:12.342 INFO - Selenium Server is up and running
Selenium started
The integration tests rely on a test Github user to ensure that data is appropriately displayed in the application. You can use any Github user, but 18F uses @FederalistTestingUser.
First you need to give your app permission to access the user's data, but that is not part of the integration test at this time. Thus, you should do this just the first time you are setting up a new Github application.
Ensure that you have a Federalist application running (locally is fine). Open an Incognito browser window and login to Github.com with your test user. If you are on the 18F team, please ask in the Slack room for the test user credentials.
Once you have logged into Github, open your application (http://127.0.0.1:1337 if you're running locally) and login. You should grant the application permission to access the test user's data in the OAuth authorization view.
Now that your application can access Github on behalf of the test user, set the environment variables for the integration tests.
$ export FEDERALIST_TEST_USER=<test user>
$ export FEDERALIST_TEST_PASSWORD=<test user password>
And now you can run the tests:
$ npm run test:integration
Sauce Labs can be a bit flakey sometimes and failures on Travis might not be reproducible locally. If you're just looking for details of the last Travis run, ping the account holder for access. Otherwise, 18F personnel can request a Sauce Labs account in Slack.
Once you have an account, you'll need your access key. It's available in the
Sauce Labs account settings and looks like a GUID e.g.
abcdef78-1234-1234-1234-abcdef789012
.
$ export SAUCE_USERNAME=<username> SAUCE_ACCESS_KEY=<sauce access key>
Use Sauce Connect to open a tunnel between your localhost and Sauce Labs (because your dev server is running on localhost).
$ brew install sauce-connect
$ sc -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY &
Then run your tests.
$ SAUCE_USERNAME=$SAUCE_USERNAME SAUCE_ACCESS_KEY=$SAUCE_ACCESS_KEY npm run test:integration
You shouldn't see Chrome launch locally because the tests are launched in on a Sauce Labs VM. Instead, you can watch the tests run live through the Sauce Labs UI.
Note: the environment variable names are important here because these are what webdriverio will be looking for.
This application is primarily a JSON API server based on the Sails.js framework. It handles authentication, managing users, sites, and builds, and receives webhook requests from GitHub.
It automatically applies a webhook to the repository new sites and creates a new build on validated webhook requests.
The front end of the application is a Backbone based application, that uses browserify in the build process. It is a very lightweight consumer of the Sails API.
The proof of concept application will have a web-based front-end to interface with the API and allow users to add new sites, configure them, and open them in Prose for editing.
It will also route requests to the published static websites, including preview sites for each branch.
After the initial proof of concept, development will focus on scalability by moving to the AWS platform, using S3 for publishing, SQS for managing the publishing queue, and running publishing tasks on separate servers.
Additional development will focus on improved collaboration features, such as built-in support for forking and merging branches for drafts of changes, and automatic configuration of Prose settings.
Federalist is new open source publishing system based on proven open source components and techniques. Once the text has been written, images uploaded, and a page is published, the outward-facing site will act like a simple web site -- fast, reliable, and easily scalable. Administrative tools, which require authentication and additional interactive components, can be responsive with far fewer users.
Regardless of the system generating the content, all websites benefit from the shared editor and static hosting, which alleviates the most expensive requirements of traditional CMS-based websites and enables shared hosting for modern web applications.
From a technical perspective, a modern web publishing platform should follow the “small pieces loosely joined” API-driven approach. Three distinct functions operate together under a unified user interface:
-
Look & feel of the website Templates for common use-cases like a departmental website, a single page report site, API / developer documentation, project dashboard, and internal collaboration hub can be developed and shared as open source repositories on GitHub. Agencies wishing to use a template simply create a cloned copy of the template, add their own content, and modify it to suit their needs. Social, analytics, and accessibility components will all be baked in, so all templates are in compliance with the guidelines put forth by SocialGov and Section 508.
-
Content editing Content editing should be a separate application rather than built into the web server. This allows the same editing interface to be used across projects. The editing interface only needs to scale to match the load from editors, not visitors.
-
Publishing infrastructure Our solution is to provide scalable, fast, and affordable static hosting for all websites. Using a website generator like Jekyll allows for these sites to be built dynamically and served statically.
- 18F Blog Post on Federalist's platform launch
- Building CMS-Free Websites
- Background on relaunch of Healthcare.gov’s front-end
- HealthCare.gov uses lightweight open source tools
- A Few Notes on NotAlone.gov
This project is in the worldwide public domain. As stated in CONTRIBUTING:
This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.
All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.