code4sac / openbudgetsac.org

Code for Sacramento's Open Budget Project (landing page + D3.js files)

Open Budget: Sacramento

Contributing

If you're looking for a starter development task to get your feet wet with our codebase, any of our Issues tagged help wanted might be a good fit.

Some of the other Issues are larger and require some deeper design or architectural work; if one of those catches your eye, you'll probably want to talk with us for some more context and background. Either comment on the Issue or — even better — catch up with us at one of Code for Sacramento's weekly Hack Nights.

Developing Locally

Run with Docker

Run docker compose and view the page in your browser at localhost:8011. It is required to rerun the command after making changes to see the effects.

docker compose up --build

Quick Start Guide for Unix-based systems (Windows, Mac or Linux)

1. Sign into GitHub and fork this repo
2. Clone the fork onto your machine and navigate to the new folder
3. While still in the root directory of the repo, create a new folder called "build". This folder will be ignored by our version control system.
4. Navigate to the _src/ folder, which is where all development work takes place.
5. Install dependencies with npm install
6. Serve the website by entering npx @11ty/eleventy --serve --port=8011

Congratulations! Your local copy of Open Budget Sacramento's website should now be running at http://localhost:8011. That means you're ready to do the codez if you want to contribute to the codebase of Open Budget Sacramento. You will probably want to open a new terminal window, though, to regain access to the command line.

• Please note that after editing a SASS file you should run npm run build-css from the _src/ folder in order to incorporate your changes into the CSS

Eleventy

This site is built with Eleventy, a JavaScript-based static site generator that parses Markdown, Pug, and other template languages and runs on Node.js. That means you can reproduce our site locally with minimal setup!

You'll need these installed globally:

• Node is a prerequisite for NPM
• NPM or Yarn
• NVM is optional, but very handy for downloading, updating, and switching between versions of NPM

Install & Run Eleventy in _src/

Once you have the NPM package manager installed, you can install Eleventy and the other dependencies listed in package.json. Enter the following from the _src/ folder, where the Eleventy configuration file .eleventy.js lives.

npm install

This command usually runs without a glitch, but if you run into trouble, check your version of node. The latest version of node that we can confirm works with our set-up is v15.14.0.

To start Eleventy, simply enter the following. (You may choose any network port on your system that is available; 8011 is just a suggestion.)

npx @11ty/eleventy --serve --port=8011

Frontend Stack

This project is coded with, among other things:

• Bootstrap, a CSS framework
• D3, a data visualization library for JavaScript
• Pug, a JavaScript-friendly HTML templating language
• React, a rendering library for JavaScript
• Sass, a CSS preprocessor

Creating & Editing Pages

• Please note that it is your responsibility to keep your fork of the repo up-to-date with changes made by others working on the project. Doing this diligently should go a long way towards protecting you from scary git merge conflicts.
• All development activity occurs in _src/. The root folder is only for compiled output for deployment.
• Page content is inserted into the content block. If you are updating data, be sure you understand how it will be consumed.
• In many cases you will simply create or update a .pug file, which Eleventy will turn into HTML. If you are making another type of change, you may need to read Pug documentation (which is excellent, by the way!).
• If your page uses custom page-specific css, add it to a new .scss partial and import it into the main stylesheet. (Make sure to namespace it the same way the others are.)

Additional instructions for "flow" diagram pages

1. Flow pages are built off a template; copy one of the *-budget-flow.pug pages and update the content blocks as necessary.
2. Data files must be placed in the data/flow directory. Follow the naming convention seen there or your files won't load properly. You also will need to point your page at the appropriate files as seen in the get_datafiles content block.
3. the following columns are required in your datafile and their names should be normalized as seen here. Other columns should be removed to minimize the data download.
   • budget_year
   • department
   • fund_code
   • account_type (this should be the Expense/Revenue column, if there are duplicate names)
   • account_category
   • amount

Additional instructions for treemap diagram pages

1. Treemap pages are built off a template; copy one of the *-budget-tree.pug pages and update the content blocks as necessary.
2. Instructions for generating the necessary data files can be found here. Add them to the data/tree/ directory following the naming convention seen in the existing files.
3. Update the datafiles content block with the appropriate metadata and file path for the data files you generated.

Additional instructions for the Compare page

1. The Compare page is a React application. The source files are in _src/js/compare/ and are are bundled with Webpack.
2. When developing on the Compare page, run yarn to install all the necessary node dependencies and yarn run watch to watch the source files for changes and rebuild the asset bundles accordingly.

Publishing Changes

Make changes and review them on your local development site. If everything looks good, push your changes to your personal fork and merge the commit(s) into your main branch. Finally, issue a pull request and we'll take it from there!

Issuing a pull request

1. Create a pull request from your forked repo to the main branch of the upstream project and 2) just tell an admin of the upstream repo that your work is ready for review. Your changes will then be reviewed, tested, and (if everything looks good) pushed into the master branch.

Changes pushed to the main branch of the (original) repo will use GitHub Actions to trigger a continuous integration process that (among other things):

• runs WebPack;
• builds static files with Eleventy; and
• deploys the updated web files to GitHub Pages

openbudgetsac.org's Issues

Update README references to CodeForSacramento

There are references currently to OpenOakland in https://github.com/code4sac/openbudgetsac.org/blob/release-eleventy/README.md. Update the references to say CodeForSacramento and reference our pages for meetups etc.

[Node/React upgrade] Make copy changes to openbudgetsac

After we get the new framework on a branch, we need to change all of the copy titles, text, links, and logs from openbudgetoakland to openbudget sac. https://github.com/openoakland/openbudgetoakland

Update data to latest

Since we closed out #29, we now have access data that we didn't have access to before (years 2018-2022). We should update the application to use this data.

Update with new data for 2016-2017

Looks like this data is old. Is there a new budget available? Does the city make this information available publicly in some easily readable format?

Rewrite Compare page with newer version of React.js

The Compare page openbudgetsac.org/compare uses an old version of React.js, which uses deprecated coding standards. Upgrading to the newer version of React.js would require a complete rewrite.

Work has already been started by @PlacidSerene https://github.com/PlacidSerene/openbudget-rework

FY16 Expenses don't appear in the Treemap

This issue is caused by a misspelling in the config.json file for rendering the related file. The Aprroved.Expenses.FY16.json needs renaming and the config.json needs to have the misspelling corrected.

Update Sacramento budget data

The City of Sacramento Approved Budgets was updated July 19, 2023. It looks like there are budget fiscal years for 2023 and 2024.

This can be done by running the python scripts in the _treemap folder. The generated data is then placed in _src/data.

Treemap page wrong data source

The page sites City of Oakland instead of Sacramento

[Node/React upgrade] Ensure correct data columns exist for new framework

Ensure correct data columns exist for new framework in our current datasets

The following columns are required in your datafile and their names should be normalized as seen here. Other columns should be removed to minimize the data download.

budget_year
department
fund_code
account_type (this should be the Expense/Revenue column, if there are duplicate names)
account_category
amount

Civic.json file addition

Should we add a civic.json, like what was included in an old version of this repository?

About civic.json, from Code for DC:

It's a small file you put in the base of your project's public repository. Fill it with information explaining what your project's about, its development status, who helped build it, and where people can learn more.
Think of it like a structured "README" file that makes possible things like browseable project pages for your organization or city.

Civic JSON Creator: Civic.JSON (Extended) (dc.gov)

Docker does not watch local file changes

When making a change to the files, the changes are not reloaded in the browser.

Docker compose is running npx @11ty/eleventy --serve --port=8011, which normally watches for changes and reloads the page. However, it doesn't function the same when running in a Docker container.

Currently, to see the changes you have to stop and start again the Docker container with docker compose up --build.

[SPIKE] Rewrite OpenBudgetSac using modern framework and libraries

This project has outdated code and libraries from many years ago. It is not broken at the moment, but there could be some benefits to rewriting the project.

1. Use newer technologies that are more well-known and approachable for new developers learning modern frameworks and libraries
2. Rewriting would strip out any unnecessary or unused pages/logic that is hanging around and adding complexity
3. If we rewrite with the understanding that we need to update the data every year or two, we can optimize the process to make it easier in the future.

This might be good project to tackle for a developer with a few years of experience.

Generate Compare page data locally

To update the Compare page to current budget data, we want to use local files instead of a WordPress API. The data should be placed in the _src/data/compare folder. The files that are currently in that folder are not used, as they are for Open Budget Oakland.

The 2015-2018 data currently being consumed from the Open Budget Sacramento WordPress API can help show the format we need to follow.

1. Budget Totals: https://api.openbudgetsac.org/?rest_route=/obo/v1/fiscal-years-expenses
2. Expenses:
   a. https://api.openbudgetsac.org/?rest_route=/obo/v1/fiscal-years-expenses/depts/FY15
   b. https://api.openbudgetsac.org/?rest_route=/obo/v1/fiscal-years-expenses/account-cats/FY15
3. Revenue:
   a. https://api.openbudgetsac.org/?rest_route=/obo/v1/fiscal-years-revenue/depts/FY15
   b. https://api.openbudgetsac.org/?rest_route=/obo/v1/fiscal-years-revenue/account-cats/FY15

The files should be named FY13.json, FY14.json, to FY22.json and be placed in their corresponding folders. We can convert the data from CSV to JSON, similar to the Treemap process data Python script. It should be noted that the old 2015-2018 data is in a different format than the current 2013-2022 data from the Sacramento City Budget API. We should try to keep 2015-2018 the same, and only update 2013-2014 and 2019-2022 data.

Update the transform CSV to JSON script to python3

Under _treemap folder there is a python script that creates the JSON files from the CSV data. It currently uses python2. We could update to python3 and propose the changes back to open budget oakland.

https://github.com/openoakland/openbudgetoakland
https://openbudgetoakland.org/

[Node/React upgrade] Publish new site

Once the data and copy changes are in place, we can merge the changes into master and publish the new site. There are already scripts to publish the site to the gh-pages branch -- ensure it works properly.

[Node/React upgrade] Transform data and treemaps

Transform the current data sets to conform to the formats consumed in https://github.com/openoakland/openbudgetoakland

There seems to be three different types of data sets:

1. compare
2. flow
3. tree

If there's a lot of work here, break it out into separate issues

use same treemap implementation as Philadelphia open budget viz

City of Philadelphia has an open budget visualization based on this. I think the current rectangular visualization is confusing when there is a quantity represented by a rectangle whose height is less than another rectangle representing a smaller quantity (e.g., for 2015-16, more money was allocated to "Fire" than "Debt Service", but the rectangle for "Debt Service" has a larger y dimension).

Multiple references to Oakland

There are still references to Oakland in the codebase that should be changed or removed.

Already Solved

• Treemap contains reference to Oakland's Open Data Platform #51
• Files in the root directory f69c74e

I did a previous cleanup that did not change any application files under the _src folder. This is because there is a lot of cruft and I didn't want to spend time renaming things that we wouldn't use.

We can continue to rename references as we see them, and focus on a thourough rename after we have deleted unnecessary files.

Recreate README file

This repo was taken from Open Budget Oakland. They have a good readme to document how to use this project. Adapt the readme for this project from Oakland's: https://github.com/openoakland/openbudgetoakland

Request for new budget data from City of Sacramento

We need to request for the 2019, 2020, 2021 and possibly 2022 budgetary information/datasets from the City of Sacramento.

[Node/React upgrade] Setup API for compare page

The compare page is a react app that is fed from an API: https://github.com/openoakland/openbudgetoakland#additional-instructions-for-the-compare-page

There are instructions for setting this API up as a wordpress plugin: https://github.com/openoakland/openbudgetoakland#generating-the-api

I didn't see instructions on how to load the data into the database, but the sql queries in the php files look like the table is just structured the same as the csv. Indexes on the search and group by columns would be needed for best performance.

In order to get the compare page to work for open budget sac, we would need to setup this api to feed the react app. There are a couple options here:

1. Set the API up on a wordpress instance so we can re-use the code from openbudgetoakland
2. Rebuild the API using Node/Harp which are already used in this project. This will be more work, but I think it will be easier to maintain along with this project in the long run. The wordpress plugin code didn't seem all that complicated, so this approach seems reasonable if someone is willing to re-engineer it.

Potentially upgrade to latest OpenOakland OpenBudget framework

Open Oakland has updated their Open Budget site with some new technology (Node/React/Jade instead of Ruby/Jekyll/Markdown): https://github.com/openoakland/openbudgetoakland

Explore the work necessary to rebuild OpenBudgetSac with the latest Open Budget framework. My initial sense is this is a significant amount of work, but the new site does have some interesting new visualizations and tools.

Delete release-eleventy branch

This is a reminder to delete the release-eleventy. I want to wait a month to avoid confusion. We also need to update the GitHub action to build the site for GitHub Pages.

The default branch has been switched to "main".

move CSS and JS out of HTML templates

For example: https://github.com/code4sac/openbudgetsac.org/blob/3eb4a0212d78677f95918ce9881b3c09abd88919/_includes/departments-programs.html

Understand and drive impact of openbudgetsac.org

This site uses open data from https://data.cityofsacramento.org/ and presents it in an interactive and easier to digest format, which can be valuable.

However, currently not many people visit the site which limits the impact of this project. How can we dig into the value of the site, who might be interested in using the site, and how we might raise awareness of its existence the expand its impact?

Might it be possible to feature the site on the open data portal or another site that gets more visitors?

Update Build Your Own Queries page with new data for 2017-2018

Now that we have the other visualizations updated with the 2017-2018 data, we should tackle the "Build your own queries" feature now.

The dataset is here: http://data.cityofsacramento.org/datasets/approved-budget-fy-2015-fy-2018

Address in footer needs updating

The production site shows the old urban hive address