pantsbuild / pantsbuild.org

The Pants website

pantsbuild.org

All of the material and stitching for pantsbuild.org.

Layout

This repository contains three categories of content. Most of it should be edited in this repository, except for the auto-generated documentation for each version of Pants.

Evergreen pages

src/ contains content and configuration that isn't associated with a particular version of Pants. In particular, in src/pages/ you'll find the definitions of the front page and community information among others. src/ also contains helpers like MDX/React components, CSS and JS.

These should be edited directly in this repository.

Blog

blog/ contains the Pants blog. To write a post, follow the structure of other blog posts.

These should be edited directly in this repository.

Per-version documentation

docs/ and versioned_docs/ contains the documentation for each 2.x release series of pants. After each release of Pants, the documentation is synced to this repository automatically from https://github.com/pantsbuild/pants: the sync_docs.yml workflow manages this, and is triggered by the main repository release process.

This should not be edited directly in this repository. Instead, edit in https://github.com/pantsbuild/pants. To make changes to documentation for stable release series, edit there and cherry-pick to the appropriate milestone (still in https://github.com/pantsbuild/pants), so that it's included in the next release in that series.

Development

The docs site is a JS project, so you'll want nvm and yarn installed.

Afterwards

nvm use
yarn install

Dev server

The following command will start the dev server:

npm start

By default, only the "next" docs (e.g. the docs for the version that maps to main) will get built.

To include the blog, use:

PANTSBUILD_ORG_INCLUDE_BLOG=1 npm start

To include any version(s) in addition to the "next" version:

PANTSBUILD_ORG_INCLUDE_VERSIONS=$version1,$version2 npm start

Building the real deal

To build the site, run:

NODE_OPTIONS="--max-old-space-size=12288" npm run build

(Note: Node needs more than the default amount of RAM because this site is beefy)

Tech

The site is fully managed via https://docusaurus.io/ and is published via GitHub Actions to GitHub Pages.

pantsbuild.org's Issues

Validating doc syncing (likely) works in PR CI

We've had a few changes recently that have broken the doc syncing CI, both of which are fixed in #56. It'd be good to have some sort of validation on PRs of the syncing process, to increase our assurance in it.

Ensure 404 page is visible for all wrong URLs

From #4:

A "404" shows the menu structure but no actual content: https://docs.pantsbuild.org/xyz/abc. It'd be good to have some sort of real content describing the problem.

Ensure edit URL is correct

(From the Pants meetup)

Specifically, we should ensure the edit link (when it exists) is relevant to the repo it links to, and that there are no reference links

CI isn't validating the files are prettier-ified

I just noticed format-all formatting files on main 😢

Scraped prose docs don't match structure of pants repos, thus sync PRs have spurious changes

It seems the content scraped from readme doesn't match the structure of what's in the Pants repo, which means the doc syncing PRs don't fit into that smoothly: e.g. #32 has a massive diff stat, despite being a no-op update, theoretically (readme should be showing version 2.18.1 already).

I imagine these differences may result in mis-rendered docs, if they were merged.

Notable differences in prose docs:

#	PR behaviour	main behaviour	PR link	main link
1	content is contained within docs/markdown/ directory	just docs/	PR	main
2	directory names are Sentence Case	snake-case	PR	main
3	file names use the slug	use title	PR	main
4	no _category_.json files	yes files	PR	main

Potentially others: would be easiest to identify once we solve more of those to reduce the number of diffs to filter through.

(See #37 for differences in the reference docs)

`[GLOBAL].local_execution_root_dir` default is wrong in 2.12 and earlier

          I think we can let this one go, it looks like a "problem" in existing docs before 2.13:

• 2.12: https://www.pantsbuild.org/v2.12/docs/reference-global#local_execution_root_dir
  

• 2.13: https://www.pantsbuild.org/v2.13/docs/reference-global#local_execution_root_dir
  

And indeed, the docs match, e.g. 2.20:

pantsbuild.org/docs/reference/global-options.mdx

Lines 695 to 696 in 09d492d

	env_repr='PANTS_LOCAL_EXECUTION_ROOT_DIR'
	default_repr={`<tmp_dir>`}

Originally posted by @huonw in #50 (comment)

"404" on https://www.pantsbuild.org/sponsorship

Engineers from a variety of organizations around the world [collaborate](/404)

Doc syncing uses PAT without permission to open PRs

It seems the PAT we attempt to use for creating a PR (#83) doesn't have permission to open PRs here: https://github.com/pantsbuild/pantsbuild.org/actions/runs/7521799879/job/20473054842

pull request create failed: GraphQL: Resource not accessible by personal access token (createPullRequest)

Instructions for running locally appear to be out of date

From #4:

The README instructions didn't work for me, e.g. there's website/ directory for cd website, running npm start can't find docusaurus, I tried installing deps with npm ci but that failed too... probably because this is a yarn project? yarn start seemed to work

Clicking "Docs" title bar link gets trapped in Reference

The "Docs" link in the title now appears to go to the reference. Specifically, the page https://docs.pantsbuild.org/2.19.x/docs/introduction/welcome-to-pants now appears to be "owned" by the reference sidebar, rather than that field just being a link. For example, the sidebar shows the reference items, and the title bar shows Reference as selected:

This means it's currently ~impossible to find the rest of the prose docs (although a direct link still works: https://docs.pantsbuild.org/2.18.x/docs/introduction/how-does-pants-work).

Likely a regression from #23

Add search functionality

Built-in search is great, let's get it.

Per #15 (comment), Josh has applied for Algolia doc search.

Misformed links from Google search to Pants docs site ampersand expansion

Describe the bug
For some links when opening from Google search results, ampersands are expanded incorrectly (& -> &amps;) so the resulting opened page 404s.

e.g. on an incognito window in Chrome (link: https://www.pantsbuild.org/2.18/docs/using-pants/remote-caching-&-execution)

Relevant Slack thread: https://pantsbuild.slack.com/archives/C046T6T9U/p1705335830401029

CI doesn't work

https://github.com/pantsbuild/pantsbuild.org/actions/runs/7364130452/job/20044246488?pr=19

Run npm run prettier .
npm ERR! Missing script: "prettier"
npm ERR! 
npm ERR! To see a list of scripts, run:
npm ERR!   npm run

npm ERR! A complete log of this run can be found in: /home/runner/.npm/_logs/2023-12-30T12_41_26_640Z-debug-0.log

Support redirecting for URL fragments (page subheadings), especially `code...code` in target fields

From #4:

It looks like the heading fragments differ for some items, e.g. https://www.pantsbuild.org/v2.19/docs/reference-deploy_jar#coderesolvecode links to the resolve field specifically, while the redirect https://docs.pantsbuild.org/2.19.x/reference/targets/deploy_jar#coderesolvecode does not. Can we run client side JS easily (as part of the redirect)? If so, I guess we could replace codeXYZcode with XYZ, and maybe that handles most cases: the only ones I could find are the field headings for a target.

Code "tabs" aren't working

See https://docs.pantsbuild.org/2.8/docs/python/goals/check

Do temporary/example deploys for PRs

It'd be great if PRs to this repo deployed the static site somewhere it could be viewed, so that a reviewer can validate.

See #4 (comment) for initial discussions.

(It'd also be interesting to have temporary deploys on the Pants repo, e.g. make a change to the in-repo docs there and be able to view it in some form, but that's likely best solved in that repo.)

Pre-release testing questions

I've done some testing in advance. It looks really nice. I suspect there's a bunch of known issues, that I'm observing here, but I haven't seen a todo list/plan so I'm just going to write them all down. This also ends up as a mega issue and we can break up a bit after initial triage.

Random grab bag of things I like a lot (not exhaustive 😄 ):

• looks good, loads fast! 🏎️
• switching versions with the version selector preserves the page (if possible), rather than switching to the intro page 🎉
• table of contents and deep linking that works on mobile ☎️
• it runs locally 💻
• automation for updates! 🤖
• git for blog and general website content!

Original discussion, now real issues

Here's some comments (numbers for reference, not priority/importance):

1. Redirects of unversioned URLs don't seem to work; is that expected?
   • versioned: original: https://www.pantsbuild.org/v2.18/docs/python, new: https://docs.pantsbuild.org/v2.18/docs/python, redirect: https://docs.pantsbuild.org/2.18.x/docs/python/python-overview (shows content ✅ )
   • unversioned: original: https://www.pantsbuild.org/v2.18/docs/python, new: https://docs.pantsbuild.org/docs/python, not redirect and shows no content ❌
2. It looks like the heading fragments differ for some items, e.g. https://www.pantsbuild.org/v2.19/docs/reference-deploy_jar#coderesolvecode links to the resolve field specifically, while the redirect https://docs.pantsbuild.org/2.19.x/reference/targets/deploy_jar#coderesolvecode does not. Can we run client side JS easily (as part of the redirect)? If so, I guess we could replace codeXYZcode with XYZ, and maybe that handles most cases?
3. Somewhat related to the above, a "404" shows the menu structure but no actual content: https://docs.pantsbuild.org/xyz/abc. Is that expected?
4. The blog only seems to have 5 posts? http://docs.pantsbuild.org/blog (No, it's just the recent posts in the sidebar)
5. It seems a bit confusing for users to split the long-form text ("Docs") and detailed API descriptions ("Reference") into separate categories as they are: if I'm looking for how to learn something, I'd be hoping that everything I need is easy to find. Alternatives:
   • Include the reference sections in the sidebar for the docs view (maybe with a divider, if that's possible), but still have a direct link to it in the headings. That is, these things could be after "Tutorials - Coming soon"
     
   • If that doesn't work great, just have a link to "Reference" at the end of the Docs sidebar too, and a link to "Docs" in the "Reference" sidebar. (I.e. repeating links from the header, but in the place users might be looking)
6. The current 2.19.x reference docs show a TOML snippet, e.g. https://www.pantsbuild.org/v2.19/docs/reference-add-trailing-comma, but the new ones don't show this https://docs.pantsbuild.org/2.19.x/reference/subsystems/add-trailing-comma . Is that expected?
7. 1.30 is not marked as deprecated on the current docs, but doesn't appear to be preserved and redirects don't work. Are we just giving up on 1.30 entirely? e.g. https://www.pantsbuild.org/v1.30/docs -> https://docs.pantsbuild.org/v1.30/docs
8. Items in the sidebar seem be sorted alphabetically, but I think there's more meaningful order than that, e.g. for "Getting started", compare old (first) vs. new: the order in which people go through these matters! (There's other examples too)
   
9. Tweaks to the docs syncing automation:
   • It'd be good to assign reviewer(s), or else the doc update PRs would be easy to get lost. Suggestion: have the workflow dispatch take user name(s), and then if we have the release CI from the pants repo trigger this one, it can insert the release manager.
   • Can we include a link to the releases tag page in the PR, so that someone looking at it can more easily confirm what's included etc.?

Small things (just noting to have them written down, I could definitely fix them myself too, but I'm in "review" mode, not "code" mode):

1. Maybe we could just remove the "search coming soon" text completely: it gives me geocities vibes, where everything was always 🚧 under construction 🚧/coming soon and never actually did 😄 Once we have search, we can then just restore it.
2. "Whos Hiring?" in the footer should be "Who's Hiring?"
3. There's two levels of expanders for "Getting started": https://docs.pantsbuild.org/2.18.x/docs/getting-started/getting-started
   
4. The reference listing page for goals etc. seems to have large cards for each item, all with content ---. Is it possible to switch to either switch to a (denser) plain list, and/or put real content in those cards? https://docs.pantsbuild.org/2.18.x/reference/goals
   
5. (Why Node 18.0.0? AIUI, Node 20 is the most recent LTS?)
6. The README instructions didn't work for me, e.g. there's website/ directory for cd website, running npm start can't find docusaurus, I tried installing deps with npm ci but that failed too... probably because this is a yarn project? yarn start seemed to work
7. What would it take to have pants orchestrate this repo? 🤔
8. What's the difference between the docs/ and versioned_docs/ subdirectories?13.

Questions that I don't know if they should be issues:

1. Why Node 18.0.0? AIUI, Node 20 is the most recent LTS and I imagine there's minor/patch releases of Node 18 that might be relevant?
2. What would it take to have pants orchestrate this repo? 🤔
3. What's the difference between the docs/ and versioned_docs/ subdirectories?
4. Do you have ideas about how we could get doc previews for pants changes both locally and on a PR?

Docs sidebar items are sorted alphabetically inappropriately

From #4:

Items in the sidebar seem be sorted alphabetically, but I think there's more meaningful order than that, e.g. for "Getting started", compare old (first) vs. new: the order in which people go through these matters! (There's other examples too)

Don't show "Search coming soon" while it's pending

From #4:

Maybe we could just remove the "search coming soon" text completely: it gives me geocities vibes, where everything was always 🚧 under construction 🚧/coming soon and never actually did 😄 Once we have search, we can then just restore it.

Consider showing both text docs "Docs" and API docs "Reference" in single sidebar, same as old site

From #4:

It seems a bit confusing for users to split the long-form text ("Docs") and detailed API descriptions ("Reference") into separate categories as they are: if I'm looking for how to learn something, I'd be hoping that everything I need is easy to find. Alternatives:

• Include the reference sections in the sidebar for the docs view (maybe with a divider, if that's possible), but still have a direct link to it in the headings. That is, these things could be after "Tutorials - Coming soon"
  
• If that doesn't work great, just have a link to "Reference" at the end of the Docs sidebar too, and a link to "Docs" in the "Reference" sidebar. (I.e. repeating links from the header, but in the place users might be looking)

Search always coming up blank

Try any query on https://docs.pantsbuild.org/ and you'll get no results 😢

https://github.com/pantsbuild/pantsbuild.org/pull/38 left dead links

That PR moved files, but didn't update their relative links in those files

"tabbed" codeblocks aren't tabbed after version 2.11

Looks like the readme markdown changed from [block:code] blocks to triple-backtick blocks one-after-another.

Day-of Tasks (e.g. "No turning back" tasks)

Thought-dumping the "No Turning Back" tasks here:

• subdomain

  • Have @benjyw update the DNS
  • Update the DNS config on GitHub
  • Update URL in docusaurus.js

• Updating https://github.com/pantsbuild/pants:

  • (For each relevant branch)
  • Sync the content to readme.com
  • Re-generate the entire docs site for that version
  • PR it here
  • PR it on the repo under the docs/ path

• Search

  • In Algolia crawler, edit the crawler settings to crawl the new URL
  • See if we can wipe the index
  • Trigger a new crawl

• update maintainers docs with any relevant new steps (e.g. when cutting a new version)

Make `local_environment` (etc.) links to the docs for that environment?

          Is it possible/easy to make `local_environment` (etc.) links to the docs for that environment?

Originally posted by @huonw in #62 (comment)

`src/pages/community/members.mdx` links to doc `/docs/contributions`

This isn't great because we'll be redirecting to 2.18.x docs. Instead we should make this page dynamic enough to render the latest version.

Proposal: skip the .x in URLs/dropdowns

The 2.18.x version signifier seems like it could easily be 2.18 with identical meaning for users, but results in slightly neater URLs and simpler drop-downs:

(Labelled for Switch-over because this'll change URLs.)

Generate social cards for pages?

With the image head metadata, we could generate a little .png for each page as the social card.

For at least the reference docs, this would be easy (otherwise we could generate at build time)

Inappropriate h1 unlinkable & missing from table of contents on "targets" page

https://www.pantsbuild.org/2.19/docs/using-pants/key-concepts/targets-and-build-files has various headings that aren't able to be linked and don't appear in the table of contents. It appears to be the h1 headings.

This may due to violating the "Avoid using multiple <h1> elements on one page" guideline.

Potentially this is just inherited from our old docs, and docusaurus handles it different to readme, or maybe it's a bug in the generator?

Ensure people aren't editing prose docs in this repo

As @benjyw pointed out in the Pants meetup was that it's not completely obvious people shouldn't edit certain docs in this repo (/docs docs)

We should add a PR CI check for this (or similar automation)

v1 docs have disappeared

From #4:

1.30 is not marked as deprecated on the current docs, but doesn't appear to be preserved and redirects don't work. Are we just giving up on 1.30 entirely? e.g. https://www.pantsbuild.org/v1.30/docs -> https://docs.pantsbuild.org/v1.30/docs

Assign a reviewer to automatic doc syncing PRs, to reduce chance of falling through cracks

From #4:

It'd be good to assign reviewer(s), or else the doc update PRs would be easy to get lost. Suggestion: default to the person who triggered the workflow... but also have the workflow dispatch take user name(s), and then if we have the release CI from the pants repo trigger this one, it can insert the release manager (or else it might be @WorkerPants).

Our own docs still have `#code<>code` fragment links

This is a result of not stripping them during migration.

Let's do a single wholesale replacement

Fix broken links, and make them an error

Currently running a build shows a lot of warnings about broken links, that can't be resolved. It'd be nifty to fix them, and make them an error:

pantsbuild.org/docusaurus.config.js

Lines 43 to 45 in 38e1bda

	// @TODO: This should throw on prod
	onBrokenLinks: isDev ? "warn" : "warn",
	onBrokenMarkdownLinks: isDev ? "warn" : "warn",

e.g.

$ NODE_OPTIONS="--max-old-space-size=6144" npm run build
...
[WARNING] Docs markdown link couldn't be resolved: (../../using-pants/key-concepts-to-know-about-pants/targets-and-build-files.mdx) in "/Users/huon/projects/pantsbuild/pantsbuild.org/versioned_docs/version-2.8/docs/getting-started/create-initial-build-files.mdx" for version 2.8
[WARNING] Docs markdown link couldn't be resolved: (../../using-pants/key-concepts-to-know-about-pants/targets-and-build-files.mdx) in "/Users/huon/projects/pantsbuild/pantsbuild.org/versioned_docs/version-2.8/docs/getting-started/create-initial-build-files.mdx" for version 2.8
[WARNING] Docs markdown link couldn't be resolved: (../../using-pants/resources-and-archives.mdx) in "/Users/huon/projects/pantsbuild/pantsbuild.org/versioned_docs/version-2.8/docs/getting-started/create-initial-build-files.mdx" for version 2.8
...

Before we make it an error, we'll likely also want to have validation on the Pants side, or else we'll only find out about new broken links when doing a doc sync here for a new release, and that would block merging that new release's docs.

Codeblocks in admonitions got dedented

See https://github.com/pantsbuild/pantsbuild.org/pull/95/files#r1450665508

https://docs.pantsbuild.org/2.18/docs/using-pants/using-pants-in-ci

vs

https://www.pantsbuild.org/docs/using-pants-in-ci

Search doesn't find any results

Currently, using the search gives no results ever. For instance, searching for "pants":

This is potentially expected (e.g. waiting for Algolia to index), but just filing to have a record/place to discuss if it lasts longer than expected.

`help-all` contains `scie-pants-linux-x86_64` snippets

          This change looks like a good one. I think I'll make a new issue for this and a new PR.

Originally posted by @thejcannon in #71 (comment)

Show pants.toml snippet for global/goal/subsystem options, like current site

From #4:

The current 2.19.x reference docs show a TOML snippet, e.g. https://www.pantsbuild.org/v2.19/docs/reference-add-trailing-comma, but the new ones don't show this https://docs.pantsbuild.org/2.19.x/reference/subsystems/add-trailing-comma . Is that expected?

Auto-synced reference docs don't match checked-in code, thus sync PRs have spurious changes

It seems the current content for the reference docs (scraped from readme?) doesn't match those generated by the auto-sync PR, which means the doc syncing PRs don't fit into that smoothly: e.g. #32 has a massive diff stat, despite being a no-op update, theoretically (readme should be showing version 2.18.1 already).

I imagine these differences may result in mis-rendered docs, if they were merged.

Notable differences in reference/generated docs:

#	PR behaviour	main behaviour	PR link	main link
1	only a few subsystems (e.g. no add-trailing-comma)	lots of subsystems	PR	main
2	spacing differences	-	PR diff of anonymous-telemetry.mdx	(see diff)
3	escaping etc. differences	-	PR diff of target.mdx	(see diff)
4	no related subsystems in goals	-	PR diff of check.mdx	(see diff)
5	default_repr differences	-	PR diff of global-options.mdx	(see diff)

Potentially others: would be easiest to identify once we solve more of those to reduce the number of diffs to filter through.

(See #34 for differences in the prose docs)

Include link to relevant release in doc sync PR

From #4:

Can we include a link to the releases tag page in the PR, so that someone looking at it can more easily confirm what's included etc.?

'environments' docs have `:` in file/URL

The https://github.com/pantsbuild/pantsbuild.org/blob/main/docs/docs/using-pants/environments%3A-cross-platform-or-remote-builds.mdx file has a : in it, which potentially makes it impossible to check this repo out on windows? It unfortunately also appears in the URL which makes rewriting it hard to do, without breaking any existing links.

"Getting started" docs show two levels of expanders

From #4:

There's two levels of expanders for "Getting started": https://docs.pantsbuild.org/2.18.x/docs/getting-started/getting-started

"Who's Hiring" in footer should be "Who's Hiring" (missing apostrophe)

From #4:

Proposal: Check-in help-all output for each version, for easier doc regeneration

As I understand it, all the files in the docs/reference and versioned_docs/*/reference directories are code-generated from the JSON output of pants help-all for the appropriate version. Currently, it seems that JSON output is transient, so, for instance, adjusting the templates and regenerating all the docs seems like it'd require a script that re-runs the pants help-all commands and then the generate.js script, for each version.

What do you think about committing the help JSON output for each version (since it's the source of truth)? Assuming my understanding is correct, this would unlock a few things:

• easier regeneration of docs
• ability to validate that someone isn't editing the generated markdown files directly in the repo in CI (i.e. catch mistakes where someone tries to fix a typo here)
• potentially, especially with Pants orchestration, no checking-in of the generated markdown files at all (either that, or easier management pantsbuild/pants#18235)

No redirects for unversioned URLs

From #4:

Redirects of unversioned URLs don't seem to work; is that expected?

• versioned: original: https://www.pantsbuild.org/v2.18/docs/python, new: https://docs.pantsbuild.org/v2.18/docs/python, redirect: https://docs.pantsbuild.org/2.18.x/docs/python/python-overview (shows content ✅ )
• unversioned: original: https://www.pantsbuild.org/v2.18/docs/python, new: https://docs.pantsbuild.org/docs/python, not redirect and shows no content ❌

`sync_docs.yml` missing most backends

          OK 1 seems to be a misunderstanding between @kaos and I about how `help-all` works with no backends enabled. I'll split it into another issue for tracking.

(See Slack)

Originally posted by @thejcannon in #37 (comment)

What's the plan for the old blog.pantsbuild.org site?

There's a bunch of content currently hosted at https://blog.pantsbuild.org, some of which is often linked a year and a half after it was written (e.g. https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/).

With the new site seeming to try to subsume the blog right away, should we be redirecting its URLs too? Or can we potentially cut scope for the initial roll-out and leave https://blog.pantsbuild.org as is for now and then handle that as follow-up (i.e. get the main switch-over out sooner)?

Make the 404 page smart, e.g. search, "Did you mean"

Our not found page is currently static, but typically we can infer things from the URL that allow us to do more than just 'block' a user, e.g.

• break the URL up into parts and do a search: pantsbuild.org/something-here could turn into a search for something here, as would pantsbuild.org/docs/v2.19/something-here (i.e. ignore obvious 'cruft' like docs, reference or version slugs)
• bonus, iterating on #10: if it contains /v1., they're likely looking for v1 docs so direct to v1.pantsbuild.org (potentially including a site:https://v1.pantsbuild.org search) (#87 (comment))

Example: https://doc.rust-lang.org/stable/std/tree

Optimise reference "list" pages (goals, subsystems etc.)

From #4:

The reference listing page for goals etc. seems to have large cards for each item, all with content ---. Is it possible to switch to either switch to a (denser) plain list, and/or put real content in those cards? https://docs.pantsbuild.org/2.18.x/reference/goals

Generate some kind of changelog for pantsbuild/pants releases

With the switch to GitHub Releases, @huonw lamented at not having a nice changelog anymore, as the changelog is essentially spread across the github releases.

Now that we have a docs site and automation, we should be able to fold in changelog generation 🎉