axodotdev / harbor Goto Github PK

Right now we have one file that handles both get and put and we can divide it using https://nuxt.com/docs/guide/directory-structure/server#matching-http-method

When working on #33 I noticed that some baseline responsiveness is missing here. These are the two big things that should be fixed.

Text Line-Breaks

The text is being ingested with fixed line-breaks. We should find out why and not do that. We can either stop them where they are coming in and leave white-space-pre-wrap for paragraph breaks, or we can find a way to break out the paragraphs and wrap each in a p and then the internal line-breaks will be ignored.

Mobile Size CSS

This is how it currently looks. I would probably come up with something so that the sidebar is a separate layout that is navigated to, but one could also use something similar to the header list and let users pick an item from that. I don't have a real vision but it seems like we should do something.

I'm curious why we use safe-to-run and safe-to-deploy, which map to safe for dev and safe for production instead of straightforwardly saying the latter?

Also, in some cases couldn't safe for dev be a higher standard because it can access your file system? I guess in Rust that's always assumed, so no?

Currently the UI frames the Solutions/Recommendations as if they are Goals (or the updates that actually happened to your dependencies). In reality they are just "the easiest fix" and may be fairly disjointed from the goal, especially with imported audit files.

The "failures" array in the Report (which we currently ignore) contains the "goals". I have some local experimental changes to merge that info in and tweak the UI.

For instance you may have a situation like:

• failure: need audit for 3.0 (say you just update to that version)

And your audits/imports look like:

• review for 1.0 (yours)
• review for 2.0 -> 3.0 (imported from mozilla who has an exemption for 2.0)

cargo-vet will likely end up recommending 1.0->2.0 as a way to achieve an audit for 3.0. The goal and the solution are quite disjointed (and there's basically no relation to what actually changed in your dependencies, which not even cargo-vet knows)!

(I dream of cargo-vet emitting graphs to explain this stuff but that's way out of scope).

Say a user has a TODO list of "audit A and B". They opt for an incremental approach, auditing A and hitting "Commit" before working on B.

They are now in a problematic situation as the UI currently works. They need to remember to manually "clear" A before submitting B, or we when they hit "Commit" it will attempt to add another audit for A to the ledger.

More problematically, they may just realize "oops I forgot to add a note!" and intentionally resubmit A with the new comment.

There are many possible solutions to this, in increasing levels of difficulty:

• Fuck it get good (perhaps acceptable for the MVP)
• Encourage them to wait for the CI to do a full loop and open a new report id to work in
• Auto-clear everything (dangerous, would discard work in the case of a failure)
• Give them a clear button
• Remember what was done but somehow mark it as "done" in the UI and simply don't resubmit it (problematic if they DO want to resubmit it)
• Make our workflows understand old submissions and merge them
• Make cargo-vet itself understand "duplicate" or "overwrite" audits

Github has a safety valve against workflows-triggering-workflows that extends to commits created by workflows. This is Good because it prevents infinite loops and infinite CPU waste.

However we specifically do want this, especially since cargo-vet-certify is only triggered by humans, so there's no loop.

It seems doing a git commit "properly" "with oauth" may bypass this safety valve: https://stackoverflow.com/questions/72421030/merge-by-github-action-does-not-trigger-ci-that-listen-by-branch

This might be a weird one, but I thought it would be worth discussing.

Rather than throwing an "audit failed" error when packages are unaudited, could be call it something like "audit outstanding"? I know that computer people are used to the first phrasing, but framing it as something that is fixable is a nicer vibe and I like the idea we could do nice vibes.

Is this totally silly?

PROBABLY NOT NEEDED FOR MVP

If cargo-vet says you only need safe-to-run but you decide to audit for safe-to-deploy anyway that's good and should be encouraged/allowed. Ultimately the UI will need two bits of state per criteria:

• Did cargo-vet recommend this?
• Did the user audit for this?

This also necessitates "showing" non-recommended criteria somehow (maybe collapsed? in a sidebar?).

The "shields" in the sidebar for whether you've done something could also potentially be more subtle:

• nothing done (no shield)
• you did something, but not what we asked (yellow?)
• you did what we asked (green)
• you went above and beyond (blue? purple? gold star?)

If you try to submit with yellow shields there should probably be an "are you sure?" prompt.

Currently we use the generic github actions account which is ~fine but is bad branding and also may have issues (see #6).

The Sourcegraph button needs a hover state and needs to be ported to Fringe. Part of this is also considering what kinda of buttons we want: primary/seconday, success/error.

Don't really fully understand the implications of this but presumably this is better than "copy these files into your CI, hope you never have to update them!".

Various organizations will have standards for "well-formed commits" and not adhering to them can break CI.

Related to #4.

Based on discussion about the text newlines in #35 and meetings, we should interpret criteria text as markdown.

Include text to preface suggestions that makes the status clear. One way might be to insert between toggle and text either:
Toggle on to agree to text or Currently this package is marked as or something like that.

We don't need to implement these other platforms but we should make sure we understand how we would add them and don't over-hardcode github things. We should do an audit pass over the endpoints/schemas to make sure there's a reasonable path to add "gitlab" or "bitbucket" or "mozilla's phabricator" as an integration (if only so it's easier for folks to self-host this app with a bit of config).

Potentially relevant extension points off the top of my head:

• Authentication/Login
• Repo/pr "style" (github PR, phabricator PR...)
• Callback/endpoint "style" ("trigger a github action")
• Callback/endpoint url(s) ("my server is at https://phabricator.mozilla.com/cargo-vet-api")

https://github.com/axodotdev/cargo-vet-school/tree/main/.github/workflows

Now that I've learned github-script lets you do inline JS, we can just kill the certify dir which exists to create a single JS workflow. Of course this would lock the current impl even more into github but it's also not exactly complex logic. The challenge is mostly getting github to not freak out about permissions and to trigger in the right situations, which is necessarily domain-specific.

There's also just a bunch of commented out gunk from experiments and bad formatting/style.

Not sure how much of this stuff should be "published actions".

Not sure how much should be abstracted into a new rust cli (or made into builtin functionality of cargo-vet!).

We need to work out the UX for when two people submit audits at the same time (or one person submits, possibly changes, then resubmits).

• Currently the two tasks will race and the loser will get a "please pull" error.
• Thankfully no data is "lost" because we save the form state
• We could rebase and retry?
  • May result in duplicate audits being committed
• We could run the checkout action later to minimize the time when racing can happen
• ??? something smarter ???

Another case that may scope-creep #4.

Right now on error in the frontend we just show There has been an error and we should be showing the actual error so the user knows what is happening

API routes that need to be checked:

• https://github.com/axodotdev/harbor/blob/main/server/api/reports/[id]/commit.js
• https://github.com/axodotdev/harbor/blob/main/server/api/reports/[id]/commit.js#L1

We need to differentiate between the commit form and the note form, and add affordances to show the note has been saved.
.
The UI components for this may be taken care of via Sunfish and then ported here through Fringe.

proposed: axo - cargo vet

axodotdev/cargo-vet#1

There's a lot of random crap I did in the PR that I need to go over and and cleanup now that I Understand what I want out of it.

Anywhere there's ${{ something }} is a place where Bad Shit can happen. Some of them are Fine, others Aren't. Ideally I should extract more from the github context object that gets injected into github JS workflows, so that strings are already strings and not something I need to inject into quotes!

Do an audit pass before shipping.

We need to work out exactly what to do in the cross-repo commit+push situation.

• There is a permission for repo owner editing branches of cross-repo PRs and this permission is granted by default!
• However it seems like triggering the action is an extra permission? investigation required
• Can we run the action on OUR repo while still fetching THEIR branch?

It's possible dealing with this will scope-creep some aspect of #4 into the mvp.

(Also going off of what @SaraVieira said, I haven't tested that this is busted myself.)

The current design has two background tasks:

• cargo-vet-check-submit-failure task responds to cargo-vet-check failing, unpacks the zip, POSTS to the webapp, and posts a comment

• cargo-vet-certify is triggered by the webapp when you hit the "Commit" button, runs cargo vet certify in a loop, then commits and pushes

In both of these cases there can be Something Important And Useful Related To The PR happening and the the only way to know is to open the Actions admin panel on the repo (but not always the same repo!).

Potential improvements:

• Have the bot comment on the PR more about "I'm gonna do this"
• Have the "Commit" button in the GUI link you the workflow
• Have the "Commit" button disable and/or display status of the workflow
• Find a way to associate the tasks with the PR so they show up along with other CI tasks
  • May be multiple inflight copies, can be problematic

Probably lots of docs need to be written for lots of things?

I notice in https://gist.github.com/Gankra/d9ffe2cde001c6a615a3a9cf3928cddf, you mention the following filenames:

    config.toml: your project-private configuration, similar to Cargo.toml
    audits.toml: your audits and auditing-criteria, ready to be checked and shared
    imports.lock: the audits you've imported from other organizations, similar to Cargo.lock

I think we'd benefit from naming these somewhat differently both for differentiation and for grouping, something like: vet.toml vet-audits.toml vet.lock.

Right now it's not obvious what SourceGraph is, and its position in the top-right feels a bit de-emphasized. I'm not really sure what the Ideal design is, but it should probably more loudly scream "this is the button to do the review!!!".

For reference, the ostensible workflow of a user would be:

1. Open the report
2. Click on a library they want to review
3. Check what criteria they need to review it for (maybe review the definitions, maybe just remember them)
4. Click on SourceGraph
5. Review The Code
6. Come back to the UI
7. Sign off on all the EULAs
8. Maybe add some notes
9. GOTO 2
10. Click Commit When Satisfied

OUT OF SCOPE FOR THE MVP

I expect in the future we'll want this button (final submit button for the GUI):

To be a lot more like this button (github "merge PR" button):

in that it could support a few different options:

• create a commit and push it (current design)
• post a suggested edit comment on the PR
• submit a PR against the PR (META PR)
• submit a PR against the main repo (audits are ostensibly good/useful work to be saved regardless of the PR getting accepted!)

Currently the green being used for the commit button and for the toast are different — they should be the same to cutdown on the number of green variations on the screen.

Some errors:

• audit failed (normal, triggers our webapp)
• violation conflict (you have audits/exemptions that contradict the violations, not currently in scope for the webapp)
  • currently this error is a bit late-bound, because it only triggers if you have assertions that contradict exemptions
  • because it's technically fine to have a dev-dep which is known not-safe-to-deploy!
  • I don't think we have safe-guards against recommending audits that contradict known violations :(
  • technically this is actually difficult for the CI to do (#22)
• integrity errors
  • hey your file refers an undefined criteria (just print the string, more or less)

• Textarea
• ShieldIcon
• InfoIcon
• CloseIcon

cargo-vet has a mechanism where you can import audits.toml files from other projects (e.g. "I peer with mozilla, so I import their audits.toml").

cargo-vet was originally designed on the premise that humans would be running it on their local machines without --locked so we can opportunistically update imports. A mechanism was also added to minimize the churn for this: the imports.lock file only contains required information. We GC anything that wasn't (seemingly) necessary to get things audited.

CI would then run --locked which only used the imports.lock to avoid spurious breakage.

However this limits the capacity for ci/bots to know about additional useful information. In the simplest case there may simply be a new import that covers a new requirement. More subtly it may just make a requirement simpler (smaller diff). Even more subtly an import may introduce a violation and we won't know, and may even suggest audits that overlap with that violation!

Also the bot is actually better poised to update imports, as it's more trustworthy than "some human pushed a commit with imports.lock updated, trust me it's legit". Clearly the bot should have a mechanism for pulling in imports and updating them.

Before the imports GC thing I had suggested it periodically opening dependabot-style PRs, but now I think it would have to just surreptitiously insert them as part of other ops. Although maybe a cron job for checking violations is merited.

No idea what this entails yet, but probably involves #17!?