entropic-dev / ds Goto Github PK

We need i18n support for the Rust side of the client!

We previously had a spike in this direction over at entropic-dev/dstopic#2, but since the main CLI was going to be node-based, we decided to drop that work for now. Well, it turns out we changed our minds about what the primary CLI is, so it seems like it's time to revive this and hack it into ds.

Hopefully someone is interested in picking this one up!

/cc @emkay

Similar to npm-pick-manifest, we need something that looks at Entropic packuments and selects the version according to a given selector.

One super-critical part of the dependency download process for dstopic is the ability to batch downloads according to what will probably be required. To this end:

We need a Rust-based parser that can handle:

• Modern JS
• Typescript
• JSX

And return a list of toplevel import-ed and require-ed paths. dstopic will use this to speed up dependency downloads by batching requests together. Note that the parser doesn't need to completely support the above -- just enough to extract imports and requires, and it doesn't need to handle dynamic ones, since this is largely an optimization. In theory, this could even be as simple as a regex, but I'd rather go the parser route if possible.

After doing some research, I found RESSA, which looks like it would be up to the task -- most other existing Rust-based parsers seem to be fairly incomplete, or designed specifically for Neon-based usage (like swc).

(original thread)

As part of development on dstopic (the Rust side of the client), we need to have a shiny new http client with certain features:

• http-compliant caching, to cacache
• retry support
• keepalive support
• async/await support
• (optional, unnecessary, low-priority) proxy support

No existing http clients on crates.io currently support all these features (or even most of them), so we need to write one from scratch!

(original thread)

Blocked by

• Land surf-middleware-cache
  • Needs http-cache-semantics to land. (#16)
• Get surf 2.0 over the finish line. The planned API changes are gonna be important to implement this client. (http-rs/surf#107)
• Implement a baseline client API wrapper around surf
• (Optional) Integrate backoff to automatically retry http requests. Maybe this needs to be implemented as a surf middleware?

because we're handling NPM directly now

We need to design what workspaces will look like in ds. We should discuss what needs need to be fulfilled and examine the space a bit to see what others have done and how it can be improved on.

Some things I'd really like to have:

• Easy running of commands in sub-packages.
• Nested workspaces.
• Central version management for shared packages.
• Easy cross-subpackage dependency definitions (don't just rely on paths, maybe? -- I think pnpm does an interesting bit here, but maybe Yarn's approach of plain version definitions is better?).
• Look at https://bit.dev/ for more inspiration on monorepo management.
• As a stretch goal, we should have Lerna-like features for publishing subpackages.

As part of our current push on the dstopic side of things, @yoshuawuyts and I are working on a caching http client that can cache its downloaded data down to cacache.

But there's a piece missing! make-fetch-happen was fortunate enough to be able to use an excellent package called http-cache-semantics which is a fairly complete implementation of http caching logic.

Your mission, should you choose to accept it, is to port http-cache-semantics over to Rust!

Now, I've already spoken with @kornelski about it, and they are sadly unable to pick up another project right now, but they were willing to share that they had already started working on a port, so this would be a great place to start, I think, for anyone willing to pick this up?

Are you interested in doing this? Can you do it in the fairly near future (since this is currently a blocker)? Reply below and let me know! We'd love your help! I'm totally willing to collaborate on this, as well. This is a fantastic way to learn Rust, if you're coming from the JS ecosystem, too.

(original thread)

So it dawned on me that doing things from Rust means we have access to the execve(2) syscall, which replaces the current process with another process.

The idea with this spike is to prove out that a dstopic node ... command is possible where the node "child" process takes on the same pid and assumes it was invoked directly by the user.

This is an important cross-cutting concern with the Entropic server and we really need to have this hashed out and finalized before we can really commit to a lot of things.

In the interest of getting the ball rolling with #17, we should implement a ds ping command that does the very basics of communication with the entropic server.

It's a bit rough around the edges right now. We might want a custom error type and a few well-placed cuts here and there.

Welcome! If you're interested in available and current tasks and general progress on ds, please check out the project tracking board

The DsErrContext codes in packages/ds-error-context are mostly missing docstrings. It would be great to do a pass through all the current codes and make sure they have helpful docstrings attached to them!

So, the initial pick-version spike was an interesting thing to try out, but things have changed now that ds is supposed to handle NPM packages directly. We probably want to design a more general/extensible system for doing this.

The issue has been brought up that NPM's npmrc config system where it looks up rc files upwards and merges them together has lead to a lot of footguns. We should talk about this and come up with something a bit better. Maybe during our next meeting?

We need to figure out what the cross-package error signaling, code, messaging, etc, system is going to be for ds. Do we use some kind of centralized error code system? Do we stick with anyhow?

This involves implementing an initial ds check command that supports typechecking typescript source files.

This should bundle tsc, if possible.

This should create two build/test workflows: one for ds that triggers on ds-relevant pushes, and one for dstopic that does the same.

I'd like to have a prototype of fluent-based localization for the Rust side of things. This should include setting up the locale dir with all the localization files, a basic hello-world string in two languages (or three?), and proper negotiation of the language to be used at startup built into main.rs so that we can see it working and get a sense of how it feels to add new strings and actually use them in the app.

Need to create a standalone tree builder library that can represent and manipulate dependency trees. It needs to be able to read lockfiles, read node_modules, and have an understanding of links, abstract nodes, and physical nodes.

Notes

• petgraph might be helpful here. Especially StableGraph.

Blocked

I'd like to have a working HTTP client for this so we can build out a sample graph based on actual data (#17)

cacache and ssri are both licensed under the Parity license, which is a non-OSI copyleft license. This means that so long as dstopic is using those crates (which are kinda foundational), ds can't be used in proprietary contexts unless a proprietary license is purchased from those crates' author (well.. me).

Obviously, I'm biased in favor of continuing it, and I've written about this particular model of sustaining community-based software.

Proposal

I propose we:

1. Relicense ds (and by extension, dstopic, since it's in this repo now) under Parity 7.0
2. Apply the the Patron license pointed at our OpenCollective with an appropriate tier.
3. Add a note to our OpenCollective that sponsorships from military organizations, ICE, or any organization working directly with them will not be accepted.

If we do this, I'm willing to grant a blanket waiver on all my libraries such that anyone who is sponsoring Entropic on OpenCollective automatically gets a waiver to use those libraries without having to sponsor me directly.

Implications

Note that this licensing change is almost certainly going to prevent large organizations like Google, MSFT, Apple, IBM, etc from using this software, since they tend to require OSI-approved licenses. It will also prevent it from being distributed by default in many package managers, since some of them, like Homebrew, actually require that software distributed on them be licensed under OSI or FSF-approved licenses.

I believe this is fair to do, and my interest in making this change is largely to start shifting the needle towards a more fair world where community-based software maintainers are compensated for their efforts. I believe permissively-licensed software is fundamentally exploitative and mostly exists to enrich these large entities. I want to create a project by the community, for the community, that helps enrich a commons we can all benefit from while being fairly compensated for the hard work we put into it.

This is a fairly sizeable project. In short, the functionality of npm-package-arg's git parsing has to be ported over, including the stuff hosted-git is responsible for.

Oops, I thought we had one in place already. Most likely, I'd like to use https://github.com/wealljs/weallbehave#readme but we can discuss.

One big challenge for the ds/dstopic divide is this idea that there will be a single, long-lived dstopic binary that ds will then communicate with to request data be warmed up. But this comes with a tricky bit: There’s no easy way to do synchronous IPC in Node, and require is a synchronous operation.

So here’s a challenge: We need at -least- a proof of concept of how to send requests from ds to dstopic synchronously. This includes a way to wait for dstopic to be done with its task and send a success response back before the call is over.

Hint: one way to possibly do this is by using fs.writeSync/fs.readSync with a local socket of some sort. I forget the details, but that’s the first avenue I’d explore to accomplish this.

(original thread)

Currently, there's a number of open issues in the semver crate that involve bringing it to feature parity with node-semver. We need this because of the sorts of things we'll run into in the wild. This issue is a tracking issue for that work -- it's likely we'll need to do it ourselves if we want it done because dev on the crate seems stalled.

Off the top of my head, these are the missing features:

1. "loose" parsing, which involves handling garbage in semver strings. Because it's actually needed. This is an extra option in node-semver but it's used throughout NPM and the ecosystem has come to rely on it.
2. Space-separated range specifiers. This is supported by semver-parser but semver hasn't been updated to use it.
3. Or (||)-separated ranges
4. Other stuff?

Most likely, build this on top of structopt, with subcommands as subpackages.

This should add a ds shell command to ds which calls node as a child process, passes it any args it receives, and patches node with the dstopic extensions patches for typescript.

For performance reasons, we could -really- use an async version of the tar crate that is able to work on AsyncReads and Streams.

Right now, building a new version of dstopic is a bit of a manual process. You have to cd into the dstopic dir, make sure to have npm installed, and then you need to do npm run build, and then you can build a new ds release.

Until such a time when ds can host itself and manage the dstopic distro itself, we need to have a better build process for this, hopefully automating more of it so we don't forget these steps?

I'm very excited about Entropic but there doesn't seem to be any planning for a "shrinkpack" style offline cache which could be reliably added to source control. Npm in its current form does not support this so at the moment we are using Yarn. We don't want to have to rely on warm caches, we should be able to clone from source control to a fresh machine and do an install without any internet comms. What are your thoughts on this?