entropic-dev / entropic Goto Github PK

It forwards errors to self.destStream, which may point at different streams over time. I've tried listening on errors on each part coming through, but... no dice.

Invitations should be working already, more or less. We need to fill out the namespaces workflow.

First: The two-booleans approach to mark status is not so great. We need a status enum. Valid statuses for namespace_memberships:

• active: has full privileges
• invited: has been asked to join
• inactive: a rejected invitation or removed member is inactive; can be invited again
• banned: cannot be invited again (trying to anticipate a future need here)

Invitations & permissions code will need to be updated;

After that, we can fill out the namespaces feature:

• There should be a way to create namespaces in the cli.
• The api endpoint that creates a namespace should make the logged-in user a member of the namespace.

The top-level readme needs to document our goals & major choices in an easily-grasped narrative. This description should not be particularly technical in focus.

Looks like maybe sonic-boom's "expose intrinsics" approach may need to be updated to work with Node 12?

return typeof s === "string" ? %FlattenString(s) : s
                               ^

SyntaxError: Unexpected token %
    at actualWrite (/Users/chris/projects/personal/entropic/src/node_modules/sonic-boom/index.js:242:3)
    at SonicBoom.write (/Users/chris/projects/personal/entropic/src/node_modules/sonic-boom/index.js:142:5)
    at Pino.write (/Users/chris/projects/personal/entropic/src/node_modules/pino/lib/proto.js:136:43)
    at Pino.LOG [as error] (/Users/chris/projects/personal/entropic/src/node_modules/pino/lib/tools.js:35:26)
    at Timeout._onTimeout (/Users/chris/projects/personal/entropic/src/registry/middleware/dev-only.js:36:16)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)

It should be easy (eventually; possible is a good start) to point one entropic instance at another and make a read-only mirror of its packages. Perhaps the packages would go under a namespace that is the sitename of the site being mirrored?

If we have workers doing things, we suddenly need an API for the workers to talk to. This API should not be the same thing as the one we expose to the outside world. CONGRATS we've just grown a second service! Make it happen.

I wish to make it clear that I am in fact giving it away, and getting it out of my gh namespace will help.

Two api endpoints to implement here, at least:

• /packages/authored-by/:namespace : packages owned by the namespace ("namespace" is in their name0
• /packages/maintained-by/:namespace: packages that the namespace has write permissions for (not necessarily in that namespace). Now that I think about it, shouldn't that be packages the user can write? Users are part of namespaces, and users have maintainership relationships with packages, so... (@chrisdickinson confirm intentions here?)

Then implement cli commands to invoke these endpoints.

If the registry we're providing a read-through cache for is down or slow, we want to short-circuit and respond with what's in cache. Ideally we'd be able to configure the timeout.

This is a feature we can let wait, but it would be neat to block a package-version with a security warning associated.

There's a function registered as a handler for /-/whoami but it isn't implemented yet

In the readme you mentioned that you want to use a content addressable storage.

There are existing content addressable systems like IPFS that you can leverage.

I’ve recently spoken with IPFS engineers and they are really interested in making IPFS easy to use for package managers so they might be open to implement features you need.

• help
• add-maintainer/remove-maintainer
• namespace-create (need to talk about namespace-yank)
• add-to-namespace / remove-from-namespace
• abandon/yank package
• deprecate/yank package-version

Depends on:
#52
#53

Issue should probably get broken up but I wanted to record the work somewhere.

We should probably discuss our approach toward command design. As an opening salvo I will say that I prefer short words for commands and no elaborate options for the targets of those commands. Argument order should preferably not be meaningful, or if meaningful unambiguous given existing context.

Example: ds add-maintainer chrisdickinson namepace/pkg is unambiguous in this example, but hard to predict without an example. Is it ds add-maintainer target-package target-human or does the human go first? Maybe it's great, though, because that last argument can be implicit if you're in a directory with a package.json.

ds add-maintainer user=chrisdickinson pkg=namespace/pkg is unambiguous but kinda weird.

ds maintainers add chrisdickinson to namespace/pkg is unambiguous if you speak English, might be hard to parse or translate.

yank is internal jargon that means two things: 'abandon' and 'deprecate'. Do we want to just let the jargon leak out into the user experience?

We can download a package and stuff it into the cache, but we are not yet able to install it into a node modules folder in a usable format.

We're mirroring metadata, but the tarpaths are hard-coded.

The conflation between packages and namespaces isn't so great. Right now, it's ds invite <user|namespace> --to <namespace|package>. This is fairly clear about the order of arguments if you speak English, but what can you can invite to what isn't so obvious.

Some facts:

• Namespaces can have many users. You invite users to namespaces.
• Packages are maintained by namespaces. You invite namespaces to package.
• Users accept or decline invitations to namespaces as their logged-in selves.
• Users accept or decline invitations to maintain packages on behalf of namespaces.

I suggest adding a --namespace or --package parameter to make it unambiguous.

• ds invite potter --namespace gryffindor: dumbledore invites potter to gryffindor
• ds accept --namespace gryffindor: potter accepts
• ds invite malfoy --namespace gryffindor: potter invites malfoy
• ds decline --namespace gryffindor: malfoy declines
• ds invite gryffindor --package quiddich-cup: dumbledore invites gryffindor to maintain the quiddich-cup package
• ds accept --package quiddich-cup --as gryffindor: potter accepts the invitation to maintain the cup on behalf of gryffindor
• ds remove malfoy --package quiddich-cup: potter removes malfoy-the-namespace from the quiddich cup

To fill out the interactions for the invitations feature, I propose changing ds invitations as follows:

• ds invitations --packages: list all maintainership invitations the logged-in user can accept
• ds invitations: list all namespace invitations for the logged-in user

In each case, the invitations lister gives the exact commands the user can run to accept.

Self-hosting for the cli is a goal within our sight.

pacote uses cacache which has a file system backing. We need to make s3 or some other kind of object store available as a pluggable storage engine.

Right now we have just enough website to do an oauth flow. We need one that shows package information.

After running curl -sSL https://www.entropic.dev/install.sh | bash, trying ds login gives the error below:

Error: /Users/flxwu/.entropicrc
    at load (/Users/flxwu/.nvm/versions/node/v8.11.4/lib/node_modules/ds/lib/config.js:21:11)
    at main (/Users/flxwu/.nvm/versions/node/v8.11.4/lib/node_modules/ds/lib/main.js:19:26)
    at Object.<anonymous> (/Users/flxwu/.nvm/versions/node/v8.11.4/lib/node_modules/ds/lib/main.js:49:3)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:191:16)

Our first take on maintainer add & remove makes the changes directly. We should instead generate invitations that are sent to the would-be maintainer, so that they may accept or refuse.

Are blocklists a thing? I sometimes wish I could block anything from ever installing oniguruma in any project I work on.

The threat model is: somebody inherits the domain name for a server, gets a valid cert for it, and then runs an entropic with malware-injected versions of the packages from the original entropic instance. Proposal: If one entropic instance installs packages from another, they should exchange some kind of fingerprint that identifies them to each other. Sweat the details, find a good solution.

Implement flows for enabling and disabling 2FA, including using recovery codes.

Implement login flow for 2FA. (This looks like "oauth success -> user requires 2fa, enter otp code -> on success continue, on error retry")

First, congrats on the project!
I was not present in the announcement in the JS Conf EU but it looks like it was amazing.

At my company, Wasmer, we created a WebAssembly Package Manager and used GraphQL for the API. We iterated very quickly with it.

I would really recommend to use GraphQL as a first class citizen for the API, it helps both for the perspective of the server and also for any client using it.
It could also help for any other entity trying to launch a registry that is complaint with your API (since they will just need to validate the their schema against the official one).

Hopefully it's useful... keep up the great work! 👏

Authenticated, authorized users can:

• create new packages
• publish new package-versions

It would simplify MAINTAINER's workflow if they could use GitHub, or other foreign credentials to tie to their registry account. This would simplify team management (e.g. one could setup so that anyone with push access to the repo can publish, etc.).

So. Close...

We need some kind of administrative tools to provide a first-line counter to abusive content, spam content, or just plain illegal content. These tools will need to do at least the following:

• delete individual packages
• ban users and delete all their packages

• elastic IP
• repoint dns to the eip
• upsize the instance from t3.medium maybe
• backups or other safety mechanism for the packages
• redis -> small elasticache
• postgres -> small rds

Pinning this issue (first reported by @flxwu & underlying issue identified by @itsananderson – thanks!)

If you're running into issues logging in after installing ds, check your Node version. At the time of writing, ds only supports Node 12+.

Lots of options:
Publish it to a legacy-api registry so existing clients can install it.
Put the tarball somewhere existing clients can install from.
And publish it to the home entropic node so it can self-update. (Well, if it can self-update. I sense a research topic.)

Preferred option: curl to shell.

No, you cannot embed an ad on Vox media that creates entropic tokens for people, no matter how helpful you think it'd be.

There are good templates for this available.

I just ripped this out when I ripped out the legacy proxy endpoints, but it would be a nice-to-have feature for the core registry.

Hello and thanks for the project. It’s not clear from the manifesto will it be possible to have package scripts? At the moment it’s very handy part of package.json

80% is a conservative goal, but I think it's an acceptable first goal.

Chris was muttering something about switching from mocha to ava, which is fine by me. Whatever works.

• create a namespace
• add/remove users from the namespace

Is abandoning a namespace a thing?

Supporting json based description of package seems pretty important, even support for incompatible version of package.json would be a plus.

And my personal preference would be also adding an option to load package description as a js object (without necessity of placing quotes) and with possibility to define scripts not only as a shell commands, but also as functions to which command line arguments and environment variables are passed.

{
  name: "[email protected]/ds",
  version: "0.0.0-beta",

  scripts: {
    test: "node test.js" // classic shell script
    compile: function(args, env, shell) {
      shell.run("ts " + args[0]);
    }
  },

  dependencies: {
      "@iarna/toml": "^2.2.3",
      "[email protected]/figgy-pudding": "^3.5.1"
  }
}

Your solution could support all these 3 formats (toml > json > js like)

Run scripts.

start
pre-start
pre-pre-start etc etc
test
-- to pass arguments to a script
the bash environment set up to make things run

API probably looks like this:

• GET /packages/package/:namespace/:name/maintainers
• PUT /packages/package/:namespace/:name/maintainers/:maintainer
• DEL /packages/package/:namespace/:name/maintainers/:maintainer

NB: package maintainers are a separate concept from namespace members. Namespace members implicitly get write access to all packages in the namespace.

We don't have tests. It will help potential contributors contribute if we have tests. We should write some.

We should see if we can make it easier to build a test rig for the registry, or if the provider-function approach we have now is enough.

It would be good to introduce the world to the project, what its goals are, what its goals aren't, and how people can help if they want to.

Inspired by the closing keynote of JSConf EU. We have some gestures toward localizability in error codes, but we need to be rigorous about it. Also, we need to have a plan for getting there with the client.

Here's what I'm thinking the priority list is:

• tests
  • fix linting
• validation
• offline work
  • syncing
    • create legacy maintainer link to synced packages
  • revalidation
  • unify sync and publish
• EMFILE in storage layer
• s3 backend
• cli
  • bin files
  • better output on publish
  • init
  • help functionality (thanks @toddself!)
  • build
    • devDependencies
    • install
    • install
    • stanza bonanza (support the following load-bearing stanzas from package.json in Package.toml & package-versions)
      • bin stanza
      • main stanza
• www
  • styles?
  • separate process

We want somewhere to talk about the project, with decent moderation tools. I do not think Slack is a good community support tool any more.

Replacing #27 in order to pin this. I want give more context on the pending software design decisions we need to make in order to build out a website (this affects #102 and #103, which are much appreciated!)

the problem

Entropic needs a website, because:

1. In order to allow people to sign up for accounts, we use GitHub OAuth login for signup and token creation
2. In order to allow people to manage their account, we display a token management UI.
3. In order to aid discovery & vetting of packages, we want to display information about packages (dependencies, dependents, etc.)
4. In order to aid vetting of packages, we want to display information about package contents (styled, rendered package contents as html)

The current website doesn't have any styles, is likely inaccessible, and is hard for developers to extend. Importantly, it's also embedded into the registry service, which could present security & caching concerns down the line.

constraints, rakes in the yard, & open questions

splitting into services

At present, Entropic runs in a single process. Because certain tasks (like syncing external packages) are too time-consuming to run in the duration of a request/response cycle, we need to split this out into multiple services. Eventually, I believe we'll need 4 services:

1. The existing external registry API service we expose today, talking to...
2. A ✨new✨ internal api encapsulating our data stores that the website, worker, and registry service can leverage.
3. A ✨new✨ external website service talking to that same internal api
4. A ✨new✨ worker service for processing syncing requests, talking to the internal api

For the production entropic instance, @ceejbot took a clever approach with nginx proxy_pass's so that we can simulate privileged APIs for the website to use by mounting them on the registry at non-exposed routes (for example, /private/.) This lets us start working on the website without having completed the other two ✨new✨ apis. However, until we split out the website process from the registry process, web handlers will use HTTP clients to talk to internal handlers in the same process. We'll start this process for the registry as well.

This is quite a mouthful: this is all to say that when the website starts needing to get at registry data, please pull in @chrisdickinson and/or @ceejbot.

Single page app or Server Side Template Language?

We'll want to consider whether we use a server-side templating language like Nunjucks to render pages in a traditional fashion, or go a single page app route using something like React + Next.js. Right now I am leaning towards server side templating using Nunjucks (or similar):

• Our present Oauth & CLI signup needs lean on url routing. (They could, yanno, not. But.)
• Our form needs are pretty basic.
• SPA's introduce a certain level of maintenance complexity (bundling FE assets, etc.) Next solves this pretty nicely, which is one of the (many) things I like about it, but it's still a cost.

I could be wrong here, so please poke holes in this. What are we buying with a single page app?

CSS

Let us do the simplest thing that works, then iterate. I propose we use Tachyons for a base coat, then specialize with an added stylesheet. (When this gets unwieldy, that's when we should start looking into css-modules or css-in-js as an additional layer on top of tachyons.)

Document the config options, the run scripts, the requirements.

Document how to use the Docker compose thingie to develop locally.

Key config concepts like:

• set the hostname, so users created on this instance have a fully-qualified name
• create the first user
• set up admin privileges for that first user
• or perhaps build in an admin user with randomly-generated credentials that the owner must store or change somehow

This all should take place out of band of the website, preferably.

Things like abandonware, legacy, and git will already be in use in entropic. Other names might need to be disallowed to prevent user confusion. This might belong in an editable text file. Think it through.