[Feature Request] Movey, a package metadata repository for Move, features and designs about move

[jolestar]

• All move package has an address and can be published to a chain.
  => Is there a way to reliably scan/get this address from the Move.toml or any metadata/config file? From existing sample Move.toml files we see that a lot of packages are using addresses like 0x1 (which probably correspond to their specific chain), so it's a bit hard to differentiate them using addresses.

Like this project starswap, is a swap project on Starcoin.

The address in Move.toml is 0x8c109349c6bd91411d6bc962e080c4a3, and we can get its package from the address from the chain, such as by an RPC call:

curl 'http://main.seed.starcoin.org/' \
  -H 'content-type: application/json' \
  --data-raw '{"jsonrpc":"2.0","method":"state.list_code","params":["0x8c109349c6bd91411d6bc962e080c4a3",{"decode":false}],"id":0}'

If you run a Starcoin node by yourself, you can get it from the blockchain database directly.

• => How should we do this? Movey can issue a challenge using the user's public and requires user private key signature. This will link verified Movey users to pushlished packages but there's no guarantee it will be correct for all chains. Or MoveCLI can request private keys/mnemonic but this requires trust. What do you think?

Currently, it is hard to share the Move library between Move chains, this is another topic #100.

[wrwg]

I think all this sounds like a good start, but we may need to think a bit more what is the right approach for a blockchain package repo.

• Even packages which are network neutral, cannot be referenced on every chain. They need to be deployed (at a specific requested version) on a given chain. Indeed a user could just include a package into their own deployment artifact (using address aliases?), but this is not the intended workflow of using package dependencies which are owned by others.
• This suggest that the blockchain state (as @jolestar brought up) might need to be the source of truth about what packages are available for someone to use on a particular chain. (As @gregnazario also brought up.)
• Packages need to have well-known fixed addresses on a given chain. There need to be some governance on how those addresses are allocated. Would this governance be cross-chain or may each package have a different address on a given chain?
• More than the download count is probably the # of times a given package is used on chain(s).
• There may be different versioning disciplines per chain, auditing requirements, voting models for reputation of a package, etc.

I wonder whether we will end up more with a generic Move package registry, and then still need to have some hub on top of it which is chain specific. In that case, Movey may become also an API for other package managers, which take the metadata from Movey and combine it with chain specific data. We can discuss more in the community meeting today.

@sblackshear

[lwsinclair]

To share some high level thoughts: from our perspective (at East Agile) we want to create a destination for people to use, discover, compare, and evaluate Move-based blockchain packages: that is the vision behind what we are trying to deploy at https://www.movey.net. Everything else we do is a means to that end.

So from that perspective, we don't want to be just an API for other package managers. We can be that as well, just as GitHub is being used by Movey. But if more complex chain-specific needs become necessary, we will try to be the place to go for them as well. But of course these things need to be implemented in an iterative fashion with an MVP first. We're striving for that MVP at this stage.

[sblackshear]

@ea-open-source @jolestar

I think connecting Movey package data to Move code published on various chains is an intriguing item for future work. But I do not think we should use the blockchain as the package repo for several reasons:

• There are >1 Move blockchains, and we want to emphasize the cross-platform nature of Move. I don't think it would be appropriate for us to pick one of these chains as the source of truth.
• On-chain storage costs money, but uploading to Movey or updating metadata should (ideally) be free
• Blockchains store bytecode, but Movey needs to store metadata and (eventually) source code

Now, to be clear: I absolutely think we should look at how to connect Movey to the various Move chains that will exist at some point--lots of interesting directions to explore here! But let's start with the basics of a "crates.io"-like service that accepts package uploads, counts downloads, etc. and worry about connecting with blockchains later. In other words, I think the EA starting point

Therefore, the website current goal is to act as a central metadata hub for packages hosted on Github only.

is the right one.

[ea-open-source]

@sblackshear

• I think the upload metadata should also include the git hash of the move repo used to build the package.

=> Do you mean the current commit hash of the repo when the upload command is performed? If yes we're already including it in the metadata as rev.

• Finally, I'm wondering how package de-duplication works. Is it by name (e.g., there can only be one package named move_stdlib), name + github url + rev, or something else? When you upload a duplicate, does it replace the original or does the upload fail?

=> Right now, we are checking a uniqueness combo of package name + version, so if a package Foo already existed with version 0.1 and 0.2, and a new upload request is for package Foo and version 0.3 it would register as a new entry. If the request is for package Foo and version 0.2 it would return an error (record already exists).

• To make sure I understand: the API key is associated with a particular GitHub username?

=> Currently we are associating the API key with a user account on Movey (similar to rubygems's behavior). That account may or may not yet have a connection with a github account (through github login or an explicit connect flow we can have later). Is that okay or do you feel logging in / associate user account with a github login (crates.io's way) must be done from the start?

We also agree that for the first phase Movey would be a conventional package registry, not quite blockchain-aware yet. We can add more in future versions.

[jolestar]

I was thinking about this too.

I think we can use the blockchain as the package repo and also put the package metadata on-chain.

• All move package has an address and can be published to a chain.
• Move project can add an on-chain binary package dependency (this need to change move-cli).
• Movey can sync all Move blockchain for finding the new Move package.
• Movey can analyze the dependency relationship by the on-chain package.
• Movey can verify the publisher by the private key signature.

[ea-open-source]

Thanks @jolestar for the suggestions, we have some follow up questions if you don't mind:

• I think we can use the blockchain as the package repo and also put the package metadata on-chain.
  => Currently, we are attempting to store the package metadata in a conventional database, with perhaps a reference to the address on the blockchain if possible. If we go with the storing on blockchain approach we can probably hash the package and its metadata into a persistent on-chain record, but we're not sure how much more beneficial that is (what problem it would solve, what are the things we that we want proof for?).

• All move package has an address and can be published to a chain.
  => Is there a way to reliably scan/get this address from the Move.toml or any metadata/config file? From existing sample Move.toml files we see that a lot of packages are using addresses like 0x1 (which probably correspond to their specific chain), so it's a bit hard to differentiate them using addresses.

• Move project can add an on-chain binary package dependency (this need to change move-cli).
  => Makes sense ✓

• Movey can sync all Move blockchain for finding the new Move package.
  => Could you give us some direction to accomplish this? We're not quite blockchain-savvy so we're not sure how one would sync all Move blockchain to find/scan for new packages.

• Movey can analyze the dependency relationship by the on-chain package.
  => Makes sense ✓

• Movey can verify the publisher by the private key signature.
  => How should we do this? Movey can issue a challenge using the user's public and requires user private key signature. This will link verified Movey users to pushlished packages but there's no guarantee it will be correct for all chains. Or MoveCLI can request private keys/mnemonic but this requires trust. What do you think?

@sblackshear @tnowacki could you guys share your thoughts about these questions?

[gregnazario]

How does this handle packages that use modules from different chain's frameworks / adapters?

[sblackshear]

@ea-open-source working through your questions:

Does this approach make sense to you, or do you have a more orthodox way in mind?

Yep, makes a ton of sense. My only feedback/questions are:

• I think the upload metadata should also include the git hash of the move repo used to build the package. We currently don't have this in Move.toml or elsewhere (though that may change soon), but I think it is useful to record this in Movey to enable reproducible builds.
• Similarly, I think the upload metadata should include the hash of the package source + build artifacts (also useful for reproducible builds). CC @tnowacki , who might know which of these hashes the build system already knows about
• Finally, I'm wondering how package de-duplication works. Is it by name (e.g., there can only be one package named move_stdlib), name + github url + rev, or something else? When you upload a duplicate, does it replace the original or does the upload fail?

[sblackshear]

Movey may also provide each registered user with an api token, and we plan to integrate it to the move-cli tool (either via environment variable or a config file), then sends it along with the upload request so that Movey can recognize the user and set them as the package owner. Is this preferable or do you have any suggestion of wrapping this?

Makes sense to me. To make sure I understand: the API key is associated with a particular GitHub username?

[sblackshear]

Does this work with the way Move packages are managed, or is there a better way to recognize a new "download" of a package? After all, our goal is to have a metric similar to "downloads count" of crates.io or rubygems.

Ah, interesting. I suppose in other systems, the package is hosted on the Movey server and the server would be in charge of tallying a download for each request. But here, since GitHub hosts everything, the CLI hook is the only feasible way?

I can't think of a different approach, but my concern with this one is that it will break Move builds in CI. Most CI systems (wisely) won't let the CI build machines hit any endpoint as part of the build process, only a few special build-relevant ones in an allowlist. However, if make the request best-effort and ensure that it doesn't break the build if the "increment" request to Movey fails, I think it's ok.

If/when Movey is actually hosting the package source as well, I think we can look at getting popular CI providers to add the Movey servers to the allowlist.

[sblackshear]

Because of the nature of storing these packages/their compiled binary on chains, it would be super helpful if you can point us in a direction to accomplish this (or if it's feasible or not).

In the short term, I don't think (b) will be feasible. I think that is actually a nice growth hack, though--good "nudge" for folks to upload their packages to Movey if they want them to be visible/discoverable!

In the longer term, I think we can ask block explorers to implement some set of public API's that help Movey with (b) (e.g., do you have a module with this name, what are the dependencies of this module, ...), and create an allowlist of block explorers that Movey knows to query. But I would punt this to the future "phase" of going beyond a package repository to a blockchain-aware package repository.

[jolestar]

I think connecting Movey package data to Move code published on various chains is an intriguing item for future work. But I do not think we should use the blockchain as the package repo for several reasons:

• There are >1 Move blockchains, and we want to emphasize the cross-platform nature of Move. I don't think it would be appropriate for us to pick one of these chains as the source of truth.
• On-chain storage costs money, but uploading to Movey or updating metadata should (ideally) be free
• Blockchains store bytecode, but Movey needs to store metadata and (eventually) source code

Assume a scenario:

Project ABC uses the address 0xabc.

Project DEF uses the address 0xdef, and wants depends on ABC. if it wants to deploy its package to X Chain, its developer must check two things first:

1. Is the ABC exists on the X Chain?
2. Which version of the ABC is deployed on the X Chain?

So, the developer can not just copy a dependency URL from Movey and paste it to Move.toml like Rust developer do. He must know which chains the package has been deployed on and which versions are on the chains.

Finally, My suggestion "use the blockchain as the package repo" means Movey needs to care about the binary package version on Move chains.

[sblackshear]

Do you mean the current commit hash of the repo when the upload command is performed? If yes we're already including it in the metadata as rev.

The build system has a hash called a PackageDigest that summarizes all the contents of a package. If it's possible, I think it's also useful to include this in the upload so users can quickly verify that a package they have locally matches one in Movey.

=> Right now, we are checking a uniqueness combo of package name + version, so if a package Foo already existed with version 0.1 and 0.2, and a new upload request is for package Foo and version 0.3 it would register as a new entry. If the request is for package Foo and version 0.2 it would return an error (record already exists).

Got it, this makes sense. And only the API key that published (Foo, 0.1) has the permission to publish (Foo, 0.2)?, or other versions of Foo?

=> Right now, we are checking a uniqueness combo of package name + version, so if a package Foo already existed with version 0.1 and 0.2, and a new upload request is for package Foo and version 0.3 it would register as a new entry. If the request is for package Foo and version 0.2 it would return an error (record already exists).

Discussed this with the Move core team and there was a preference for login via GitHub. The main reasons for this preference is to piggyback on GitHub account security (e.g., most folks working on crypto projects in GitHub have strong passwords + 2FA) and avoid the friction of needing to create (and secure) yet another account.

[sblackshear]

I wonder whether we will end up more with a generic Move package registry, and then still need to have some hub on top of it which is chain specific. In that case, Movey may become also an API for other package managers, which take the metadata from Movey and combine it with chain specific data.

This is the direction that makes sense to me. I think this means that the Movey team is on the right track with their initial prototype, but that we may need to work closely with them in the future on the API's for chain-specific package managers.

[wrwg]

@lwsinclair it would be great and beneficial for every individual network if we have a unified package landing page. However, we aren't yet in a position to formulate requirements how this would look like. For example, the versioning model for Move is still under discussion. We may also need to experiment with network specific workflows under real world (mainnet) conditions before we know more. Going with Movey for an MVP, and then refining as things become clearer sounds like a good direction.