monadic-xyz / ipfs Goto Github PK

I've been installing from git, but would be nice to use version from hackage for CI, etc.

This stemmed from a conversation with @kim . Currently in oscoin we have a BaseN.hs module which is quite pleasant as it uses data kinds to provide type safe base conversions, something that (from what I could see by skimming this library) it's a bit more verbose in the current implementation.

Ideally we should pull the oscoin code into this repo and finally switch oscoin to be using this library.

Have I synthesised things correctly?

As expected, this creates too much memory fragmentation. Trouble is, we will have to re-implement all BaseN-encoding on top of an alternative representation (see also #7), as otherwise we'll end up incurring a memcpy in almost every place multihashes are used.

Even with a medium sized repo like oscoin/radicle a push eats up all of my 16GB of RAM.

ApiV0PinLs has arg as required argument, while https://docs.ipfs.io/reference/api/http/#api-v0-pin-ls states that it is optional. By enforcing the arg parameter one is unable to list all pinned objects of a node. Would it be possible to rectify this? Is editing the file and sending a PR an option or would those changes be lost because you run the auto-gen?

Deprecation indicator

When installing with nix:

setupCompileFlags: -package-db=/build/setup-package.conf.d -j4 +RTS -A64M -RTS -threaded -rtsopts
[1 of 1] Compiling Main             ( Setup.hs, /build/Main.o )

Setup.hs:3:1: error:
    Could not find module ‘Distribution.Extra.Doctest’
    Perhaps you meant Distribution.Simple.Doctest (from Cabal-3.2.1.0)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
3 | import Distribution.Extra.Doctest (defaultMainWithDoctests)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When cloning an existing project, I get the following error:

testing ; rad project checkout 12D3KooWSxYcebkWUAFzoinjtVjFviUBNc5gTcPwJMQdS3APhsac
Test
Initialized empty Git repository in /Users/cory/Code/testing/Test/.git/
Resolving IPNS name 12D3KooWSNQQiHkDatpsAicSVM2MuAGQkb9JrdjcrfYGAEQL1yCW
Unexpected link type: 0 -- Object (fromList [("Objects",Array [Object (fromList [("Hash",String "QmdH1b5Rcdqxn64Qe3QzdPjrvjMhLzZjN8DaQeaF4APBM5/refs/heads"),("Links",Array [Object (fromList [("Hash",String "z8mWaFMWEuUL1irweNi8qzKtjb9qvbkcA"),("Size",Number 0.0),("Name",String "master"),("Type",Number 0.0)])])])])])  (src/Network/IPFS/Git/RemoteHelper/Client.hs:136)
The repo is currently not available. Cloning "Test" from "ipfs://ipns/12D3KooWSNQQiHkDatpsAicSVM2MuAGQkb9JrdjcrfYGAEQL1yCW" failed.
This may be because you have no internet connection, or the IPNS link is stale, or because no online node with the data could be found.

This error comes up when i am hosting the project from another computer on the same LAN, but also i get this same error when trying to clone in a different folder on the same device where the repo is being hosted from..

Will file as an issue in the radicle project as well for visibility, but it seems like this is specific to the git-ipfs integration.

Right now, I cannot write a function that uses Multibase.encode . encodeAtBase base where it takes base as an argument because I cannot write the ToCode class restriction, since that class is not exported.

Seems useful to be able to tag a Multibase value by its Base, to be able to enforce a certain encoding in application code.

Should respect the environment variables IPFS_API_HOST and IPFS_API_PORT

Seems at least the base32 bytestring library is broken under GHC9

Not quite sure this codepath actually works.

Output some meaningful and/or entertaining progress during push/fetch.

It would be nice to have some semi-scientific experiments which could help answer the following questions:

• What is the space overhead of loose vs. packed object storage for non-trivial git histories?
• What overhead does the ipfs local storage add on top of that?
• How does fetch scale, considering that a lot of (smallish) objects have to be replicated?
• What is a good number of concurrent connections to the (local) ipfs daemon?
• Consequently, what is a good number of OS threads to reserve for the remote helper?

There are some opportunities for concurrency (mainly network IO with the ipfs daemon). Would require a bit of restructuring of the application, though

See oscoin/oscoin#505 . We now have a bunch of tests in oscoin that belongs to multibase and multihash-cryptonite, really:

tests :: TestTree
tests = testGroup "Multihash"
    [ testGroup "Examples from https://github.com/multiformats/multihash"
        [ testGroup "sha1(\"multihash\")"
            [ testCase "Base16" $
                example BaseN.Base16 C.SHA1 "multihash"
                    "111488c2f11fb2ce392acb5b2986e640211c4690073e"
            , testCase "Base58" $
                example BaseN.Base58btc C.SHA1 "multihash"
                    "5dsgvJGnvAfiR3K6HCBc4hcokSfmjj"
            , testCase "Base64" $
                example BaseN.Base64 C.SHA1 "multihash"
                    "ERSIwvEfss45KstbKYbmQCEcRpAHPg=="
            ]
        , testGroup "sha2-256(\"multihash\")"
            [ testCase "Base16" $
                example BaseN.Base16 C.SHA256 "multihash"
                    "12209cbc07c3f991725836a3aa2a581ca2029198aa420b9d99bc0e131d9f3e2cbe47"
            , testCase "Base58" $
                example BaseN.Base58btc C.SHA256 "multihash"
                    "QmYtUc4iTCbbfVSDNKvtQqrfyezPPnFvE33wFmutw9PBBk"
            , testCase "Base64" $
                example BaseN.Base64 C.SHA256 "multihash"
                    "EiCcvAfD+ZFyWDajqipYHKICkZiqQgudmbwOEx2fPiy+Rw=="
            ]
        ]

    , testGroup "Roundtrip"
        [ testProperty "Base16: decode . encode = id" . property $ do
            bs <- forAll genBytes
            propRoundtrip BaseN.Base16 C.Blake2b_256 bs

        , testProperty "Base58: decode . encode = id" . property $ do
            bs <- forAll genBytes
            propRoundtrip BaseN.Base58btc C.Blake2b_256 bs

        , testProperty "Base64: decode . encode = id" . property $ do
            bs <- forAll genBytes
            propRoundtrip BaseN.Base64 C.Blake2b_256 bs
        ]
    ]

propRoundtrip
    :: ( Multihashable a
       , DecodeBase    b
       )
    => Base b
    -> a
    -> ByteString
    -> PropertyT IO ()
propRoundtrip base algo bs =
    let digest = C.hashWith algo bs
        enc    = encodeAtBase base . Multihash.fromDigest
        dec    = decodeAtBase base . BaseN.encodedBytes
     in (dec . enc) digest === Right digest

example
    :: Multihashable a
    => BaseN.Base b
    -> a
    -> ByteString
    -> ByteString
    -> Assertion
example base algo input expected =
    BaseN.encodedBytes (multihash base algo input) @?= expected

--------------------------------------------------------------------------------

genBytes :: MonadGen m => m ByteString
genBytes = Gen.utf8 (Range.constantFrom 8 8 512) Gen.unicodeAll

multihash
    :: Multihashable a
    => BaseN.Base b
    -> a
    -> ByteString
    -> BaseN.AtBase b
multihash base algo bs =
    encodeAtBase base $ Multihash.multihash algo bs

• Base2
• Base8
• Base10
• Base32z

The dubious Base1 can be skipped, I guess.

what information would a proficient haskell user need in order to install the package and use it within their project.

describe any important design decisions made within this repo

link to relevant IPFS documentation or tutorials

Would be nice to get some interactions (create git repo, add commits, push, clone) going programmatically. Not sure how to run ipfs daemon on CI.

When I follow the IPNS example pushing results in an error

$ repoid=$(ipfs key gen --type=ed25519 myrepo)
$ git remote add ipns ipfs://ipns/$repoid
$ git push ipns
Resolving IPNS name 12D3KooWDzhxCXPWYet1NwKmWQomkqDVypV3VWED3b2zmMznNsYd
git-remote-ipfs: FailureResponse (Response {responseStatusCode = Status {statusCode = 500,
statusMessage = "Internal Server Error"}, responseHeaders = fromList [("Access-Control-Allow-
Headers","X-Stream-Output, X-Chunked-Output, X-Content-Length"),("Access-Control-Expose-
Headers","X-Stream-Output, X-Chunked-Output, X-Content-Length"),("Content-Type","application/json"),
("Server","go-ipfs/0.4.18"),("Trailer","X-Stream-Error"),("Vary","Origin"),("Date","Fri, 25 Jan 2019 14:05:10
 GMT"),("Transfer-Encoding","chunked")], responseHttpVersion = HTTP/1.1, responseBody = "{\"Message
\":\"could not resolve name\",\"Code\":0,\"Type\":\"error\"}\n"})

The problem is that the IPNS record for the key was never set.

A workaround is to set the the IPNS record to some dummy entry

ipfs name publish --key myrepo $(echo | bin/rad-ipfs add -q)

I was looking to manipulate and serialize/deserialize IPLD objects, and I came across a few of the libraries listed here in the process.

It looks like the current set of libraries does not include the above, so I figured I'd make an issue in case you thought it was in scope.

I may end up making the libraries myself in my free time, likely on top of some of your libraries, in which case I'll open source them and we can discuss if it's worth merging them into your repo later.

Right now uses a bouquet of baseN packages, which also do not support all the variants demanded by multibase. It might be worth inlining all of those, and add efficient implementations of the variants (eg. upper/lower case alphabets).