ipfs-inactive / js-ipfs-mfs

[ARCHIVED] now part of the https://github.com/ipfs/js-ipfs repo

🔒 Archived

The contents of this repo have been merged into ipfs/js-ipfs.

Please open issues or submit PRs there.

MFS (Mutable File System) JavaScript Implementation

JavaScript implementation of the IPFS Mutable File System

The MFS spec can be found inside the ipfs/specs repository

Lead Maintainer

Alex Potsides

Table of Contents

• Install
  • npm
  • Use in Node.js
  • Use in a browser with browserify, webpack or any other bundler
  • Use in a browser using a script tag
  • A note on concurrency
• Contribute
• Changelog
• License

Install

npm

> npm i ipfs-mfs

Use in Node.js

const mfs = require('ipfs-mfs')

Use in a browser with browserify, webpack or any other bundler

The code published to npm that gets loaded on require is an ES5 transpiled version with the right shims added. This means that you can require it and use with your favourite bundler without having to adjust asset management process.

const mfs = require('ipfs-mfs')

Use in a browser using a script tag

Loading this module through a script tag will make the mfs obj available in the global namespace.

<script src="https://npmcdn.com/ipfs-mfs/dist/index.min.js"></script>
<!-- OR -->
<script src="https://npmcdn.com/ipfs-mfs/dist/index.js"></script>

A note on concurrency

The mfs works by storing a reference to the root node's CID in LevelDB. LevelDB does not support concurrent access so there are read/write locks around bits of the code that modify the the root node's CID.

A lock is kept on the main thread and any requests to read/write from workers or the main thread itself are queued pending release of the lock by the existing holder.

Reads are executed together, writes are executed sequentially and prevent any reads from starting.

If you are using IPFS in a single process or with the node cluster module this should be completely transparent.

If you are using Web Workers there is no way to globally listen to messages sent between workers and the main thread so you will need to also use the observable-webworkers module to ensure the right message transports are set up to allow requesting/releasing the locks.

Contribute

All are welcome, please join in!

This repository falls under the IPFS Code of Conduct.

Open an issue or send a PR - see CONTRIBUTING.md for how to make sure your branch is ready for PRing.

Changelog

See CHANGELOG.md for details of what has changed between releases.

License

MIT

js-ipfs-mfs's Issues

Test failures for files.read*Stream

Getting test failures in js-ipfs for the new files.read*Stream methods:

1) interface-ipfs-core tests .files.readReadableStream should not read not found, expect error:
   TypeError: stream.on is not a function
    at Context.it (node_modules/interface-ipfs-core/js/src/files/read-readable-stream.js:41:14)
    at process.topLevelDomainCallback (domain.js:121:23)

2) interface-ipfs-core tests .files.readReadableStream should read file:
   Error: Timeout of 40000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.


3) interface-ipfs-core tests .files.readPullStream should not read not found, expect error:
   TypeError: read is not a function
    at next (node_modules/pull-stream/sinks/drain.js:16:11)
    at sink (node_modules/pull-stream/sinks/drain.js:37:9)
    at pull (node_modules/pull-stream/pull.js:41:14)
    at Context.it (node_modules/interface-ipfs-core/js/src/files/read-pull-stream.js:40:7)
    at process.topLevelDomainCallback (domain.js:121:23)

4) interface-ipfs-core tests .files.readPullStream should read file:
   Error: Timeout of 40000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

Cannot read data after write with cidVersion=1

Dependencies have moved and causing warnings

• Version: 0.11.5 (from js-ipfs 0.36.4)
• Platform: Multiple
• Subsystem: Dependencies

Type: Bug

Severity: Low

Description:

During installation or yarn upgrade IPFS is generating warnings about deprecated versions.
Specifically ...

warning ipfs > ipfs-mfs > [email protected]: This module has moved and is now available at @hapi/boom. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
warning ipfs > boom > [email protected]: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
warning ipfs > ipfs-mfs > [email protected]: This module has moved and is now available at @hapi/joi. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
warning ipfs > ipfs-mfs > joi > [email protected]: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
warning ipfs > ipfs-mfs > joi > [email protected]: This module has moved and is now available at @hapi/topo. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
warning ipfs > ipfs-mfs > joi > topo > [email protected]: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.

Some of these are in this module's dependency, but probably will go away if joi and hoek are fixed.

Everything seems to work though when people are installing library, then real problems are getting masked by the warnings.

Steps to reproduce the error:

yarn upgrade in any application that has a dependency on IPFS.

mfs flush output changed

+#### Changed: ipfs files flush
+
+The ipfs files flush command now returns the CID of the flushed file.

ipfs/kubo#6223 (comment)

ls directory size 0

The size property for directories in output from ls is always 0.

ipfs.files.write hangs without error message with certain arguments

Opening an issue here per @achingbrain's response to my IRC post.

Is it immediately obvious to anyone what would happen if you passed the string '/[object Object]' as the path in the ipfs.files.write method in js-ipfs? Is it possible it can't handle either spaces or square brackets in the string and is not returning any errors for that condition? Other incorrect things I pass in (like objects) cause error messages, but this (or other examples of strings with spaces and/or square brackets) appear to just hang. I can't guarantee it's IPFS's fault because of the context within ProtoSchool, but this is my working theory at the moment. 😂 For full context and to see what other theories I've experimented with, see the ProtoSchool issue: ProtoSchool/protoschool.github.io#252

per @achingbrain:

I don't think it's related to the path, I think it's something to do with the file content argument. That is, writing Buffer.from('hello world') to /[object Object] works for me but writing the passed file argument hangs as described in that issue
Could you open an issue on the js-ipfs-mfs repo with a reference to the protoschool issue above please and I'll look into it next week
Weird, adding the file to IPFS via ipfs.add(file) errors out too, I guess it's failing to turn it into a stream

I'm not sure I understand this theory, since I have no trouble writing the same kind of browser file object in general when I provide reasonable strings as the path (from that ProtoSchool lesson).

To be abundantly clear, I don't need to be able to write things to the path /[object Object], I'd just love for it to return an error if it's not going to work. :) (Error messages sent back by IPFS get surfaced as the feedback we show to the user in ProtoSchool unless we've provided a fancier error message for something predictable.)

I also readily admit the possibility that it's a ProtoSchool problem as opposed to an IPFS problem, just need an IPFS pro to help me confirm the source. Thank you!!

Cannot append to end of file with CLI

Seems to be the case for daemon on and daemon off:

js-ipfs:

$ echo "hello" | ipfs files write /test
$ ipfs files read /test
hello
$ echo "hello" | ipfs files write /test --offset=6
$ ipfs files read /test
hello

go-ipfs:

$ echo "hello" | ipfs files write /test
$ ipfs files read /test
hello
$ echo "hello" | ipfs files write /test --offset=6
$ ipfs files read /test
hello
hello

files.ls ordering incorrect

Currently in js-ipfs master:

1) interface-ipfs-core over ipfs-api tests .files.ls should ls directory:

    Uncaught AssertionError: expected [ Array(2) ] to deeply equal [ Array(2) ]
    + expected - actual

     [
       {
         "hash": ""
    -    "name": "b"
    +    "name": "lv1"
         "size": 0
         "type": 0
       }
       {
         "hash": ""
    -    "name": "lv1"
    +    "name": "b"
         "size": 0
         "type": 0
       }
     ]
    
    at ipfs.files.ls (node_modules/interface-ipfs-core/js/src/files/ls.js:56:27)
    at transform (node_modules/ipfs-api/src/files/ls.js:8:3)
    at send (node_modules/ipfs-api/src/utils/send-request.js:217:7)
    at f (node_modules/once/once.js:25:25)
    at streamToValue (node_modules/ipfs-api/src/utils/stream-to-json-value.js:30:5)
    at concat (node_modules/ipfs-api/src/utils/stream-to-value.js:12:22)
    at ConcatStream.<anonymous> (node_modules/concat-stream/index.js:37:43)
    at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:630:14)
    at afterWrite (node_modules/readable-stream/lib/_stream_writable.js:492:3)
    at _combinedTickCallback (internal/process/next_tick.js:144:20)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

2) interface-ipfs-core over ipfs-api tests .files.ls should ls -l directory:

    Uncaught AssertionError: expected [ Array(2) ] to deeply equal [ Array(2) ]
    + expected - actual

     [
       {
    +    "hash": "QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"
    +    "name": "lv1"
    +    "size": 0
    +    "type": 1
    +  }
    +  {
         "hash": "QmcZojhwragQr5qhTeFAmELik623Z21e3jBTpJXoQ9si1T"
         "name": "b"
         "size": 13
         "type": 0
       }
    -  {
    -    "hash": "QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn"
    -    "name": "lv1"
    -    "size": 0
    -    "type": 1
    -  }
     ]
    
    at ipfs.files.ls (node_modules/interface-ipfs-core/js/src/files/ls.js:76:27)
    at transform (node_modules/ipfs-api/src/files/ls.js:8:3)
    at send (node_modules/ipfs-api/src/utils/send-request.js:217:7)
    at f (node_modules/once/once.js:25:25)
    at streamToValue (node_modules/ipfs-api/src/utils/stream-to-json-value.js:30:5)
    at concat (node_modules/ipfs-api/src/utils/stream-to-value.js:12:22)
    at ConcatStream.<anonymous> (node_modules/concat-stream/index.js:37:43)
    at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:630:14)
    at afterWrite (node_modules/readable-stream/lib/_stream_writable.js:492:3)
    at _combinedTickCallback (internal/process/next_tick.js:144:20)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

read with offset/count returns buffer bigger than count

Just trying to update ipfs-fuse to work with js-ipfs now that it has MFS (https://github.com/tableflip/ipfs-fuse/tree/feat/embedded-ipfs). I'm getting this error:

  ipfs-fuse:read { path: '/gif/5097.gif', fd: 42, len: 4096, pos: 0 } +582ms
fuse: read too many bytes
   read[42] 15234 bytes from 0

Empty hash, zero size and zero type for directory listing

Test script:

const IPFS = require('ipfs')
const Path = require('path')
const Os = require('os')

const ipfs = new IPFS({ repo: Path.join(Os.tmpdir(), `${Date.now()}`) })

ipfs.on('ready', async () => {
 const files = Array.from([1, 2, 3], i => ({
   path: `test-file-${Date.now()}-${i}`,
   content: Buffer.from(`data ${Date.now()} ${i}`)
 }))

 const res = await ipfs.files.add(files, { wrapWithDirectory: true })

 console.log('Added files:')
 console.log(res)

 const { hash } = res[res.length - 1]

 const ipfsPath = `/ipfs/${hash}`
 console.log(`Copying from ${ipfsPath}`)

 const mfsPath = `/test-dir-${Date.now()}`
 console.log(`Copying to ${mfsPath}`)

 await ipfs.files.cp(ipfsPath, mfsPath)

 console.log(`Directory listing of ${mfsPath}:`)
 console.log(await ipfs.files.ls(mfsPath))
})

Output:

$ node ls-bug.js
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Swarm listening on /ip4/127.0.0.1/tcp/4002/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Swarm listening on /ip4/192.168.1.85/tcp/4002/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Added files:
[ { path: 'test-file-1540199056673-3',
   hash: 'QmRDi9soVyiuwbV5t51v7KnmGeY1tNGSLqeXsnFVupf7fS',
   size: 28 },
 { path: 'test-file-1540199056673-2',
   hash: 'QmV7nqHzxvjwcCD8Qda9DpwJ5wwLEWpZCTf48gJswiUVbD',
   size: 28 },
 { path: 'test-file-1540199056673-1',
   hash: 'QmVd2S29ciJvwZutbNQKrnnBfrUxwztWMDPCoVUjoDZFud',
   size: 28 },
 { path: '',
   hash: 'QmQCdSt7BNokYuGL5prjhcv22eSSumqza3vN7YWWDFHXAX',
   size: 289 } ]
Copying from /ipfs/QmQCdSt7BNokYuGL5prjhcv22eSSumqza3vN7YWWDFHXAX
Copying to /test-dir-1540199056875
Directory listing of /test-dir-1540199056875:
[ { name: 'test-file-1540199056673-3', type: 0, hash: '', size: 0 },
 { name: 'test-file-1540199056673-2', type: 0, hash: '', size: 0 },
 { name: 'test-file-1540199056673-1', type: 0, hash: '', size: 0 } ]

All tests fail on master

Empty hash, zero size and zero type when ls directory

Test script:

const IPFS = require('ipfs')
const Path = require('path')
const Os = require('os')

const ipfs = new IPFS({ repo: Path.join(Os.tmpdir(), `${Date.now()}`) })

ipfs.on('ready', async () => {
  const files = Array.from([1, 2, 3], i => ({
    path: `test-file-${Date.now()}-${i}`,
    content: Buffer.from(`data ${Date.now()} ${i}`)
  }))

  const res = await ipfs.files.add(files, { wrapWithDirectory: true })

  console.log('Added files:')
  console.log(res)

  const { hash } = res[res.length - 1]

  const ipfsPath = `/ipfs/${hash}`
  console.log(`Copying from ${ipfsPath}`)

  const mfsPath = `/test-dir-${Date.now()}`
  console.log(`Copying to ${mfsPath}`)

  await ipfs.files.cp(ipfsPath, mfsPath)

  console.log(`Directory listing of ${mfsPath}:`)
  console.log(await ipfs.files.ls(mfsPath))
})

Output:

$ node ls-bug.js 
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Swarm listening on /ip4/127.0.0.1/tcp/4002/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Swarm listening on /ip4/192.168.1.85/tcp/4002/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Added files:
[ { path: 'test-file-1540199056673-3',
    hash: 'QmRDi9soVyiuwbV5t51v7KnmGeY1tNGSLqeXsnFVupf7fS',
    size: 28 },
  { path: 'test-file-1540199056673-2',
    hash: 'QmV7nqHzxvjwcCD8Qda9DpwJ5wwLEWpZCTf48gJswiUVbD',
    size: 28 },
  { path: 'test-file-1540199056673-1',
    hash: 'QmVd2S29ciJvwZutbNQKrnnBfrUxwztWMDPCoVUjoDZFud',
    size: 28 },
  { path: '',
    hash: 'QmQCdSt7BNokYuGL5prjhcv22eSSumqza3vN7YWWDFHXAX',
    size: 289 } ]
Copying from /ipfs/QmQCdSt7BNokYuGL5prjhcv22eSSumqza3vN7YWWDFHXAX
Copying to /test-dir-1540199056875
Directory listing of /test-dir-1540199056875:
[ { name: 'test-file-1540199056673-3', type: 0, hash: '', size: 0 },
  { name: 'test-file-1540199056673-2', type: 0, hash: '', size: 0 },
  { name: 'test-file-1540199056673-1', type: 0, hash: '', size: 0 } ]

Problems read/writing files in sharded directories

Sometimes files that have been deleted are still present and other times files that have been created do not appear.

This only affects HAMT-sharded directories where the previous file manipulation results in new sub-shards a being created a few levels deep. It does not affect top-level subshards.

Empty hash, zero type and zero size for ls files in directory

Test script:

const IPFS = require('ipfs')
const Path = require('path')
const Os = require('os')

const ipfs = new IPFS({ repo: Path.join(Os.tmpdir(), `${Date.now()}`) })

ipfs.on('ready', async () => {
  const files = Array.from([1, 2, 3], i => ({
    path: `test-file-${Date.now()}-${i}`,
    content: Buffer.from(`data ${Date.now()} ${i}`)
  }))

  const res = await ipfs.files.add(files, { wrapWithDirectory: true })

  console.log('Added files:')
  console.log(res)

  const { hash } = res[res.length - 1]

  const ipfsPath = `/ipfs/${hash}`
  console.log(`Copying from ${ipfsPath}`)

  const mfsPath = `/test-dir-${Date.now()}`
  console.log(`Copying to ${mfsPath}`)

  await ipfs.files.cp(ipfsPath, mfsPath)

  console.log(`Directory listing of ${mfsPath}:`)
  console.log(await ipfs.files.ls(mfsPath))
})

Output:

$ node ls-bug.js 
Swarm listening on /ip4/127.0.0.1/tcp/4003/ws/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Swarm listening on /ip4/127.0.0.1/tcp/4002/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Swarm listening on /ip4/192.168.1.85/tcp/4002/ipfs/QmXk7xmWA4yShKUwXN7JZXatFX6ATRVYHzkL52tngTxhiW
Added files:
[ { path: 'test-file-1540199056673-3',
    hash: 'QmRDi9soVyiuwbV5t51v7KnmGeY1tNGSLqeXsnFVupf7fS',
    size: 28 },
  { path: 'test-file-1540199056673-2',
    hash: 'QmV7nqHzxvjwcCD8Qda9DpwJ5wwLEWpZCTf48gJswiUVbD',
    size: 28 },
  { path: 'test-file-1540199056673-1',
    hash: 'QmVd2S29ciJvwZutbNQKrnnBfrUxwztWMDPCoVUjoDZFud',
    size: 28 },
  { path: '',
    hash: 'QmQCdSt7BNokYuGL5prjhcv22eSSumqza3vN7YWWDFHXAX',
    size: 289 } ]
Copying from /ipfs/QmQCdSt7BNokYuGL5prjhcv22eSSumqza3vN7YWWDFHXAX
Copying to /test-dir-1540199056875
Directory listing of /test-dir-1540199056875:
[ { name: 'test-file-1540199056673-3', type: 0, hash: '', size: 0 },
  { name: 'test-file-1540199056673-2', type: 0, hash: '', size: 0 },
  { name: 'test-file-1540199056673-1', type: 0, hash: '', size: 0 } ]

limit-stream-bytes missing return?

https://github.com/ipfs/js-ipfs-mfs/blob/505cd6d205be8dbb0f18d41a9998de3d26ad44eb/src/core/utils/limit-stream-bytes.js#L9-L11

Is this missing a return statement?

Remove `format` option from mfs commands

Following on from ipfs-inactive/interface-js-ipfs-core#575

Remove dependency on IPFS

We pass in and instance of js-ipfs to all core modules in this repo but all MFS operations could just use js-ipld and js-interface-datastore directly instead.

We should refactor the core to accept these as arguments instead and so remove the circular dependency of js-ipfs -> js-ipfs-mfs -> js-ipfs.