GithubHelp home page GithubHelp logo

ipfs / js-ipfsd-ctl Goto Github PK

View Code? Open in Web Editor NEW
151.0 36.0 67.0 3.33 MB

Control an IPFS daemon (go-ipfs or js-ipfs) using JavaScript!

Home Page: https://ipfs.github.io/js-ipfsd-ctl

License: Other

JavaScript 1.25% TypeScript 98.75%
ipfs daemon ctl go-ipfs js-ipfs

js-ipfsd-ctl's Introduction

ipfsd-ctl

ipfs.tech Discuss codecov CI

Spawn IPFS Daemons, Kubo or...

About

This module allows you to spawn long-lived IPFS implementations from any JS environment and interact with the as is they were in the local process.

It is designed mostly for testing interoperability and is not suitable for production use.

Spawning a single noder: createNode

Example - Spawning a Kubo node

import { createNode } from 'ipfsd-ctl'
import { path } from 'kubo'
import { create } from 'kubo-rpc-client'

const node = await createNode({
  type: 'kubo',
  rpc: create,
  bin: path()
})

console.info(await node.api.id())

Manage multiple nodes: createFactory

Use a factory to spawn multiple nodes based on some common template.

Example - Spawning multiple Kubo nodes

import { createFactory } from 'ipfsd-ctl'
import { path } from 'kubo'
import { create } from 'kubo-rpc-client'

const factory = createFactory({
  type: 'kubo',
  rpc: create,
  bin: path()
})

const node1 = await factory.spawn()
const node2 = await factory.spawn()
//...etc

// later stop all nodes
await factory.clean()

Override config based on implementation type

createFactory takes a second argument that can be used to pass default options to an implementation based on the type field.

import { createFactory } from 'ipfsd-ctl'
import { path } from 'kubo'
import { create } from 'kubo-rpc-client'

const factory = createFactory({
  type: 'kubo',
  test: true
}, {
  otherImpl: {
    //...other impl args
  }
})

const kuboNode = await factory.spawn()
const otherImplNode = await factory.spawn({
  type: 'otherImpl'
})

Spawning nodes from browsers

To spawn nodes from browsers, first start an ipfsd-ctl server from node.js and make the address known to the browser (the default way is to set process.env.IPFSD_CTL_SERVER in your bundle):

Example - Create server

In node.js:

// Start a remote disposable node, and get access to the api
// print the node id, and stop the temporary daemon

import { createServer } from 'ipfsd-ctl'

const port = 9090
const server = createServer(port, {
  type: 'kubo',
  test: true
}, {
   // overrides
})
await server.start()

In a browser:

import { createFactory } from 'ipfsd-ctl'

const factory = createFactory({
  // or you can set process.env.IPFSD_CTL_SERVER to http://localhost:9090
  endpoint: `http://localhost:${port}`
})

const node = await factory.createNode({
  type: 'kubo'
})
console.info(await node.api.id())

Disposable vs non Disposable nodes

ipfsd-ctl can spawn disposable and non-disposable nodes.

  • disposable- Disposable nodes are useful for tests or other temporary use cases, they create a temporary repo which is deleted automatically when the node is stopped
  • non-disposable - Disposable nodes will not delete their repo when stopped

Install

$ npm i ipfsd-ctl

Browser <script> tag

Loading this module through a script tag will make it's exports available as IpfsdCtl in the global namespace.

<script src="https://unpkg.com/ipfsd-ctl/dist/index.min.js"></script>

API Docs

License

Licensed under either of

Contribute

Contributions welcome! Please check out the issues.

Also see our contributing document for more information on how we work, and about contributing in general.

Please be aware that all interactions related to this repo are subject to the IPFS Code of Conduct.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

js-ipfsd-ctl's People

Contributors

achingbrain avatar alanshaw avatar bcomnes avatar daviddias avatar dependabot-preview[bot] avatar dependabot[bot] avatar dignifiedquire avatar dryajov avatar edsilv avatar fraubienenstich avatar greenkeeperio-bot avatar haadcode avatar hackergrrl avatar hugomrdias avatar ipfs-mgmt-read-write[bot] avatar jbenet avatar krl avatar lidel avatar pleerock avatar ralphtheninja avatar richardschneider avatar scott113341 avatar semantic-release-bot avatar sgtpooki avatar vasco-santos avatar victorb avatar vmx avatar web-flow avatar web3-bot avatar wraithgar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

js-ipfsd-ctl's Issues

Passphrase when initing

Need to allow an optional passphrase when initing.

Add pass: null to the initOpts. If not null, then produce jsipfs init --pass ... -b 2048.

Make sure that this is also done for in-proc deamons.

Future of ipfd-ctl (for consideration)

Taken from: ipfs/js-ipfs#556 (comment)


Satelite modules ipfsd-ctl vs ipfs-daemon vs ipfs-factory

We need a thing that lets us spawn nodes (js-ipfs and go-ipfs) easily, so that apps like orbit, our tests and benchmarks can switch between different combinations (js-ipfs, js-ipfs daemon + js-ipfs-api, go-ipfs + js-ipfs-api) without too much configuration.

This endeavor doesn't have to be part of "Improving Init", but it would be definitely useful to a lot of contributors, users and even for our testing.

Currently, we have:

ipfsd-ctl - spawns go-ipfs daemons and returns a js-ipfs-api instance to talk with them
ipfs-factory (not a module, just a tool within this repo) - spawns daemons from Node.js and the Browser.
ipfs-daemon - https://github.com/haadcode/ipfs-daemon
ipfsd-ctl is the one that has more adoption.

Passphrase for in-proc deamon

The new --pass ... option should be set for the in-proc deamon. For example

const df = DaemonFactory.create({ type: 'proc', exec: IPFS })
const options = {
  args: ['--pass ipfs-is-awesome-software']
}
df.spawn(options, (err, _ipfsd) => ...)

Needed by ipfs/js-ipfs#1133

Double Kill

go-ipfs is notorious about not shutting down after receiving a kill signal, I think we should probably put a timeout that will trigger the send kill signal to force it to shutdown after a timeout.

Simple example in README.md doesn't work

When trying to run the disposable daemon example in the README.md, I get this:

shinichi@valentine ~/source/test $ node index.js
{ ID: 'QmYTcwsNtCbCBGUTSa671oCrD8XXnVfhGjyHzBhzncdDLs',
  PublicKey: 'CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxZtTfFEBEfVKPJoWnwZVU/vkjmjDq+5njoC69lX1+xHq+xB/uaX/M/NzYwywFRhwwSIFhYfOswGxcZzsmqt+ZZ+2WJZfjClTNR3VG+vvYhO2EdWvenszecizVtPVnRZ6uT4poECaZIAEHYLvCnMZ6f4D73qn2DHHtSv0upKxClupirIlm633sm+Okly05NW8KtsVpqQT1MSBVZHHRg17tp2eI1B45FxSQYQwTPUFd/j52LX6RSxJkPx11dAeDGjzEOHjzPtxvuUrynabiLW+AONx6jJf7cKWFv8GoFAhFiCm5gb4N/AND6O82svHS8fzGchj4aSWSi38Cb4jm8sU1AgMBAAE=',
  Addresses: 
   [ '/ip4/127.0.0.1/tcp/33400/ipfs/QmYTcwsNtCbCBGUTSa671oCrD8XXnVfhGjyHzBhzncdDLs',
     '/ip4/192.168.1.109/tcp/33400/ipfs/QmYTcwsNtCbCBGUTSa671oCrD8XXnVfhGjyHzBhzncdDLs',
     '/ip4/192.168.20.1/tcp/33400/ipfs/QmYTcwsNtCbCBGUTSa671oCrD8XXnVfhGjyHzBhzncdDLs',
     '/ip4/192.168.238.1/tcp/33400/ipfs/QmYTcwsNtCbCBGUTSa671oCrD8XXnVfhGjyHzBhzncdDLs',
     '/ip4/92.206.50.145/tcp/44173/ipfs/QmYTcwsNtCbCBGUTSa671oCrD8XXnVfhGjyHzBhzncdDLs' ],
  AgentVersion: 'go-ipfs/0.3.11',
  ProtocolVersion: 'ipfs/0.1.0' }
node.js:791
        throw new TypeError('invalid pid');
        ^

TypeError: invalid pid
    at process.kill (node.js:791:15)
    at /home/shinichi/source/test/index.js:7:13
    at finish (/home/shinichi/source/test/node_modules/wreck/lib/index.js:319:16)
    at wrapped (/home/shinichi/source/test/node_modules/wreck/node_modules/hoek/lib/index.js:867:20)
    at onReaderFinish (/home/shinichi/source/test/node_modules/wreck/lib/index.js:365:16)
    at g (events.js:273:16)
    at emitNone (events.js:85:20)
    at emit (events.js:179:7)
    at finishMaybe (_stream_writable.js:468:14)
    at endWritable (_stream_writable.js:478:3

Expose the API daemon's host/port

It would be great if the resultant host and port were exposed by the resultant disposableApi object, so that other modules using ipfs-api could make use of the disposable daemon.

Error npm test fills tmp dir

while running the tests over and over again the following error appears more often than it should (MacOS)

Uncaught AssertionError: expected [Error: Error: ipfs configuration file already exists!
Reinitializing would overwrite your keys.

some tests use a temporary dir

const dir = os.tmpdir() + `/tmp-${Math.ceil(Math.random() * 100)}`

This temporary dir gets quite full and seems to be the cause for the Error: ipfs configuration file already exists

ls /var/folders/vc/g0gp6cj15vg3v62bwyn3pdbc0000gp/T
ipfs_00923925270015502				tmp--100
ipfs_013408338790235286				tmp--12
ipfs_014171347120929045				tmp--14
ipfs_027350138789616674				tmp--16
ipfs_029120182442286202				tmp--18
ipfs_03184452858010567				tmp--20
ipfs_05955467903437994				tmp--22
ipfs_06095803926897725				tmp--23
ipfs_06740066115484877				tmp--24
ipfs_06795640497725586				tmp--27
ipfs_06993119814338011				tmp--28
ipfs_07298509408059695				tmp--31
ipfs_07456547692741511				tmp--33
ipfs_08634351266617024				tmp--35
ipfs_0888132984196397				tmp--43
ipfs_09132801630670873				tmp--44
ipfs_09625399003146295				tmp--55
ipfs_10890433268035649				tmp--56
ipfs_11631035864891426				tmp--64
ipfs_11924458870556665				tmp--65
ipfs_1258814504860306				tmp--66
ipfs_15681934416640764				tmp--70
ipfs_17131383315111948				tmp--72
ipfs_18804510593267376				tmp--75
ipfs_1948177500601116				tmp--77
ipfs_2147503549547347				tmp--78
ipfs_2232495460229984				tmp--83
ipfs_22882884304077367				tmp--87
ipfs_23387611613097037				tmp--88
ipfs_23837606108405018				tmp--9
ipfs_23894842144610107				tmp--94
ipfs_24736804932801193				tmp--99
ipfs_25719110414888413				tmp-15
ipfs_26157466717751476				tmp-17
ipfs_2641200796085763				tmp-19
ipfs_2773751683075154				tmp-20
ipfs_2803329334680442				tmp-21
ipfs_2936031290305703				tmp-26
ipfs_29968229753631714				tmp-29
ipfs_30119929804576495				tmp-3
ipfs_30441192907999914				tmp-31
ipfs_31623216900189366				tmp-32
ipfs_31702761525505974				tmp-35
ipfs_3172711048314747				tmp-37
ipfs_3402164871532263				tmp-38
ipfs_34512968956193846				tmp-40
ipfs_34583661573929714				tmp-41
ipfs_34953624994277366				tmp-42
ipfs_3519190337624385				tmp-44
ipfs_35623569175824343				tmp-45
ipfs_36390902331498554				tmp-47
ipfs_36773338672719014				tmp-49
ipfs_3739060917992727				tmp-51
ipfs_3877153073043258				tmp-55
ipfs_38958602596407643				tmp-59
ipfs_38985268222695146				tmp-66
ipfs_41038052270594316				tmp-67
ipfs_42283424663901337				tmp-68
ipfs_43023788944091046				tmp-70
ipfs_4313452800835389				tmp-71
ipfs_43614948601380754				tmp-74
ipfs_44713233089427984				tmp-76
ipfs_4532905602398307				tmp-80
ipfs_47697856610365497				tmp-84
ipfs_484533320083242				tmp-85
ipfs_4988948069120409				tmp-87
ipfs_5115775686403132				tmp-90
ipfs_5174758984852121				tmp-91
ipfs_5244204515432622				tmp-94
ipfs_5258167387515746				tmp-96
ipfs_5331830162285807				tmp-99

(go-ipfs 0.4.5-pre1) Error: ipfs configuration file already exists!

Running the tests with go-ipfs 0.4.5-pre1 binary, the following error gets thrown:

     Error: non-zero exit code 1

            while running: /Users/haad/code/js-ipfsd-ctl-github/node_modules/go-ipfs-dep/go-ipfs/ipfs init -b 2048


            Error: ipfs configuration file already exists!
Reinitializing would overwrite your keys.

I believe something was changed in 0.4.5 re how the init is handled (this is what I was trying to explain on IRC the other day). @whyrusleeping @Kubuxu would you be able to tell us what was changed? Was '--force' removed? Is this behaviour intentional?

If this is intentional, I wonder how we should handle / fix it...

(With 0.4.4 the tests pass)

cc @ipfs/javascript-team

Node inspector doesn't work: address already in use

Hello guys, thank you for your amazing work.

I use ipfsd-ctl for spawning IPFS instance in mocha test in a very similar way as in OrbitDb tests.
It's cool but it doesn't work with node inspector:

node_modules/.bin/mocha --inspect test/**/*.js

Debugger listening on ws://127.0.0.1:9229/7c0ad2d6-146b-4ae1-bf2c-7e9873c18304
For help see https://nodejs.org/en/docs/inspector


  Storage
    1) "before all" hook: before


  0 passing (33ms)
  1 failing

  1) Storage
       "before all" hook: before:
     Error: non-zero exit code 12

            while running: /Users/ivanshumkov/Projects/dashevo/dashdrive/node_modules/go-ipfs-dep/go-ipfs/ipfs init -b 2048


            Starting inspector on 127.0.0.1:9229 failed: address already in use

      at Stream.listeners.done.once (node_modules/ipfsd-ctl/src/exec.js:39:11)
      at Stream.f (node_modules/once/once.js:25:25)
      at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)

ipfs.id is not a function

There isn't much info on ipfsd.local but from this example it seemed like this should work.

const ipfsd = require('ipfsd-ctl')

ipfsd.local((err, ipfs) => {
    if (err) { throw err }

    ipfs.id(function (err, id) {
        if (err) { throw err }
        console.log(id);
    });
});

Error:

Uncaught TypeError: ipfs.id is not a function
    at ipfsd.local (file:///home/me/prj/web/viddist/index.js:8:10)
    at process.nextTick (/home/me/prj/web/viddist/node_modules/ipfsd-ctl/src/index.js:57:7)
    at _combinedTickCallback (internal/process/next_tick.js:73:7)
    at process._tickDomainCallback (internal/process/next_tick.js:128:9)

Had the same result with and without IPFS_EXEC set. Also tried having a regular go-ipfs node running and not.

{"diskinfo":{"free_space":108446048256,"fstype":"61267","total_space":95631613952},"environment":{"GOPATH":"/home/me/prj/go","IPFS_PATH":""},"ipfs_commit":"c5d362e3f","ipfs_version":"0.4.11-dev","memory":{"swap":0,"virt":227436000},"net":{"interface_addresses":["/ip4/127.0.0.1","/ip4/10.2.202.244","/ip6/::1","/ip6/fe80::80ca:2835:20da:f537"],"online":false},"runtime":{"arch":"amd64","compiler":"gc","gomaxprocs":4,"numcpu":4,"numgoroutines":6,"os":"linux","version":"go1.8.3"}}

Latest version of `local` is broken

I've just upgraded station, but it seems that ipfsd.local and the subsequent node.startDaemon are broken. When running on an already initialized state node.startDaemon just hangs without ever calling back

Connect to already running daemon

After talking to @dryajov, we noticed that ipfsd-ctl doesn't connect to already running daemons. This code:

DaemonFactory.create().spawn({
  repoPath: "/a/beautiful/path",
  disposable: false,
  init: false,
  start: false
}, (err, node) => { console.log(node.started) })

Should print true if the there was a daemon on that repo and node.api should be populated. 😄

Running JS on Windows

A few of our tests (see block.js) run JS code like this

const df = DaemonFactory.create({ exec: 'src/cli/bin.js' })

On Windows it fails with

>aegir test -f test\http-api\interface\block.js
Test Node.js


  .block
    1) "before all" hook


  0 passing (766ms)
  1 failing

  1) .block
       "before all" hook:
     Uncaught expected [Error: internal/child_process.js:340
    throw errnoException(err, 'spawn');
    ^

Error: spawn UNKNOWN
    at _errnoException (util.js:1031:13)
    at ChildProcess.spawn (internal/child_process.js:340:11)
    at exports.spawn (child_process.js:499:9)
    at module.exports (C:\Users\Owner\Documents\GitHub\js-ipfs\node_modules\comandante\index.js:11:14)
    at Object.<anonymous> (C:\Users\Owner\Documents\GitHub\js-ipfs\node_modules\subcomandante\subcom:23:13)
    at Module._compile (module.js:641:30)
    at Object.Module._extensions..js (module.js:652:10)
    at Module.load (module.js:560:32)
    at tryModuleLoad (module.js:503:12)
    at Function.Module._load (module.js:495:3)
] to not exist
  AssertionError: expected [Error: internal/child_process.js:340
      throw errnoException(err, 'spawn');
      ^

  Error: spawn UNKNOWN
      at _errnoException (util.js:1031:13)
      at ChildProcess.spawn (internal/child_process.js:340:11)
      at exports.spawn (child_process.js:499:9)
      at module.exports (C:\Users\Owner\Documents\GitHub\js-ipfs\node_modules\comandante\index.js:11:14)
      at Object.<anonymous> (C:\Users\Owner\Documents\GitHub\js-ipfs\node_modules\subcomandante\subcom:23:13)
  ] to not exist
      at factory.spawnNode (node_modules\interface-ipfs-core\src\block.js:32:30)
      at df.spawn (test\http-api\interface\block.js:17:20)
      at waterfall (node_modules\ipfsd-ctl\src\daemon-ctrl.js:132:16)
      at node_modules\async\internal\once.js:12:16
      at next (node_modules\async\waterfall.js:21:29)
      at node_modules\async\internal\onlyOnce.js:12:16
      at run (node_modules\ipfsd-ctl\src\daemon-node.js:128:16)
      at f (node_modules\once\once.js:25:25)
      at Object.done (node_modules\ipfsd-ctl\src\exec.js:23:18)
      at Stream.listeners.done.once (node_modules\ipfsd-ctl\src\exec.js:45:18)
      at Stream.f (node_modules\once\once.js:25:25)
      at Socket.reemit (node_modules\duplexer\index.js:70:25)
      at Pipe._handle.close [as _onclose] (net.js:558:12)

Fragmentation

TIL that there is a @haad/ipfsd-ctl. I don't understand why there is a need to create such a module, but it seems it has been going for a while, making it more than an experiment, but an actual fork/module available in NPM.

I very much disagree with this approach because it is clearly fragmenting the ecosystem set by creating yet another module that is not being tracked as a PR and doesn't show any future where that will happen, making us maintain 2 modules instead of one.

When the challenge on ipfs/js-ipfs#556 was launched, the proposal was exactly to reduce fragmentation and to make the launch of IPFS nodes more easily, but what I see that got created was another module, ipfs-daemon, which has its set of issues, which uses a module that is a fork of this one, namely @haad/ipfsd-ctl.

@haadcode can we converge all of these, it really doesn't help. Others are invited to provide their feedback, I might also be missing the value and apologize in advance if I'm doing so.

Cant' find go-ipfs-dep module

index.js (copy-paste from readme.md):

const IPFSFactory = require('ipfsd-ctl')
const f = IPFSFactory.create()

f.spawn(function (err, ipfsd) {
  if (err) { throw err }
  ipfsd.api.id(function (err, id) {
    if (err) { throw err }
    console.log(id)
    ipfsd.stop()
  })
})

Running:

$ node index.js
/home/user/projects/victorbjelkholm/asciinema-ipfs/node_modules/ipfsd-ctl/src/utils/find-ipfs-executable.js:34
  throw new Error('Cannot find the IPFS executable')
  ^

Error: Cannot find the IPFS executable
    at module.exports (/home/user/projects/victorbjelkholm/asciinema-ipfs/node_modules/ipfsd-ctl/src/utils/find-ipfs-executable.js:34:9)
    at new Daemon (/home/user/projects/victorbjelkholm/asciinema-ipfs/node_modules/ipfsd-ctl/src/ipfsd-daemon.js:53:60)
    at FactoryDaemon.spawn (/home/user/projects/victorbjelkholm/asciinema-ipfs/node_modules/ipfsd-ctl/src/factory-daemon.js:130:18)
    at Object.<anonymous> (/home/user/projects/victorbjelkholm/asciinema-ipfs/index.js:4:3)
    at Module._compile (module.js:660:30)
    at Object.Module._extensions..js (module.js:671:10)
    at Module.load (module.js:573:32)
    at tryModuleLoad (module.js:513:12)
    at Function.Module._load (module.js:505:3)
    at Function.Module.runMain (module.js:701:10)

Install with yarn:
image

Install with npm:
image

{ npm: '5.6.0',
  ares: '1.13.0',
  cldr: '32.0',
  http_parser: '2.7.0',
  icu: '60.1',
  modules: '59',
  napi: '2',
  nghttp2: '1.29.0',
  node: '9.5.0',
  openssl: '1.0.2n',
  tz: '2017c',
  unicode: '10.0',
  uv: '1.19.1',
  v8: '6.2.414.46-node.18',
  zlib: '1.2.11' }

Here is a commit that moves go-ipfs-dep from dependencies to devDependencies, which might have something to do with it: 1cfbd08#diff-b9cfc7f2cdf78a7f4b91a753d10865a2

Browser tests failing on MacOS

What is port 9999?

See build https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fjs-ipfsd-ctl/detail/graceful-stop/1/pipeline/14

Test Browser
listen EADDRINUSE 127.0.0.1:9999
Error: listen EADDRINUSE 127.0.0.1:9999
    at Object._errnoException (util.js:1031:13)
    at _exceptionWithHostPort (util.js:1052:20)
    at Server.setupListenHandle [as _listen2] (net.js:1367:14)
    at listenInCluster (net.js:1408:12)
    at GetAddrInfoReqWrap.doListen [as callback] (net.js:1523:7)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:102:10)
error Command failed with exit code 1.

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

CI

Let's set up circle CI and/or Travis.

Tests fail on my machine currently btw.

`go-ipfs` is required from the top-level module

When the ipfsd-ctl package is installed and required by another module, the following error is given:

/Users/mappum/Projects/inpm/node_modules/ipfsd-ctl/index.js:13
var IPFS_EXEC = path.join(requireResolve('go-ipfs').pkg.root, '/bin/ipfs')
                                                   ^

TypeError: Cannot read property 'pkg' of null
    at Object.<anonymous> (/Users/mappum/Projects/inpm/node_modules/ipfsd-ctl/index.js:13:52)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/Users/mappum/Projects/inpm/bin/main.js:11:16)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)

This happens because the go-ipfs dependency gets loaded as a direct dependency of the top-level module, not as a dependency of ipfsd-ctl. So even though npm install installs go-ipfs in node_modules/ipfsd-ctl/node_modules/go-ipfs, the ipfsd-ctl code tries to require it from node_modules/go-ipfs.

If this is actually the intended behavior, the README should tell the user to also install go-ipfs, and go-ipfs should be a peer dependency rather than a direct dependency.

Debug logging

If debug logging is enabled, via DEBUG environment variable, then all the js-ipfs tests fails. It appears that ipfsd-ctl does not like debug output.

> set DEBUG=*
> aegir test -t node -f test\http-api\interface\block

See the debug.txt

In proc node config not applied

I'm trying to enable EXPERIMENTAL.dht for my in-proc node:

df.spawn({ config: { EXPERIMENTAL: { dht: true } } }, /* ... */)

The DHT experiment is not enabled because the in process node does not use the config property to configure the node, it expects EXPERIMENTAL to be in the root of the options object:

this.opts.EXPERIMENTAL = defaults({}, opts.EXPERIMENTAL, {

Hardcoded IPFS_EXEC path not always correct

The current version of ipfsd-ctl hardcodes the following path:

`var IPFS_EXEC = __dirname + '/node_modules/.bin/ipfs'

This works fine when the module is at the root (e.g. git clone this repo and run npm test), but doesn't work when ipfsd-ctl is used as a dependency. In this IPFS_EXEC evaluates to

node_modules/ipfsd-ctl/node_modules/.bin/ipfs

however, if the root module depends on go-ipfs the proper path to the binary will actually be node_modules/.bin/ipfs.

I'm preparing a PR that will use require-resolve to get the proper module path and look into its /bin directory.

Adding js-ipfs support

NOTE: I don't intend to start a big discussion around this, as I think 1) is relatively low risk, 2) is pretty self evident. This is mostly a brain dump after taking an initial stab at it - but feedback is welcome as I want to make sure I don't miss something obvious either.

Here are some questions that have come up after starting this effort:

  1. ipfsd-ctl execs the Go executable and communicates with it over cli (this is natural since it's not a node proces)

    • In order to integrate node ipfs support, we have two options
      • Usejs-ipfs cli
        • The advantage is that we can reuse a lot (how much?) of it the current implementation
        • The disadvantage is that there are differences which could invalidate reusing the current implementation - I feel like until we dig in, we won't have a clear understanding
      • Use js-ips as a library
        • The advantage is that it's a lot simpler to interact with js-ipfs over the library interface
        • The disadvantage is that we implement it from scratch - tho we can pull/copy existing implementations of this from the various available out there
  2. using ipfsd-ctl now becomes context aware

    • if ran from within js-ipfs we must likely want the option to use that js-ipfs, this will allow us to test it against itself running as a standalone node.
    • if ran from any other project or standalone, just use the bundled js-ipfs

IRC discussion for background:

09:55 <@dryajov> daviddias: victorbjelkholm: you around? want to bounce a few ideas of you for the ipfsd-ctl stuff… 
09:55 <@victorbjelkholm> yup!
09:56 <@dryajov> basically, for the js daemon, do we want to spawn an external process a la Go? or use the ipfs library directly (basically reusing the factory stuff in js-ipfs)
09:57 <@dryajov> I think it comes down to, which interface is going to be more consisten/stable - cli or library?
09:58 <@dryajov> thats one thing… #2 is:
09:58 <@victorbjelkholm> #1:
09:59 <@victorbjelkholm> could spawn it directly in the process, as long as we expose the http-api so js-ipfs-api will be used for both js-ipfs and go-ipfs, so we can be 100% interface works the same
10:03 <@dryajov> hmm, your right… http-api will give us a consistent interface either way (I was thinking more from ipfsd-ctl internals perspective tho, but voising it out kinda makes the choise obvios :D  - library is way easier to use)
10:03 <@dryajov> your right, http-api will give us a consistent interface either way
10:03 <@dryajov> 👍
10:04 <@dryajov> #2: running inside js-ipfs vs running standalone
10:04 <@dryajov> if running from js-ipfs we want to use _that js-ipfs_ 
10:05 <@dryajov> if running from outside, we just require the bundled one?
10:06 <@dryajov> victorbjelkholm: does #2 make sense, I can elaborate if its too confusing - brain dump at this point
10:12 <kyledrake> hsanjuan let me know when you're ready for me to take a wrecking ball to ipfs-cluster. :)
10:13 <@victorbjelkholm> dryajov: hm, thinking about the internals, way easier to skip the library and spin up process, then it doesn't matter if it's go/js/rust/scheme
10:14 <@victorbjelkholm> then #2 would always run bundled one
10:14 <@victorbjelkholm> the daemon for each language should be as a optionalDependency as well, as you might only use go-ipfs for example
10:23 <@dryajov> victorbjelkholm: hehe… my dilema...
10:23 <@dryajov> for #1
10:23 <@dryajov> tho, we still have differences from go to js cmd and outputs
10:24 <@dryajov> so we can’t reuse %100, but it should be close enough 
10:24 <@victorbjelkholm> yeah, but the startup/shutdown sequence should be the same (ipfs init, ipfs daemon)
10:24 <@victorbjelkholm> the outputs should be standarized at one point as well (maybe we need ipfs-interface-cli)
10:28 <@dryajov> +1 for ipfs-interface-cli...

@victorbjelkholm @diasdavid

In proc nodes can not be spawned on top of existing repos

In-process js-ipfs instances are not aware of whether or not they have spawned on top of an existing repo. This basically breaks the node as it can neither .start or .init.

I believe this is the cause of an error mentioned in a TODO here:

// `Error: Not able to start from state: uninitalized`

I think a fix for this can largely mirror how a js-ipfs instance works if the repo exists.

Another in-proc node issue that might be good to group with this: #208

NPM install is failing on peer dependencies

C:\Users\Owner\Documents\GitHub\js-ipfsd-ctl>npm i
npm WARN [email protected] requires a peer of eslint-plugin-react@>=6.10.3 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

added 121 packages, removed 7 packages and updated 27 packages in 146.092s

C:\Users\Owner\Documents\GitHub\js-ipfsd-ctl>npm i [email protected]
npm ERR! code E404
npm ERR! 404 Not Found: [email protected]

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Owner\AppData\Roaming\npm-cache\_logs\2017-11-25T04_28_07_852Z-debug.log

C:\Users\Owner\Documents\GitHub\js-ipfsd-ctl>npm i [email protected]
npm WARN [email protected] requires a peer of eslint@^2.0.0 || ^3.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

+ [email protected]
added 129 packages in 59.481s

C:\Users\Owner\Documents\GitHub\js-ipfsd-ctl>npm  i [email protected]
npm WARN [email protected] requires a peer of eslint@^4.1.1 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of eslint@>=3.19.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of eslint@>=3.19.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of eslint@>=3.1.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] requires a peer of eslint@>=3.19.0 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

+ [email protected]
added 152 packages, removed 3 packages, updated 15 packages and moved 8 packages in 61.669s
``

ipfsd.api.name.publish fails with opaque error message.

OS: Sierra, 7.3.0 Darwin Kernel Version 17.3.0 X_86_64

The following code fails:


// ... daemon.spawn(ipfsd => ...

ipfsd.api.util.addFromFs(dist, { recursive: true }).then(filesAdded => {
       // Files add works. 
       !err && console.log('Files were added...')
      
      // Hash of directory to publish is valid
      const rootDirHash = filesAdded.pop().hash

      // Publishing fails :(
      ipfsd.api.name.publish(rootDirHash).then((err, result) => {
          !err && console.log(result)
      }).catch(err => {
          console.log(err)    
      })
})

With this error:

{ Error: Sorry, something went wrong, please retrace your steps.
    at parseError (/Users/MK/vue-ipfs/node_modules/ipfs-api/src/utils/send-request.js:16:17)
    at ClientRequest.res (/Users/MK/vue-ipfs/node_modules/ipfs-api/src/utils/send-request.js:40:14)
    at Object.onceWrapper (events.js:324:30)
    at emitOne (events.js:125:13)
    at ClientRequest.emit (events.js:221:7)
    at HTTPParser.parserOnIncomingClient (_http_client.js:550:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:117:23)
    at Socket.socketOnData (_http_client.js:439:20)
    at emitOne (events.js:125:13)
    at Socket.emit (events.js:221:7) code: 1 }

I'm trying to poke into the source, but not having any luck.
✌️

Doesn't kill daemon / sub-process

Previously, js-ipfsd-ctl killed all sub-processes upon process exit. However, with the recent refactoring, that doesn't seem to be the case anymore :/

If you run the tests, for example only the 'start daemon' ones (describe.only('startDaemon')), the tests will pass but if you then do ps -A|grep ipfs you will see that there are still ipfs processes running (3 if I counted correctly). I'm on OSX, running js-ipfsd-ctl from master.

This is a major issue.

@jbenet kept trying to say that this is exactly what we need to avoid, because the old code was crafted over time to handle this exact case. We need to put fixing this on highest priority because it basically effects every program that uses js-ipfsd-ctl, leaving processes hanging.

cc @dignifiedquire @diasdavid @victorbjelkholm

Map config object to CLI args?

The in-process node maps CLI args to config but the daemon node does not map config to args.

I was trying to create a daemon node and passing config to enable pubsub, but the node was not configured to enable pubsub.

Should it also map config object to CLI args? Do we need both config and args?

CI: random fail returns valid API and gateway addresses

Ci failes "sometimes" on Node 4 sometimes on Node 6

See the 2 builds from another PR (#153 should be unrelated to the test)
fails on Node 6 https://travis-ci.org/ipfs/js-ipfsd-ctl/builds/244451297
fails on Node 4 https://travis-ci.org/ipfs/js-ipfsd-ctl/builds/245233174

each time same test

  1 failing
  1) daemons startDaemon starts the daemon and returns valid API and gateway addresses:
     Uncaught AssertionError: expected null to be an instance of Multiaddr
      at async.waterfall (test/index.spec.js:450:52)
      at node_modules/async/dist/async.js:421:16
      at next (node_modules/async/dist/async.js:5278:29)
      at node_modules/async/dist/async.js:906:16
      at node.startDaemon (test/index.spec.js:443:54)
      at f (node_modules/once/once.js:25:25)
      at Socket.data (src/node.js:236:13)
      at readableAddChunk (_stream_readable.js:176:18)
      at Socket.Readable.push (_stream_readable.js:134:10)
      at Pipe.onread (net.js:547:20)

Weird behavior with disposable daemons

If I don't pass any options to spawn, it seems like it defaults disposable to true -- inspecting ipfsd in the callback shows:

[Node]  Node {
[Node]   opts:
[Node]    { type: 'go',
[Node]      disposable: true,
[Node]      start: true,
[Node]      init: true,
[Node]      config:
[Node]       { 'API.HTTPHeaders.Access-Control-Allow-Origin': [Array],
[Node]         'API.HTTPHeaders.Access-Control-Allow-Methods': [Array],
[Node]         'Addresses.Swarm': [Array],
[Node]         'Addresses.API': '/ip4/127.0.0.1/tcp/0',
[Node]         'Addresses.Gateway': '/ip4/127.0.0.1/tcp/0' },
[Node]      exec: undefined },
[Node]   path: '/var/folders/yw/6y_wjm41465dv6bkc86m50g00000gp/T/ipfs_2acdcf543fe5dd65dbdefb45eba8ea68',
[Node]   disposable: true,
...

If I explicitly set disposable to false, as in:

DaemonFactory
  .create({ type: "go" })
  .spawn({ disposable: false }, (err, ipfsd) => {
    if (err) { throw err; }

    console.log(ipfsd);

    app.listen(app.get("port"), () => {
      console.log("--------------------------------------------------------------");
      console.log(("|  App is running at http://localhost:%d in %s mode"), app.get("port"), app.get("env"));
      console.log("|  Press CTRL-C to stop");
      console.log("--------------------------------------------------------------\n");
    });
  });

... ipfsd will have disposable set to true but ipfsd.api will be null. Should it be? Anything weird I'm doing here? I want the daemon to run in a non-disposable way (i.e. I should be able to access documents created after the daemon shuts down). And I need that ipfs-api instance :)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.