GithubHelp home page GithubHelp logo

fuge's Introduction

fuge-logo

Fuge configuration has changed if you are upgrading to fuge V2.0 read this guide upgrading.md

Fuge - The Microservice Shell

npm version npm downloads Build Status Win Status Gitter

Fuge is a microservice shell and workbench. It provides an execution environment for developing microservice systems, eliminating shell hell and significantly reducing developer friction.

If you're using this module, and need help, you can:

Install

Use npm to install globally.

npm install -g fuge

A Micoservice shell?

Yup. Fuge is a shell environment that focuses on a specific system configuration, providing an emulation environment in which services can run in development that is close to a modern container production deployment.

Usage

In order to run your system via fuge, you need to create a YAML configuration file. Fuge also integrates with docker-compose. Full documentation on configuring fuge is available here.

Full example demonstration systems are available here: https://github.com/apparatus/fuge-examples.

Example Configuration

A simple configuration is provided below:

fuge_global:
  tail: true
  monitor: true
  monitor_excludes:
    - '**/node_modules/**'
    - '**/.git/**'
    - '*.log'
myservice:
  type: process
  path: ../myservice
  run: 'node index.js'
  ports:
    - myservice=8000
webapp:
  type: process
  path: ../webapp
  run: 'npm start'
  ports:
    - webapp=3000
mongo:
  image: mongo
  type: container
  ports:
    - mongo=27017:27017

This configuration defines a front end webapp process, a single microservice and a mongodb container. The system can be started by running:

$ fuge shell <path to config file>
fuge> start all

This will start the mongo container through the local docker API and also the webapp and myservice processes. Fuge will also inject configuration information into these processes as environment variables, tail the logs and watch the file system for changes, restarting the appropriate processes as you write code.

To fully explore the capabilites of Fuge, you are encouraged to run the example systems which can be found here: https://github.com/apparatus/fuge-examples.

Docker

Fuge is fully compatible with V1, V2 and V3 docker-compose formats. Fuge services can be specified entirely using docker-compose or entirely in fuge.yml or a mixture of anything in between.

Command Reference

Fuge can be started by running:

$ fuge shell <path to fuge config file>
fuge>

The fuge command shell supports the following commands

command action
help display a list of supported commands
ps list status of managed processes and containers
info [process] [full]` show information on a specific process. The full option will output the entire environment that will be injected into this process
start [process] / all start a process or all processes
stop [process] / all stop a process and any associated watcher or all processes
debug [process] start a process in debug mode (only supported for node.js processes)
apply [Shell command] run the specified shell command in the root of each service e.g. apply npm install, apply git status
zone display dns zone information for fuge internal DNS server - requires the dns_enabled setting in fuge configuration file
watch [process] / all turn on watching for a process or for all processes
unwatch [process] / all turn off watching for a process or for all processes
tail [process] / all tail output for a process or for all processes
untail [process] / all end tail output for a specific processes or for all processes
grep 'search string' [process] searches a process' log or all processes' logs
exit terminate all managed processes and exit the shell

Shell pass through

In addition to the above commands, Fuge will pass through all other commands to the underlying shell, for example:

fuge> ps aux | grep -i node

Will bypass the fuge ps command and execute the underlying ps command.

fuge> netstat -an | grep -i listen

Will be passed through to the underlying shell to display a lists of open listening port and so on.

Contributing

The apparatus team encourage open participation. If you feel you can help in any way, be it with documentation, examples, extra testing, or new features please get in touch.

License

Copyright the apparatus team 2016, Licensed under MIT.

fuge's People

Contributors

adrianrossouw avatar codisredding avatar coryvirok avatar dberesford avatar dottorblaster avatar felipebarroscruz avatar glentiki avatar jimthedev avatar jmatsushita avatar joebligblog avatar lucamaraschi avatar matt-oc avatar mcdonnelldean avatar mihaidma avatar mtharrison avatar nebez avatar pelger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fuge's Issues

BIG: ability to add services to a running configuration

This would be very awesome.

Typically when building a microservice system I leave existing services running and just add new ones for now functionality.

At the moment I have to bring whole fuge system down and restart to do this, which sort of defeats the purpose.

I'm guessing a dynamic reparse and compare against in-memory state should be enough?

windows support

  1. prompt-sync does fs.openSync('/dev/stdin', 'rs') - preliminary research suggests there is no /dev/stdin equivalent on windows (hopefully there's something..)
  2. on occasion we're concatenating paths instead of path.join

fuge system generate fails on node 0.10 in interactive mode on OS X

this works fine on 0.12+ but on 0.10 it throws an error,
this is due to prompt-sync attempts to interact with /dev/stdin
issue here: heapwolf/prompt-sync#8

$ node --version
v0.10.40
$ fuge gen sys
? System transport (http, redis, mixed):
fs.js:488
  var r = binding.read(fd, buffer, offset, length, position);
                  ^
Error: EAGAIN, resource temporarily unavailable
    at Object.fs.readSync (fs.js:488:19)
    at Object.prompt (/Users/davidclements/z/nearForm/fuge/fuge/node_modules/prompt-sync/index.js:54:15)
    at incite (/Users/davidclements/z/nearForm/fuge/fuge/node_modules/incite/index.js:11:20)
    at transportSelection (/Users/davidclements/z/nearForm/fuge/fuge/generator.js:86:21)
    at Object.generateSystem (/Users/davidclements/z/nearForm/fuge/fuge/generator.js:322:7)
    at Command.generateSystem [as func] (/Users/davidclements/z/nearForm/fuge/fuge/fuge.js:27:15)
    at Command.call (/Users/davidclements/z/nearForm/fuge/fuge/node_modules/commist/index.js:100:8)
    at Object.parse (/Users/davidclements/z/nearForm/fuge/fuge/node_modules/commist/index.js:58:19)
    at start (/Users/davidclements/z/nearForm/fuge/fuge/fuge.js:113:27)
    at Object.<anonymous> (/Users/davidclements/z/nearForm/fuge/fuge/fuge.js:119:3)

ordinal@^1.0.0 - no compatible version found

Hi,
Is this me, or package.json needs to be fixed in order to include the right ordinal version?

$ npm install
npm ERR! Darwin 15.3.0
npm ERR! argv "/usr/local/Cellar/node/5.7.0/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v5.7.0
npm ERR! npm  v3.6.0

npm ERR! No compatible version found: ordinal@^1.0.0
npm ERR! Valid install targets:
npm ERR! 0.0.4, 0.0.3, 0.0.2, 0.0.1, 0.0.0
npm ERR!
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/blaster/Desktop/projects/fuge/npm-debug.log

Udp port mapping not supported

Given:

baseswim:
  image: matteocollina/baseswim
  ports:
    - 7799:7799/udp
    - 3000:3000

This results in this error:

ReferenceError: udp is not defined
    at Object.<anonymous> (containers:43:24)
    at Module._compile (module.js:435:26)
    at requireFromString (/Users/matteo/.nvm/versions/node/v4.2.4/lib/node_modules/fuge/node_modules/xenotype/node_modules/xeno-compiler/compile.js:30:7)
    at Object.compile (/Users/matteo/.nvm/versions/node/v4.2.4/lib/node_modules/fuge/node_modules/xenotype/node_modules/xeno-compiler/compile.js:274:15)
    at Object.compile (/Users/matteo/.nvm/versions/node/v4.2.4/lib/node_modules/fuge/node_modules/xenotype/node_modules/xeno-compiler/main.js:23:9)
    at /Users/matteo/.nvm/versions/node/v4.2.4/lib/node_modules/fuge/node_modules/xenotype/xeno.js:26:16
    at Object.translate (/Users/matteo/.nvm/versions/node/v4.2.4/lib/node_modules/fuge/node_modules/xenotype/node_modules/xeno-compose/compose.js:245:5)
    at Object.compile (/Users/matteo/.nvm/versions/node/v4.2.4/lib/node_modules/fuge/node_modules/xenotype/xeno.js:24:13)
    at Object.compile (/Users/matteo/.nvm/versions/node/v4.2.4/lib/node_modules/fuge/util.js:76:10)
    at Command.runShell [as func] (/Users/matteo/.nvm/versions/node/v4.2.4/lib/node_modules/fuge/fuge.js:98:8)

Outdated deps

Low priority but Lodash is out of data in the chain install

npm WARN deprecated [email protected]: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^4.0.0

debug command crashes fuge

It looks like in the fuge-runner system.js file the start function arguments within the debug function are incorrect. It is only passing in system, name and cb but start requires a count argument also. This is making the cb become assigned to the count variable in start and crashing the program. I have done a fix locally that works just fine, can PR the affected repos if you like?

Move assets to fuge.io

When this project started we put some of the assets in the repo, namely the logo. The seneca org does this slightly differently. Since the site is pushed to a static file host (surge.sh) we make the assets available via /files/xyx.png.

The updside of this is the assets become central and easy to consume in slides, readme's et al. The downside is if you are offline your readme won't show the logo.

@pelger @davidmarkclements @geek What are your thoughts on moving the assets? I'm happy to do this work (and update the readme's) if we are all good on it?

Service name not written in package.json

If I run the fuge generate serviec command from the path it doesn't generate a the correct name. If I run node fuge.js generate service from within the cloned source it works fine. It also didn't update the test file generated.

commad

node:v4.4.0 npm:v3.8.1 ∑ microsystems/
ø fuge generate service
? Service name (service49v3m): pdf
   create package.json
   create service.js
   create README.md
   create test/serviceTest.js
   create Dockerfile
   create .jshintrc
   create .istanbul.yml
   create .gitignore
npm WARN deprecated [email protected]: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^4.0.0

> [email protected] install /Users/rocky.assad/code/microsystems/pdf/node_modules/pre-commit
> node install.js


> [email protected] postinstall /Users/rocky.assad/code/microsystems/pdf/node_modules/spawn-sync
> node postinstall

package.json

{
  "name": "service1",
  "description": "my awesome service",
  "keywords": [
    "awesome",
    "service"
  ],
  "license": "MIT",
  "version": "0.1.0",
  "main": "./service.js",
  "engines": {
    "node": "*"
  },
  "dependencies": {
    "seneca": "^0.7.1",
    "seneca-env-plugins": "0.0.0"
  },
  "repository": {
    "private": true
  },
  "scripts": {
    "lint": "jshint **/*.js",
    "test": "jshint **/*.js && tape test/*Test.js",
    "coverage": "istanbul cover tape test/*Test.js && open ./coverage/lcov-report/index.html",
    "coverage-check": "istanbul cover tape test/*Test.js && istanbul check-coverage",
    "commit-check": "jshint **/*.js && istanbul cover tape test/*Test.js && istanbul check-coverage"
  },
  "pre-commit": [
    "commit-check"
  ],
  "devDependencies": {
    "pre-commit": "^1.1.1",
    "tape": "^4.2.0"
  }
}

install service dependencies from fuge shell

if I clone a project with a bunch of services it's a drag to cd into seach service and npm install,
so

fuge > setup <service>
fuge > setup all

I'm thinking setup here, instead of install because "fuge install service" sounds like we're installing the service, instead of install the existing services dependencies. another option is fuge> install-deps-of service but it .. setup service is catchier

Exit <service> kills all services

I am in the habit of typing exit over stop by accident. One thing I noticed is exit process will exit the whole system (as I assume it ignores the input). Can we have it so that exit does nothing or just stops a process. I'm going to go out on a limb and say if someone types exit they are not looking to exit the whole system.

Docker compose does not start with default example - missing ports

I have run the default system example steps

fuge build system
fuge start (everything is working fine)

Then I tried to use latest docker-compose to start and seems the ports are not configured properly (I see nulls below)

Attaching to service2, service1, site
service2 | 2016-02-01T02:35:33.409Z do7me5zbd5qo/1454294133360/7/- INFO hello   Seneca/0.7.2/do7me5zbd5qo/1454294133360/7/- 
service2 | 2016-02-01T02:35:33.689Z do7me5zbd5qo/1454294133360/7/- INFO listen  {host:null,port:null}   
service1 | 2016-02-01T02:35:39.094Z mxgg2ejnbd94/1454294139003/6/- INFO hello   Seneca/0.7.2/mxgg2ejnbd94/1454294139003/6/- 
service1 | 2016-02-01T02:35:39.765Z mxgg2ejnbd94/1454294139003/6/- INFO listen  {host:null,port:null}   
site     | 2016-02-01T02:35:40.040Z l30qf86xc8sp/1454294139995/8/- INFO hello   Seneca/0.7.2/l30qf86xc8sp/1454294139995/8/- 
site     | 2016-02-01T02:35:40.268Z l30qf86xc8sp/1454294139995/8/- INFO client  {host:null,port:null,pin:{role:addition}}   
site     | /api/node_modules/hapi/node_modules/hoek/lib/index.js:731
   "port" [1, 2, 3]: null
site     | }
site     | 
site     | [1] "port" must be a number
site     | [2] "port" must be a string
site     | [3] "port" must be a string
site     |     at Object.exports.assert (/api/node_modules/hapi/node_modules/hoek/lib/index.js:731:11)

So the "site" does not start

fuge-config override example

In the fuge-config.js file, I'd like to activate watch on all services, except one.
I'v tried this:

module.exports = {
  runDocker: false,
  proxy: 'all',
  tail: true,
  monitor: true,
  exclude: /node_modules|\.git|\.log/mgi,
  overrides: {
    fast_ui: {
      monitor: false
    }
  }
};

(fast_ui is my service's name, not the folder itself which is different)

But when starting fuge shell, a ps command show that the service is still watch.
Can you provide a short working overriding example ?

dokku deployment

Hi,

has anyone tried fuge with dokku? It would be awesome if they work together. I use dokku extensively for all sorts of projects, but only for monolithic apps so far. Makes the deployment very easy.

I'm new to the whole microservice and docker thing and not exactly sure how I would set up a good development/deployment process.

Services should stop in the reverse order

Generally when starting servers you put them in the other you want them to spin up. But generally when you want them to spin down it is preferable to go the opposite way. Thoughts on reversing spindown?

proxy shouldn't run on linux

Just had a support request where the user was unable to run a docker container for postgres in fuge on an ubuntu os, because of this error message:

fuge shell system.yml 
compiling...
starting proxy...
  proxy postgres 5432 -> 192.168.1.6:5432
  proxy mesh 10000 -> 192.168.1.6:20000
  proxy concorda 10001 -> 192.168.1.6:20001
  proxy dashboard 10002 -> 192.168.1.6:20002
  proxy client 10003 -> 192.168.1.6:20003
starting shell..
? fuge> start postgres
running: postgres
? fuge> [postgres - 16162]: Error response from daemon: Cannot start container f2ef544bd8ff9e03af7b2fc74d1cae59365b68012cf87c181abd1951cfdf4a6d: failed to create endpoint gigantic_fermat on network bridge: Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
process exit [16162]: postgres

What it eventually ended up being was that the proxy was running on his ubuntu machine, trying to forward localhost:5432 to localhost:5432, and therefor brought down the process.

When he changed the {proxy: 'all'} in his fuge-config, it stopped happening.

we need to be detecting whether the proxy needs to run, instead of requiring users on different operating systems to change their config files (that doesn't scale).

Dash in service names causes build failures

If I generate a service with a dash in the name, lets say my-service, and then do a fuge build, I will get the following error:

containers:107
exports.my-service = {
^^^^^^^^^^^^^^^^^^^^^^^

ReferenceError: Invalid left-hand side in assignment
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:387:25)
at requireFromString (/usr/lib/node_modules/fuge/node_modules/xeno-compiler/compile.js:30:7)
at Object.compile (/usr/lib/node_modules/fuge/node_modules/xeno-compiler/compile.js:274:15)
at Object.compile (/usr/lib/node_modules/fuge/node_modules/xeno-compiler/main.js:23:9)
at /usr/lib/node_modules/fuge/node_modules/xenotype/xeno.js:26:16
at Object.translate (/usr/lib/node_modules/fuge/node_modules/xeno-compose/compose.js:245:5)
at Object.compile (/usr/lib/node_modules/fuge/node_modules/xenotype/xeno.js:24:13)
at Object.compile (/usr/lib/node_modules/fuge/util.js:76:10)
at Command.buildSystem as func

Obviously some code Fuge is generating is create that JavaScript error. I can fix it by hand by editing my compose-dev.yml to replace 'my-service' with 'myService' as the top level element, but it would be nice of the Fuge generator system could handle this. Either remove the dash and camel case the service name in the compose files, or make the code generator create JavaScript legal code for the export blocks (e.g. exports['my-service'] = {}).

REPL loop is not fully featured

I was trying to get ctrl-d aliased to exit on the shell for #19 , but it seems like it's not possible due to limitations in how incite works.

It doesn't seem to be possible to trap signals properly with it (ctrl+d means EOL for stdin), and these limitations will also mean we can't do things like tab autocompletion, and a number of other user-friendly improvements.

api and site should be separate?

Proposal:

├── fuge
│   └── compose-dev.yml
├── service1
├── service2
├── api
└── site
    └── public

api should use restify and tie directly into microservices

site should be anything defaulting to Hapi - but extremely minimal - mostly static, talks to api

The site part is the bit that will be most opinionated among dev/teams so should IMO be most sparse.

extension system

We're getting a lot of cool ideas and directions we can take this, but we also need to be strict about scope and prevent bloat. To both realise the ideas, and maintain reasonable scope we should take the small core - large ecosystem approach. This requires extensibility. We already have extensibility when it comes to generators, so I'm focusing here on the shell and CLI side.

Proposing the following:

  1. the core should be mainly reduced to an interface contract for extensions
  2. each current shell and CLI command should in itself be an extension
    • e.g. there should be a fuge-ps module, a fuge-grep module, fuge-tail etc
  3. we then create modules that are essentially clusters of functionality
    • e.g. current functionality could be called fuge-core which contains fuge-ps, fuge-tail, fuge-grep etc.
    • we could then create new sets - for instance fuge-debug which contains fuge-debug-shell, fuge-autopsy, fuge-flame and so forth
    • these additional sets would be extensions, possibly pulled in by fuge (wrapping npm) e.g.
  $ fuge install debug # === npm install fuge-debug into global npm fuge folder

(or maybe fuge add instead of fuge install.. or maybe there's an "apparatus" expression that's appropriate)

  • ..
    • this approach would also allow for just installs of single modules within a functionality pack, according to user needs/prefs
      • e.g. instead of all the debug functionality, maybe we just want fuge install flame -
    • another thing to think about is stripping away parts, for instance fuge mutate minimal which essentially uninstalls everything, including parts of core leaving a micro-core (possibly good for general production management tooling or something)

Autocomplete for executable (not shell)

We have autocomplete in the shell via vorpal, however it would be great ux if we can have autocomplete the general executable, e.g. fuge generate or fuge shell

For os x and linux this would be meaning running source on a commands bash file (e.g. Like nvm does) - not sure how this would be achieved in windows. We could either have autocompletion as default or follow the npm autocomplete pattern

allow non-global install

It's a bad practice to clutter your global PATH with project dependent tools.

Most 'global' NPM modules, including fuge, are targeted at building and maintaining projects.

I always install these into the local node_modules of the project and add node_modules/.bin to the $PATH so that I can use another version of the 'global' tool for another project.

For some reason, Fuge can't work like that:

➜  fubar git:(master) ✗ export $PATH=PATH:node_modules/.bin
➜  fubar git:(master) ✗ npm i fuge
...
➜  fubar git:(master) ✗ fuge generate system
module.js:341
    throw err;
    ^

Error: Cannot find module '/home/nherment/workspace/nearform/fubar/node_modules/fuge/node_modules/generator-fuge/generators/app/index.js'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.require.resolve (internal/module.js:23:19)
    at locateGenerator (/home/nherment/workspace/nearform/fubar/node_modules/fuge/util.js:85:20)
    at /home/nherment/workspace/nearform/fubar/node_modules/fuge/generator.js:72:20
    at Array.forEach (native)
    at createEnv (/home/nherment/workspace/nearform/fubar/node_modules/fuge/generator.js:71:16)
    at Object.generateSystem (/home/nherment/workspace/nearform/fubar/node_modules/fuge/generator.js:426:11)
    at Command.generateSystem [as func] (/home/nherment/workspace/nearform/fubar/node_modules/fuge/fuge.js:43:15)
    at Command.call (/home/nherment/workspace/nearform/fubar/node_modules/commist/index.js:100:8)
    at Object.parse (/home/nherment/workspace/nearform/fubar/node_modules/commist/index.js:58:19)

Default setup w/ redis not working (Windows)

Hi guys,

I have just followed the instructions in the README, i.e. installed fuse globally, generated the scaffold, then started it with fuge run ./fuge/compose-dev.yml.

During the scaffolding, I selected redis as the communication mechanism, for both of the default services.

This is on Windows 7, 64bit.

The errors below indicate an issue with the redis transport layer. The whole thing went a little fast so I can only surmise that the redis docker container wasn't downloaded properly?

> fuge run ./fuge/compose-dev.yml
compiling...
starting proxy...
  proxy frontend 10000 -> 10.80.99.23:20000
  proxy service1 10001 -> 10.80.99.23:20001
  proxy service2 10002 -> 10.80.99.23:20002
running: frontend
running: service1
running: service2
running: __proxy
[service2 - 6604]: 2016-01-28T09:06:35.222Z uw9y8dlaklfo/1453971995207/3108/- INFO      hello   Seneca/0.7.2/uw9y8dlaklfo/1453971995207/3108/-
[service1 - 10472]: 2016-01-28T09:06:35.239Z mpqkftac8swp/1453971995207/10152/- INFO    hello   Seneca/0.7.2/mpqkftac8swp/1453971995207/10152/-
[frontend - 11324]: 2016-01-28T09:06:35.550Z 66urhk3syiy2/1453971995519/6376/- INFO     hello   Seneca/0.7.2/66urhk3syiy2/1453971995519/6376/-
[service2 - 6604]: 2016-01-28T09:06:35.597Z uw9y8dlaklfo/1453971995207/3108/- INFO      listen  {type:redis-queue,pin:role:db,cmd:*}
[service1 - 10472]: 2016-01-28T09:06:35.613Z mpqkftac8swp/1453971995207/10152/- INFO    listen  {type:redis-queue,pin:role:db,cmd:*}
[service2 - 6604]: 2016-01-28T09:06:35.628Z uw9y8dlaklfo/1453971995207/3108/- INFO      plugin  redis-queue-transport   ACT
0nqaqjbtovi8/4iaacxe1ih9h       hook:listen,role:transport,type:redis-queue     listen  open    {timeout:10556,type:redis-queue,host:localhost,port:6379,pin:role:db,cmd:*,msgprefix:seneca_,callmax:111111,msg        {did:(wuy8k),fixedargs:{},context:{}}
[service1 - 10472]: 2016-01-28T09:06:35.644Z mpqkftac8swp/1453971995207/10152/- INFO    plugin  redis-queue-transport   ACT
bts0f3fxccyz/0e660z758mi0       hook:listen,role:transport,type:redis-queue     listen  open    {timeout:10556,type:redis-queue,host:localhost,port:6379,pin:role:db,cmd:*,msgprefix:seneca_,callmax:111111,msg        {did:(5tt3b),fixedargs:{},context:{}}
[frontend - 11324]: 2016-01-28T09:06:35.784Z 66urhk3syiy2/1453971995519/6376/- INFO     client  {host:127.0.0.1,port:10001,pin:{role:service1}}
[frontend - 11324]: 2016-01-28T09:06:35.784Z 66urhk3syiy2/1453971995519/6376/- INFO     client  {host:127.0.0.1,port:10002,pin:{role:service2}}
[frontend - 11324]: listening on port: 20000
[service2 - 6604]: events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: Redis connection to localhost:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
    at RedisClient.on_error (c:\Git\fuge-test\service2\node_modules\seneca-redis-queue-transport\node_modules\redis\index.js:196:24)
    at Socket.<anonymous> (c:\Git\fuge-test\service2\node_modules\seneca-redis-queue-transport\node_modules\redis\index.js:106:14)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at emitErrorNT (net.js:1253:8)
    at doNTCallback2 (node.js:441:9)
    at process._tickDomainCallback (node.js:396:17)
[service1 - 10472]: events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: Redis connection to localhost:6379 failed - connect ECONNREFUSED 127.0.0.1:6379
    at RedisClient.on_error (c:\Git\fuge-test\service1\node_modules\seneca-redis-queue-transport\node_modules\redis\index.js:196:24)
    at Socket.<anonymous> (c:\Git\fuge-test\service1\node_modules\seneca-redis-queue-transport\node_modules\redis\index.js:106:14)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at emitErrorNT (net.js:1253:8)
    at doNTCallback2 (node.js:441:9)
    at process._tickDomainCallback (node.js:396:17)
process exit [6604]: service2
process exit [10472]: service1

I don't want to add too much noise, but let me know if you want me to also post the contents of the three log files (frontend, service1 and service2).

Any idea what went wrong?

postmortem capabilities (autopsy and nodux integration)

http://npmjs.com/autopsy
http://npmjs.com/nodux

Nodux can be used to generate linux core files on an OS X machine

Postmortem debugging is an interesting scenario, because it can involve the entire tool chain (fuge -> oast -> substrate -> fuge) - e.g. we develop in fuge, build with oast, deploy with substrate, a process crashes, we get a core file, and we use fuge to inspect it via autopsy.

So, I see four integration pieces:

  1. integrate autopsy with fuge: postmortem <service that died>
  2. some sort of deployment environment tunnel from fuge: postmortem <currently deployed remote service that died>
  3. generate core files with fuge: <service> abort, <service> abort-on-crash - this is where nodux comes in for OS X, when these commands are run, it may ask Restart service in Linux container (Y/n)?
  4. enable core file dumps (by default) with substrate

scaffolding tool for fuge

I'm looking at bringing our scaffolding situation in line with requirements, and it strikes me that we really are looking for a separate tool instead of using one baked into fuge.

Needs :

  1. scaffold services even without fuge being installed.
  2. scaffold both entire systems and individual services.
  3. scaffold both backend and frontend 'services'
  4. library of 'generators' that work out of the box with fuge.
  5. simple to build and maintain 'generators'.
  6. some way to install / manage 'generators'.
  7. curated 'generator' list.
  8. can publish public and private 'generators'.
  9. can search for / discover new 'generators'.
  10. ability to accept user input during scaffolding.
  11. ability to script scaffolding.

Viability.

I'm of the opinion that Yeoman doesn't tick nearly enough of these boxes, but my biggest concern is that we will never be in a situation where we can pull in any yeoman generator and have it 'just work' (tm).
This issue is not about the limitations of yeoman though, but the properties of the tool we want.

I think that the fact that yeoman is our only real option here shows that there is a gap in the available tooling, and that is at least partly due to the nature of scaffolding tools. Because you only really use them at the start of a project, they have diminishing returns ... microservices are an edge case though, since you would potentially use the generators far more regularly.

By building our own tool with it's own namespace to work with, we can ensure a curated set of generators that work well with fuge, creating another funnel for new users, but very importantly providing another way for people to start contributing to the project. ie: another extension layer.

For the generators themselves, I am partial to building on the great work done on MetalSmith, which is more than just a static site generator. It is actually a very flexible api for turning one directory containing files into another directory containing files, with pluggable transforms in between. It can be used to build a project scaffolder, and there are some other prior art projects too.

For package installs / publishing, we would use npm. I imagine we could add a keyword search into the command line tool, which would allow you to search for more generators which would be installed by a command that basically just does 'npm install -g' for you. This would net us the full abilities of npm infrastructure, including namespaced packages, private packages, npme. etc.

What do we call it?

I have a pretty good idea of what to build, but now for the fun part. Giving it a name ... because giving something a name is what makes it real ;)

I was thinking something in the vein of gene therapy. As in, taking strands of DNA and inserting them into your organism. The only word I could come up with was contig, but that's prone to typos and probably not in-line with the apparatus theme.

Remember that this is going to be a command line script, so needs to be short and memorable, and have the namespace available (npm packages, etc.)

Service names in serviceTest.js don't match names in service.js when generating service.

I'm running this:

fuge generate service
Service name (servicelvj9s0): blah
create package.json
create service.js
create README.md
create test/serviceTest.js
create Dockerfile
create .jshintrc
create .istanbul.yml
create .gitignore

The files are generated as expected. In service.js we have:

2016-03-13 07_36_56-digital-ocean-dev - cloud9

However, in tests/serviceTests.js we have:

2016-03-13 07_41_12-digital-ocean-dev - cloud9

Looking at the code (and assuming I'm not missing your intent), the problem is in generator-fuge in your serviceTest.js template:

image

I've sent you a pull request. Note I made the fix in github directly, so it hasn't been tested. Please let me know if additional steps are necessary.

document team workflow

Here is our scenario:

  • dev1 sets up repo with fuge, pushes to origin
  • dev2 pulls repo and installs fuge -g
  • dev 2 runs > fuge shell fuge/compose-dev.yml then start all
  • dev2 has dependencies issues
➜  fubar git:(master) ✗ fuge shell fuge/compose-dev.yml
compiling...
starting proxy...
  proxy frontend 10000 -> 192.168.1.6:20000
starting shell..
? fuge> start all
running: frontend
running: __proxy
? fuge> [frontend - 12501]: module.js:341
    throw err;
    ^

Error: Cannot find module 'hapi'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (/home/nherment/workspace/nearform/fubar/site/api/index.js:3:14)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Function.Module.runMain (module.js:447:10)
process exit [12501]: frontend

This was solved by executing:

cd site/api
npm install
cd -
fuge run fuge/compose-dev.yml

It'd be great if fuge could see that it's a new install and run npm install wherever it is needed. It'd be even better if someone updates a package, for fuge to see it and automatically maintain the npm dependencies.

Fuge build not running Dockerfile RUN command

In your fuge project, we have a node service that requires to be built.

Here is the Dockerfile:

FROM node
ADD . /
RUN npm install
RUN npm run build
CMD npm start

And the global compose-dev.yml:

mesh_base:
  build: ../mesh-base/
  container_name: mesh-base

fast_ui:
  build: ../fast-ui/
  container_name: fast-ui

fast_service:
  build: ../fast-service/
  container_name: fast-service

concorda:
  build: ../concorda/
  container_name: concorda

concorda_dashboard:
  build: ../concorda-dashboard/
  container_name: concorda-dashboard

The build output is :

$ fuge build fuge/compose-dev.yml
building...
building: mesh_base
building: fast_ui
building: fast_service
building: concorda
building: concorda_dashboard

But the expected npm install and npm run build output folders still don't exist.
How can I check what build as effectively done ?

faulty restart process for ui services

when we delegate to another process to watch and reload (e.g. gulp), by using an override in config.js,
fuge assigns a new port on every restart

we need some way to either lock a service to a port, or else rethink how we're doing ui rebuilds

fuge thinks it is running docker container even though the images haven't been pulled

When i start all in a docker-compose file that has infrastructure level services (in this case influxdb and elasticsearch), fuge will happily report these services as running in the ps listing.

However, fuge does not in itself do any docker pull image, so these services are not really working.

fuge pull should really also be pulling docker images, and not just git repos.

Allow for running services in containers as well

Fuge chooses to start all services as processes instead of containers. It would be great to have an option to start services in their containers as well, while of course maintaining as many of the fuge shell features as possible (like watching for changes and log grepping). This would help in dealing with services that rely for example on different versions of node, or maybe even include services that have been written in other languages as well.

Project name

"fuge" in Romanian literally means "is running" (as in "he is running around the park", not "the process is running").

Thought you may find this funny enough to include in the readme.

Cheers.

idea: rapidly create throwaway mock-services from command line

in shell:

fuge> mock my-mock-service
Pattern: {'role': 'service-to-be-done-later', cmd: 'uwatm8'}
Response:
   {
      res: 'ok', 
      nahnah: true
   }
Ctrl^D
Pattern: Ctrl^D
creating mock service
starting mock service
fuge> ps
name                         type            status          watch           tail            count
mongo                        docker          not managed
apache                       docker          not managed
service1                     process         started         yes             yes             1
my-mock-service              mocked          started         n/a             yes             1 

Pattern: and Response: are prompts, the data that comes after is user supplied

The idea is it's framework independent - simply define pattern and response

There's a few angles here

  1. create a temporary service that disappears when the shell closes
  2. as 1, except don't disappear, save it somewhere internally (not as a generated seneca service)
  3. possibly create a way to "commit" or "convert" a mock service into a full generated service, building boilerplate around supplied pattern-responses
  4. add/remove/edit pattern-responses for a mock-service

fuge generate system should accept the name of a folder

I was in a folder that had a checkout of the fuge repo, and i tried to create a new folder to test with, but I got this instead.

Sikozu:nearform adrian$ fuge generate system fuge-test
fs.js:842
  return binding.mkdir(pathModule._makeLong(path),
                 ^

Error: EEXIST: file already exists, mkdir '/Users/adrian/Projects/nearform/fuge'
    at Error (native)
    at Object.fs.mkdirSync (fs.js:842:18)
    at Object.generateSystem (/Users/adrian/.nvm/versions/node/v5.3.0/lib/node_modules/fuge/generator.js:425:8)
    at Command.generateSystem [as func] (/Users/adrian/.nvm/versions/node/v5.3.0/lib/node_modules/fuge/fuge.js:43:15)
    at Command.call (/Users/adrian/.nvm/versions/node/v5.3.0/lib/node_modules/fuge/node_modules/commist/index.js:100:8)
    at Object.parse (/Users/adrian/.nvm/versions/node/v5.3.0/lib/node_modules/fuge/node_modules/commist/index.js:58:19)
    at start (/Users/adrian/.nvm/versions/node/v5.3.0/lib/node_modules/fuge/fuge.js:145:27)
    at Object.<anonymous> (/Users/adrian/.nvm/versions/node/v5.3.0/lib/node_modules/fuge/fuge.js:153:3)
    at Module._compile (module.js:398:26)
    at Object.Module._extensions..js (module.js:405:10)
    ```

fuge test

from the compose yaml file, fuge knows where each service is - and it also knows how to run tests from the package.json file

what would be nice though is to have a way to aggregate all of the results and coverage centrally,

the https://www.npmjs.com/package/nyc tool sits on top of istanbul and checks coverage over multiple processes and then generates a single report

tape output could also be easily aggregated - as long as there's a way (by some convention) to get the TAP output of each test a user could simply run

fuge test

to test an entire deployment in a central place

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.