duojs / duo

A next-generation package manager for the front-end

Duo is a next-generation package manager that blends the best ideas from Component, Browserify and Go to make organizing and writing front-end code quick and painless.

• Features
• Installation
• Getting Started
• Command Line Usage
• Javascript API
• Duo.js Plugins
• FAQ
• Mailing List
• #duojs on freenode

Features

1. has first-class support for Javascript, HTML and CSS
2. exposes a unix-y command line interface
3. pulls source directly from GitHub with semantic versioning
4. supports source transforms, like Coffeescript or Sass
5. does not require a manifest

Installation

Install Duo straight from npm with:

$ npm install -g duo

Getting Started

To get started just write normal Javascript, requiring dependencies straight from the file system or from GitHub as you need them:

var uid = require('matthewmueller/uid');
var fmt = require('yields/fmt');

var msg = fmt('Your unique ID is %s!', uid());
window.alert(msg);

Then use duo to install your dependencies and build your file:

$ duo index.js

Finally, drop a single <script> onto your page and you're done!

<script src="build/index.js"></script>

Same goes for CSS! You can require dependencies and assets from the file system or straight from GitHub:

@import 'necolas/normalize.css';

body {
  color: teal;
  background: url('./background-image.jpg');
}

Then bundle up your CSS with duo:

$ duo index.css

And add your bundled-up stylesheet to your page!

<link rel="stylesheet" href="build/index.css">

Authenticate with Github

We recommend that you authenticate with Github so you can increase your rate limit and allow you to pull from private repositories. To do that, add the following entry to your ~/.netrc file:

machine api.github.com
  login <username>
  password <token>

You can create a new token here: https://github.com/settings/tokens/new

Debugging

If you run into an issue with Duo, often times you can resolve it by prepending DEBUG=duo* to your $COMMAND:

$ DEBUG=duo* $COMMAND

Example:

$ DEBUG=duo* duo index.js

If you can't figure it out, you should open an issue: https://github.com/duojs/duo/issues

Test

Download this repository and run:

make test

Authors

• Matthew Mueller
• Amir Abu Shareb
• Plus many more wonderful contributors!

License

The MIT License

Copyright © 2014

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

duo's Issues

use jstrace

to find bottlenecks, and potentially improving perf

https://github.com/jstrace/jstrace

somehow interface with browserify (well package.json)

lots of folks use browserify. haha. duo should probably have some avenue to interface with it.

looking at the package.json is trivial, resolving their dependencies is not. would be feature creep if added to core, so it should probably be a plugin.

colon shorthand

hey guys,

wanted to bring this up, but what do you guys see as the benefits of the : shorthand? i actually find that it ends up net negative since no one is used to it, and it introduces an unnecessary fragmentation

i much prefer:

var once = require('component/once');

to:

var once = require('component:once');

whenever you see any github repo, it's always as x/x, and even more so because pull requests now have an x:x syntax that is owner:branch which only adds more confusion

feels like a solution in search of a problem to me, especially if we're going to support the slash method anyways

duo-css: should parse each of the css files in "styles": [...]

I think this will allow us to continue to support components with JS and CSS. It's a little weird though, because there should probably only be one style that imports everything else.

@yields @ianstormtaylor @visionmedia could use some thoughts here.

duo(1): support stdin

travis

use it, duo package now supports GH_USER, GH_TOKEN env vars.

npm + component

Not sure if I'm thinking about the properly, but wanted to write it down and see what you guys think.

Right now I think Duo relies on a few different pieces from the manifest...

[
  "name",
  "main",
  "dependencies",
  "development"
]

...since the install and build process are intertwined, it actually needs to read the component.json while building to figure out which modules to install if there are conflicts, I think?

But there could be a way that we change the build step to not actually do any installing, and instead just error out if things aren't installed locally (like npm does, which isn't that insane of a requirement, and could be argued is even less error prone in some ways). If we do that, (and we install dependencies as a tree like npm does), then looking up dependencies is actually just looking at the filesystem and reading the files that exist there, so then Duo only actually relies on these fields from the manifest:

[
  "name",
  "main"
]

...which happen to be the same across both component.json and package.json (maybe even bower.json but I don't remember). And in that case we could maybe get supporting both to be as simple as...

var json = read('component.json') || read('package.json');

...which would be sweet. And even mean that Duo could be used to install server-side packages and replace npm entirely?

It's extremely likely I forgot something super obvious, cuz I'm about to pass out :)

add --global option

to expose the entry file as --global <name>

Do we even need github authentication?

Now that we change gh-resolve to use git remote-ls, fetching the tarball is the only authenticated request we are making.

We can actually fetch files by making unauthenticated GET requests to:

GET https://github.com/$user/$project/tarball/$ref

I did a bit of testing it looks like it doesn't work for branches older than 2013. However it works for releases that go way back, and it works for master.

I'm kind of indifferent, because I think 5,000 requests is reasonable. But it does simplify things by just not authenticating. Using the official API is probably a more reliable solution. The other thing downside of not authenticating is that you don't have access to private repos.

On second thought, we should probably just make everyone authenticate in the beginning and so no one runs into any problems.

duo(1) only supports stdout

Should also support:

$ duo build in.js out.js

Also, @yields does it make sense to break out the build section of _duo into duo-build?

Update readme to vs v1?

It's hard to tell the difference since I handled a lot of components issues. Maybe code examples?

rate limiting

I think this is something that needs to be solved somehow. Even limiting it to 5000 an hour could not be enough eventually. I mean, there are only ~2000 components at the moment, but still.

Some ideas:

1. Have a server running somewhere that stores the tags for packages. It would have REST endpoints for adding new tags for packages. component register 1.4.0. We can fetch that and store it locally per remote. eg. ~/.duo/github.com/:user/:project/tags.json. Or we just don't have the server and just cache them.

1.1 We could theoretically fetch the tags for every single component at once and load up the cache somewhere. You could preload your local cache from there maybe? component tags update.

1. Ask Github to lift the rate-limiting restriction for package managers and/or tag API calls (highly unlikely)

Ideally we should never, ever have to worry about being rate-limited.

duo-duplicates: show total size

if you removed all duplicates you would save 20kb etc..

comparison

with transforms doesn't make a ton of sense, you could do the same thing in component, the api wasn't great but with no streaming it's still easier, all the browserify transforms I've seen look 10x worse because of it

nice error message for creds?

segmentio/analytics.js-integrations#233

dunno if there's something nice we can do there, so that people dont get crazy confused haha

tests

@yields you mentioned you'd get us started on these.

nasty bug

here's how the file looks: require('trigger-event'), for some reason duo thinks i want component/event and simply returns it instead of ianstormtaylor/trigger-event.

duo.json:

looking into it...

duo(1): info should probably be logged to stdout, not stderr

https://github.com/component/duo/blob/master/bin/_duo#L25-L29

pluggable builds

different filetypes require different treatments:

• js: packing algorithm (duo-pack)
• css: concatenating and symlinking assets
• html: symlinking assets

break duo(1) into duo-install(1) and duo-build(1)

Based on previous discussions with @yields, it'd be nice to break duo(1) into an install and build piece.

This was originally the plan and was the case for the first implementation of duo, but when we started adding parsing on the install, the lines got blurred. I think the lines will always be blurred, but I think we can break the shared pieces out into plugins and use them in both the install and build.

The main thing I like about having the distinction is being able to install components from the command line. ex: $ duo install component/matches-selector.

One solution we could implement immediately is to have a duo-install(1) which can use duo's internal api to install components, but I'd like to reorganize at some point.

duo bin: more info about what's going

right now:

$ duo entry.js

might be pinging the moon and we wouldn't know. haha

duo-duplicates - show pkg size.

$ duo duplicates --size

will be slow but faster then checking it manually.

node support

I'd just like to put it out there that it'd be nice if we could do away with npm(1) entirely (in the nicest way possible, it'd be good to have some healthy competition in this area).

feedback

Don't know where else to put the feedback so I opened this issue.

TL;DR; you are not fixing enough real problems over browserify/npm/github-url to warrant yet another package manager/bundler/pipeline/etc

I think tying your registry and package installation tool to your asset bundler is a mistake. It was a mistake in component and I think it is a mistake in whatever duo is trying to be. Component became a god tool that does everything and as a result imo became shitty at anything because maintaining such a tool is no small undertaking. More importantly, asset packaging, installation, dependencies, etc is all very opinionated and by bundling everything into a single tool then you drastically limit experimentation or access.

I have more opinions on this but really I don't see enough wins to warrant switching and many of the items listed as "bad" have trivial workarounds.

installing from branch is fetching the github branch instead of using the local copy

Not sure if this is it exactly, but the situation is:

cd some-repo # with a `component.json` and some dependencies in it.
git checkout -b dev # in new branch
git push origin dev # push branch to github
# change the component.json
duo index.js build/build.js

I started with a fresh folder, no node_modules or components or any of that. Then npm install.

I made some changes to the component.json (changing some versions of existing deps) and it didn't pick them up.

Looking in the components folder, it downloaded the dependencies from the github branch instead of the local one with the changes made. To get it to install the local changes, I had to push to the branch.

Probably something simple, any thoughts?

CSS Imports

i personally like component.json's styles fields vs. @import statements. here are some thoughts about actually implementing it:

Import statements

I want the import statements to match the JS import statements. In other words, it should look like:

@import 'github.com/necolas/normalize.css/3.0.1/normalize.css'

this way, like the JS, it should work with and without a package manager

Manifest files

Instead of adding every single CSS file to the manifest (even with a glob), you would just allow a CSS file to be included in the .main property. thus, a main could look like: main: ["lib/emitter.js", "lib/emitter.css"]

Ordering and Multiple Imports

It's fine when you only add the @import statements at the top, but what if people add them arbitrarily? What if the ordering defined in multiple CSS files conflict with each other?

This is why I like component.json's styles since adding a CSS dependency locals only guarantees that its CSS files will be included before, which is all the assurance you need.

Duplicate Versions

I think rework-npm doesn't even bother to include duplicate versions. I think it actually includes whichever was imported first.

Slow

This is going to be much slower and more susceptible to user errors.

duo-create(1)

for quick bootstrapping.

file transforms

i've been thinking a lot about file transforms lately and I think I've come up with a decent solution. Here's what the API would look like:

duo
  .use(jade())
  .use(styl())

Where styl and jade are one of the following:

• a synchronous function that returns a string function(...) { return compiled }
• an asynchronous function: function(..., fn) { return fn(null, compiled) }
• a gulp stream

The reason I'm pushing for gulp streams is that it will allow us to tap into a huge repository of plugins for things like: transpiling ES6 => ES5 (adding import, etc), coffeescript support, and many other things I haven't thought of.

However, the problem I'm running into is the following. Maybe this is obvious to someone that's worked a lot with streams, but basically if you do the following, you will not get the intended results:

var fs = require('fs');
var map = require('map-stream');

function prepend(str, fn) {
  return fn(null, 'hi ' + str);
};

var t = map(prepend);

var a = fs.createReadStream('./a.txt', 'utf8');
var b = fs.createReadStream('./b.txt', 'utf8');
var c = fs.createReadStream('./c.txt', 'utf8');

a.pipe(t).pipe(process.stdout);
b.pipe(t).pipe(process.stdout);
c.pipe(t).pipe(process.stdout);

Basically, the transform stream pulls from all 3 sources at the same time. This is a problem because gulp streams are already initialized by the time they're passed through (ie. styl()).

What needs to happen is the files need to be queued and wait until the transform pipeline is free. I'm not sure if this is going to be a bottleneck yet, or if there is a better way, but that's what I'm facing right now.

I guess to get duo out there maybe we can just support the first 2 options (sync and async fn), but I think the gulp streams will be a huge plus for people.

---

How we'll walk the files is the following:

$ duo main.css

main.css:

@import "./one.styl"
@import "./two.less"
@import "./three"

To resolve three we will base it off the root entry (extname("main.css") => ".css" => "./three.css"). We will do the same for JS and eventually HTML.

/cc @yields @juliangruber @ianstormtaylor

support duo entry.coffee, entry.styl, etc.

will need to place the root extension (rootext) of the entry file after the transforms.

duo-duplicates exit with > 0

already in dev branch, anyone thinks it would be useful to exit > 0 ?
maybe leave it to the user ?

hmm tough call haha.

duo(1): UX for adding authentication details to .netrc

Also provide better error handling when you don't have auth details.

logo?

Can't help but get excited about the release and every good release has a sweet logo. We were talking about just keeping the component logo, which is fine with me, but I'd like to throw some other options into the pot:

:-D

idempotent

not enough, figuring it out now.

move requires from bin/_duo so commands execute faster

Just logged before and after the require statements in bin/_duo and it takes a while for those things to load up, maybe 100-500ms, so it takes a half a second or more to execute the command.

#!/usr/bin/env node

/**
 * Module dependencies.
 */

var write = require('fs').createWriteStream;
var Command = require('commander').Command;
var resolve = require('path').resolve;
var exists = require('fs').existsSync;
var mkdirp = require('mkdirp').sync;
var Logger = require('stream-log');
var stat = require('fs').statSync;
var spawn = require('win-fork');
var pkg = require('../package');
var path = require('path');
var fs = require('co-fs');
var Duo = require('..');
var co = require('co');

Ideally it would immediately execute the command and give you feedback similar to how component does it now :)

duo css: rewrite url(...)'s and symlink assets

should probably get done before we release. going to work on it tonight.

/cc @yields

too much logs ?

should we just log installed : user/project 0.0.1 and then keep the rest under a --verbose flag ?

cli tool

@matthewmueller let me know if that shouldn't be here ?
i would like to get this thing working :P

what to expose ?

i'm adding ability to expose some modules so that they are requirable outside the build.

this is useful for mainly for tests, since duo will automatically require the "main" file for you,
so doing the whole <script>require('boot')</script> in duo is unnecessary.

so what should we expose ? what do we do about conflicts (a: 0.0.1, a: 0.0.2) ?

option a

simply expose whatever is in the root component.json, and ignore everything else.
this is the simplest solution since duplicates will be edge case:

component.json

{
  "repo": "segmentio/analytics.js",
  "dependencies": {
    "component/emitter": "0.0.1"
  },
  "development": {
    "component/assert": "0.0.2"
  }
}

test.js

var analytics = require('segmentio/analytics');
var emitter = require('component/emitter');
var assert = require('component/assert');

option b

expose everything and requires must have versions.
this solution is kinda ugly because inside a module you don't have to append a version:

module.js

require('component/[email protected]');

another annoying thing about that option is that you have to update your requires that are outside of the build, if module.s changes it's version component/[email protected] will not be installed anymore.

option c

expose everything without versions.
this solution is another simple one but can be annoying:

my-module.js

var clone = require('component/[email protected]');

dep.js

var clone = require('component/[email protected]');

test.js

var clone = require('component/clone'); // which one is it ? wtf ?

i hope we find a solution that works for everyone.

/cc @matthewmueller @ianstormtaylor

gulp support (via plugin)

Something like this:

var gulp = require('duo-gulp');
var styl = require('gulp-styl');

yield duo(__dirname)
  .use(gulp(styl)(opts))
  .run()

duo#include(name, str | fn)

for adding things like jade, regenerator, etc. runtimes

no locals

Duo has no concept of locals, but root is the project's base directory. require('signup) => require('/lib/signup')`

not sure i agree with that. if you do require('signup'), then you need to explicitly set a path somewhere. i don't believe you should ever implicitly add a path or do require('signup') unless you explicitly do something like this:

module "signup" {
  export default function () {}
}

so i'm generally leaning towards ../ since that should work in browsers.

shasum 0.2.3

shasum check failed for /var/folders/hw/q68jhnmd09b3bt9hsbgdd6l40000gn/T/npm-2170-oGQe8VtG/registry.npmjs.org/duo/-/duo-0.2.3.tgz

@matthewmueller needs releasing again ? or am i the only one who gets he error ? :/

don't use streams

NOTE: i guess i confused the gulp api with duo's api. however, my general stance against node's streams remains.

my first builder used streams. node streams are honestly terrible. think about all the error handling. making it a public plugin API is even more terrible.

the worst part about node streams is that it doesn't support concurrency unless you do crazy shit. plugins should be as simple as possible, and making each a transform stream that has to do its own concurrency control is terrible. this is why i have https://github.com/cojs/chanel, but i'm sure you could create something similar for this.

the builder should be as fast as possible, using 100% CPU. streams are for concurrency and low memory usage, which is unnecessary for a build step. one of the main reasons i took out streams in the builder was because my 50kb gzipped js bundle takes 50ms to build (even with regenerator and other plugins), making streaming absolutely useless.

i'd rather have a generator-based API since this wouldn't be ready for prime time anytime soon (at least i don't think).

i also think that plugins should be file-based and very simple, which is how the current builder works. i think your transforms are similar, but component-builder@0 transformed the entire tree, which makes it extremely difficult to create plugins.

don't fail if there are no components to install

{ [Error: ENOENT, open '/Users/matt/Projects/duo/examples/regenerator/components/duo.json']
  errno: -2,
  code: 'ENOENT',
  path: '/Users/matt/Projects/duo/examples/regenerator/components/duo.json' }

/Users/matt/Projects/duo/examples/regenerator/index.js:19
  if (err) throw err;
                 ^
Error: ENOENT, open '/Users/matt/Projects/duo/examples/regenerator/components/duo.json'

support transforms from the command line

Not sure the best way to do this without being somewhat verbose:

$ duo entry.js --transform styl

Or maybe we can support a key in the component.json, transform. Not sure how args like { whitespace: true } would work though.

crc ?

fs.stat() lies sometimes for some reason, try saving and then immediately build, duo will do nothing, save again and it will work.

move repo to duojs/duo

Just want to get a discussion going on this topic. My thoughts:

Pros:

• less clutter in the component organization (duo transforms have their proper place)
• duo will consume more than just components
• less chefs in the kitchen. imo, component has too many people with admin rights atm.

Cons:

• in a different home than component, might not be considered the next component in people's minds.
• github.com/duojs/duo isn't quite as nice at github.com/component/duo. might be able to get github.com/duo/duo eventually

/cc @yields @ianstormtaylor @visionmedia @jonathanong @juliangruber

add cli tests

we only test the api ATM.

add --development

removed it accidentally when i rewrote duo(1) and removed duo-build(1), duo-install(1).

ES6 imports

TJ said that he doens't want to support <remote name>:<user>/<repo> imports. i'm cool with that.

however, i'm not down with import 'github.com/component/[email protected]'. imo, all these important statements should work without duo. in other words, we should do whatever browsers support so that all these components can work with and without Duo (or whatever). too bad we don't know what browsers are supporting yet.

of course, this doesn't matter if we create our own endpoint. this is one of the reasons i wanted to create https://github.com/component/cdn. then we can do import 'github.com/component/emitter/^1.1.0/index.js' or something and it'll work both with and without a package manager