Gist endpoint? about systemjs

[guybedford]

This could certainly be done. Making new endpoints is actually surprisingly easy - the implementation is all in these modules - https://github.com/jspm/npm/blob/master/npm.js / https://github.com/jspm/github/blob/master/github.js

That is all there is too it. A getVersions and a download function.

New endpoints can be tested with the CLI tool as well.

If you make one, I'll host it :)

[matthewp]

I'm working on this. How would we like to require them? I'm thinking:

var mod = require('gist:user/file.js@3424213');

Where user is the user, file.js is the first filename of the gist (gists can have multiple files but we can't really support that) and 3424213 is the version, if you want an older copy. Think this can all be easily done from the api.

Gists are actually looked up by id but I think that leads to an unfriendly string. We can still find them by the file name, which I think looks nicer, opinions?

[guybedford]

Nice :)

Are you basing this on https://github.com/jspm/npm/blob/master/npm.js?

Sounds like the right scheme to me, annoying about the multiple files thing, but the functionality could always be amended later with some syntax if really needed.

[matthewp]

Yep, more or less. Are you expecting getVersions to callback with an object containing key/value pair between a semver and a hash? Gists don't have a semantic version number, just a hash, so I was going to return the hash as the key and value like so:

{
  "adca60ebab5e16aa556da1855fc9debd0df5e514": "adca60ebab5e16aa556da1855fc9debd0df5e514",
  "c28579e01580872f86ddaaae3c63aa0814156be6": "c28579e01580872f86ddaaae3c63aa0814156be6"
}

Is this right?

[matthewp]

Also, if the user does this:

require('gist:user/file.js');

Are version and hash undefined (or null) in download? I would expect them to be, then I would just grab the latest version.

[guybedford]

II remember correctly, add a 'latest' property to the version map. This way
the download function always gets a hash.

On Sunday, December 29, 2013, Matthew Phillips wrote:

Also, if the user does this:

require('gist:user/file.js');

Are version and hash undefined (or null) in download? I would expect them
to be, then I would just grab the latest version.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-31305909
.

[guybedford]

Sorry it's master not latest here.

The handling of the hash in the version map is correct, but perhaps use a shorter version of the hash (normally 5/6 chars is enough I think?). Something like:

{
  "adca60": "adca60ebab5e16aa556da1855fc9debd0df5e514",
  "c28579": "c28579e01580872f86ddaaae3c63aa0814156be6",
  "master": "c28579e01580872f86ddaaae3c63aa0814156be6"
}

would make the requires nicer for revisions.

[matthewp]

Thanks @guybedford . For download I just need to fetch the gist and save it as a file inside outDir, and then callback with a package, right? In my case I think I can just always save the file to index.js and then return the package as:

{
  main: 'index',
  files: ['index.js']
}

[guybedford]

@matthewp yes exactly that should work. No need to provide the files array since that is primarily to delete other files.

[matthewp]

Think I've got it, can look here. As the readme says, I made is that the repo value in getVersions and download can be either a username/filename combo or just the gist id, so the user should be able to do any of these:

// By user and filename
require('gist:matthewp/objectis.js');
// Including the version hash
require('gist:matthewp/objectis.js@a9053ea');

// By id
require('gist:8175510');
// Including the version hash
require('gist:8175510@a9053ea');

Let me know how it looks to you.

[guybedford]

Cool, I will check this out tomorrow. Have you tested it against the jspm CLI?

It will automatically pick up the endpoint if you add it in a "gist" folder in your node_modules folder.

[matthewp]

No, will try that though, thanks.

[matthewp]

Doesn't work, wants to install in undefined directory for some reason. Any tips? By the way, had to put it in node_modules/jspm-gist

➜  test git:(master) ✗ jspm install gist:matthewp/objectis.js
     Getting version list for gist:matthewp/objectis.js
     Downloading gist:matthewp/objectis.js@master
Enter external library install location (jspm_packages): jspm_packages
     Checking gist:matthewp/objectis.js@master for package.json override
err  Error: ENOENT, no such file or directory '/home/matthew/Projects/test/undefined/gist/matthewp/objectis.js@master'
         at Object.fs.mkdirSync (fs.js:642:18)
         at prepareDir (/usr/local/lib/node_modules/jspm-gist/gist.js:60:8)
         at save (/usr/local/lib/node_modules/jspm-gist/gist.js:91:7)
         at /usr/local/lib/node_modules/jspm-gist/gist.js:113:11
         at Request.callback (/usr/local/lib/node_modules/jspm-gist/node_modules/superagent/lib/node/index.js:628:30)
         at Request.<anonymous> (/usr/local/lib/node_modules/jspm-gist/node_modules/superagent/lib/node/index.js:131:10)
         at Request.EventEmitter.emit (events.js:95:17)
         at Stream.<anonymous> (/usr/local/lib/node_modules/jspm-gist/node_modules/superagent/lib/node/index.js:773:12)
         at Stream.EventEmitter.emit (events.js:117:20)
         at Unzip.<anonymous> (/usr/local/lib/node_modules/jspm-gist/node_modules/superagent/lib/node/utils.js:124:12)

[matthewp]

Actually it looks like outDir is not a single directory but nested? I should use mkdirp then, right? Any clue why it wants to put it in undefined instead of jspm_packages?

[guybedford]

Make sure you do 'jspm init' before installing... This should be made
easier though.

On Monday, December 30, 2013, Matthew Phillips wrote:

Actually it looks like outDir is not a single directory but nested? I
should use mkdirp https://npmjs.org/package/mkdirp then, right? Any
clue why it wants to put it in undefined instead of jspm_packages?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/41#issuecomment-31327724
.

[matthewp]

K, I've updated the repo and it's working using jspm-cli. Not sure where the undefined behavior was coming from, I had done a jspm init before installing. Might get a chance to debug that later.

[guybedford]

I tested this out on the dev server, and spotted two issues. One is easy, the other is a little trickier.

Handling of "not found" is quite important that it doesn't do an errorback. Rather just provide an empty callback (undefined or null) to indicate that the repo wasn't found in getVersions.

The harder issue is that the server validates the request based on assuming that the "package name" is of fixed length. Eg for Github, the package is length 2 - "some/repo", while for NPM the package is length 1 - "repo".

In this case, we can have either.

I'll need to make this code a little more flexible on the server, perhaps with a special "parse" / "validate" hook for package names so that the endpoint code itself can validate this. Will keep you updated.

[matthewp]

I've updated it to callback with nothing when it cannot find the repo.

[guybedford]

Great! It should also do the same if the user can't be found as well.

[guybedford]

The best we can do on the other issue for now is to choose to support one or the other of:

require('gist:matthewp/objectis.js');
and
require('gist:8175510');

That way we can at least get one working on the server sooner rather than later.

I'll let you decide - then just set the Location.prototype.degree property to the number indicating the number of / parts.

[matthewp]

Cool, we'll just use ids for now because the former syntax is more error-prone (if you have multiple files with the same name it just grabs the first one). I'll update the repo with that change and to call the callback when a user isn't found.

[matthewp]

Ok, I've updated it. I wasn't able to retest with jspm-cli because I started getting rate-limited but I think everything's good. I left the code in that allows user/repo to be parsed but we'll just not document that feature until it's ready server-side. For now ids should be fine since you can always map them.

[guybedford]

Great will test this out.

You can deal with the rate limit by accepting a "username" and "password" option (https://github.com/jspm/github/blob/master/github.js#L22). See https://github.com/jspm/jspm-cli#rate-limits

[guybedford]

Actually this wouldn't work - it would have to be jspm set gist.username since the config is endpoint-specific.

[matthewp]

Awesome, will try that myself.

[matthewp]

Hm, obviously won't work in my case since I didn't build that into the module! Will look into doing that later, but I suspect I'll have to rewrite a bit since I'm doing raw http requests, not using a github api module.

[guybedford]

See https://github.com/jspm/github/blob/master/github.js#L197 for an example of authorized requests.

I'm getting dir is undefined on line 62 at the moment unfortunately.

[guybedford]

Actually that error is my input, not your side.

[guybedford]

Ahh, you need to do:

  this.baseDir = options.baseDir;

in the constructor function.

I can probably clean this API up a bit, you can tell it's evolved.

[guybedford]

Ok, it's now working on the dev server here - https://gist.jspm.io:442/8175510@master/index.js

When you've got the next changes working (auth and the baseDir) we can push this live.

[matthewp]

Ok, I've updated with those two changes. Why exactly does baseDir do? I see from the GitHub example that it is set in the constructor but never used (I did the same).

[guybedford]

Excellent, I think we're there!

Could you publish this to npm as jspm-gist, and then add me as a co-maintainer?

Then I'll include this in the loader config as well by default.

The baseDir is just the underlying folder where the entire endpoint is saved, which is used to derive the download dir.

[matthewp]

Published! https://npmjs.org/package/jspm-gist

[guybedford]

Great, this is on the live server now.

For example, you can run jspm.import('gist:8175510') on jspm.io.

I haven't pushed it back into the loader, because the codebase is frozen with the current update.

I don't yet see myself as a maintainer on jspm-gist yet? It's kind of key in case there is a bug, if we are providing it as a standard endpoint.

[matthewp]

Yeah, it's weird, if I do npm owner ls jspm-gist it shows you, but if I do npm info jspm-gist it just lists me. Do I need to republish to get you to show up in the web ui?

[matthewp]

Yep, force republish did the trick.

[guybedford]

Ahh, I see - I didn't realise that was a delay. That's great though, no worries.

Thanks so much for getting this through!

[matthewp]

It was a great way to learn, look forward to seeing the server code open-sourced and I might host some endpoints of my own.

[guybedford]

Nice to work with a quick learner as well! Yes the server code will be completely open sourced hopefully Q1. It's the next step along with builds after the current loader changes. Will keep you updated.

[probins]

thanks, guys. I will try this out

[matthewp]

@guybedford Not to waste too much of your time, but the bower endpoint turned out to be even easier... so I went ahead and wrote it. It's here. Not sure if you want to make this one official or not, doesn't matter to me either way.

[guybedford]

Nice!! It may well make sense as an official endpoint, since the bower versions of many libraries are the most maintained package registry, and have some things we don't have on github or npm. I'll check this out tomorrow.

[matthewp]

Cool, I've added you as a maintainer.

[guybedford]

@matthewp this is amazing - worked first time... Do you know what they do about GitHub credentials for rate limits? Or do they use APIs that aren't limited. Anyway, it's working well and live!

[matthewp]

AFAIK when you publish to bower it stores a copy not unlike with npm, I don't think it pulls from GitHub. And since we're using their own module I think it should be fine.

My only question about this endpoint is what happens with dependencies. Since bower components can have dependencies of their own, when we install a package its deps will be placed in the tmpDir, but we don't move them to the final outDir. Will need to do some testing to see how this can break packages.

It would be great if it a bower's dependencies were automatically plugged into jspm's packages, is there a way to do this now or does there need to be a new API? Can we return it as the package.json?

[matthewp]

Arg, yeah this is wrong, should be grabbing their bower.json instead. That will contain dependencies, will jspm automatically know to fetch those?

[guybedford]

That sounds like quite a nice solution, since by putting it in the tmp folder, does that effectively mean we cache the dependencies for when they might be loaded?

Dependencies are specified with the package.json "map" config.

If you set the "map" config from the bower dependencies, this should work, but will need some parsing of the bower dependencies into a versioned package name format ({ map: { 'three': bower:[email protected]} } sort of thing).

I may need to check that map dependencies not found in the files are still downloaded... may need to adjust this.

[matthewp]

Ah, ok, then it doesn't make sense for me to return bower.json as the package since the format is going to be so different. All I need to do is return the main, files (possibly), and a map. Shouldn't be too hard, I don't think. Since bower has a flat dependency tree it shouldn't be too much trouble for you to use the tmpDir to grab the dependencies already there, if that's not too much trouble for you.

[guybedford]

Yes exactly.

What about Bower's internal cache here? Perhaps that will be doing this work for us anyway?

[matthewp]

Good question, not sure if they cache or not. npm keeps a ~/.npm directory but I don't see one for bower.

Bootstrap's bower.json is a good example, can just use the same version logic as in the npm endpoint to create the package/version string. What about the multiple mains though, is that something jspm supports?

[matthewp]

Moving the conversion over here since this (gist) issue is resolved.