mozilla / oghliner Goto Github PK

sw-precache allows you to specify a cacheId to distinguish between apps hosted at the same origin/path. That isn't a problem for apps hosted on GitHub Pages, since each app is hosted at a unique path (assuming a one to one relationship between an app and a GitHub repository). But it can be a problem when testing an app locally, if you typically start a local web server whose root dir is an app's build output dir, such that you load it from http://localhost/.

To avoid that problem, we should set cacheId to a unique name, like the name of the repository on GitHub.

fs.exists and fs.existsSync are deprecated: https://nodejs.org/api/fs.html#fs_fs_exists_path_callback

There are a few dependency install failures with Node.js 4.

60752941.txt

The "config" task to configure Travis auto-deploy currently requires you to go to your Travis profile page and activate the repository. But the Travis API for Hooks appears to be a way to do this programmatically, so the config task should be able to do it automagically.

It's a bit complicated to use that API, because the task would first have to request a GitHub token with a particular set of scopes (that are different from the single "public_repo" scope for the GitHub token we create to enable auto-deploy), then use that token to request a Travis token, which it could then use to authenticate with the Travis API.

Fortunately, there's the travis-ci module, which simplifies the process. After prompting you for your GitHub username/password, the task could use the authenticate method to get the Travis token, then use that to get the hooks, find the one for the repository in question, and activate it.

The config task already prompts you for your username/password, however, and it shouldn't do it twice. Unfortunately, the current prompt is generated by the get-github-token module, which doesn't expose the credentials. So we may have to reimplement that module's functionality in order to be able to prompt once to generate both tokens.

(We could also request a single token with all of the scopes that are required by either use case, but that would make a token compromise more damaging. Also, I think the GitHub token for getting a Travis token is a temporary one, while the GitHub token for auto-deploy is permanent.)

We should consider renaming the module to something descriptive, like offline-gh-pages, per these best practices for naming Node modules.

On the other hand, command-line utilities benefit from being short and sweet, and Oghliner exposes a command-line interface, in addition to its module interface. So perhaps it makes sense to keep the existing, compact name.

Travis auto-deploy appears to have regressed. After reconfiguring this repository, an auto-deploy attempt fails with "Invalid username or password":

2.05s$ [ "${TRAVIS_PULL_REQUEST}" = "false" ] && [ "${TRAVIS_BRANCH}" = "master" ] && gulp deploy
[20:39:53] Using gulpfile ~/build/mozilla/oghliner/gulpfile.js
[20:39:53] Starting 'deploy'...
[20:39:53] Deploying "update GitHub token for Travis auto-deploy"
[20:39:54] 'deploy' errored after 917 ms
[20:39:54] ProcessError: remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/mozilla/oghliner.git/'
at ChildProcess. (/home/travis/build/mozilla/oghliner/node_modules/gh-pages/lib/git.js:48:23)
at ChildProcess.emit (events.js:110:17)
at maybeClose (child_process.js:1015:16)
at Process.ChildProcess._handle.onexit (child_process.js:1087:5)
Done. Your build exited with 0.

This would make the oghliner deploy functionality useful also when you only want the deploy functionality and you don't use Gulp but a Makefile or a npm script.

At least the child_process.execSync call in the configure command requires Node 0.12+, and there may be other code that assumes Node 0.12+, so Oghliner's package.json file should explicitly require that version.

We should test the code in the Travis-specific branch of lib/deploy.js.

Building the template app should interject the Git revision ID (and tag, if the current revision is tagged) into the app somewhere subtle but visible for those who know where to look for it.

Then it'll be easier to see when the template app (and any app based on the template that doesn't remove the feature) has been updated in GitHub Pages, which could be useful when debugging the update flow (#5), especially since there's a delay of up to ten minutes (although it's usually less than a minute) between when you deploy an app and when it shows up on GitHub Pages.

Since the template app has a package.json file, and npm version updates the version in the file (while also tagging the current revision, per https://docs.npmjs.com/getting-started/publishing-npm-packages#updating-the-package), gulpfile.js could use var packageJson = require('./package.json'); to get the version number from the package for folks who are using npm version to version their apps, although that requires a manual step, and it'd be better to do this automagically.

Of note:

git rev-parse HEAD: the current revision ID
git describe --abbrev=0 --tags: the tag on the branch
git describe --abbrev=0 --tags --exact-match: the most recent tag on the current revision

(If we wanted to get fancy, we could distinguish between deployment of a clean working directory, where git status shows no changes, and deployment of a dirty one, where git status shows that there are changes that have not been committed yet.)

I've seen testBootstrap timing out in a couple of builds.

Oghliner currently supports building with Gulp, but a lot of webdevs use Grunt, so we should consider supporting it. I'm only a little hesitant because it would be easy for support for the two systems to get out of sync, such that we support features in one that we don't support in the other.

Alternately, we might support the same set of features, but one system is a second-class citizen (whether explicitly or implicitly), such that it doesn't get the same level of testing and maintenance.

Note that sw-precache's README - Example describes the project as having a "sample gulpfile.js", with many changes over the history of the project, but only a "basic Gruntfile.js", with only four commits in its history as of this moment.

Oghliner doesn't set sw-precache's ignoreUrlParametersMatching option, and that option's default value of [/^utm_/] means that sw-precache takes most query parameters into account when deciding whether or not to serve a resource from its cache.

That's a problem for apps like presentations using Shower, which set query parameters like full and lang in client code to store state in a way that is bookmarkable. If a user loads an app page from a URL with a query parameter that wasn't present when the app was initially offlined, then the service worker will not serve the page from the cache, so the load will fail if the user is offline.

To solve that problem, offline should set sw-precache's ignoreUrlParametersMatching option to [/./] so it ignores all query parameters when matching URLs to fetch against URLs in its cache.

(Being able to do this for all apps automagically is one of the benefits of focusing on the GitHub Pages use case. Since GitHub Pages sites are always static, and query parameters never affect the result of a GET request for a GitHub Pages resource, we can always do this when offlining apps specifically for deployment on GitHub Pages.)

When you deploy an app with the CLI, Oghliner configures gh-pages to clone the repository to .gh-pages-cache, so we should tell users to add that directory to .gitignore.

That's because gh-pages clones to a canonical location by default, but it leaves the clone around, and it fails if you try to deploy two different apps using the same clone dir.

That isn't a problem when you install Oghliner locally and use it from a build script, because the canonical location is within the app's own directory, f.e. node_modules/gh-pages/.cache.

But it is a problem when you install Oghliner globally and use the CLI, because then the canonical location is a system directory, f.e. /usr/local/lib/node_modules/oghliner/node_modules/gh-pages/.cache, so gh-pages would fail when you tried to deploy a second app.

Thus the custom local clone directory. Which shows up as an untracked directory in git status, and which we should therefore tell users to .gitignore.

The output of the command that configure adds to the after_success script in .travis.yml is collapsed by default in Travis build logs. So you only see a single line like this when you first view a log (like this one):

2.87s$ [ "${TRAVIS_PULL_REQUEST}" = "false" ] && [ "${TRAVIS_BRANCH}" = "master" ] && gulp deploy

You can see the rest of the log by clicking on the disclosure widget (expander arrow) to the left of that line item, after which you see the output of the command, which might look something like this:

[23:31:35] Using gulpfile ~/build/mykmelez/mc-choir/gulpfile.js
[23:31:35] Starting 'deploy'...
[23:31:37] Finished 'deploy' after 1.91 s

But it would be better if Travis showed you that output by default, so you didn't have to expand the command to see it.

The Oghliner website at https://mozilla.github.io/oghliner/ was designed using the GitHub Pages generator, and it could use a redesign (both graphic design and information design) in order to better convey what the project is about and how to go about using it.

This used to be identical to #7, because we were using the website as the template for bootstrapping new apps. But now we've separated those two things, so the website is in app/, while the bootstrap command uses the template in templates/. Thus redesigning the website is now a separate concern from redesigning the template.

It was introduced in #63, because without it the test is too slow or fails on Travis.
We should either remove the option altogether and always install dependencies (which is the option I prefer) or enable it only when running tests on Travis, where we don't really need the tests to be too fast (and installing dependencies is usually very fast on Travis, so it shouldn't really be a problem).

With rootDir=".", deploy tries to deploy everything from the root directory, so also the node_modules directory.

We could suggest using a different directory in the README.

The built-in Oghliner template app in its app/ subdirectory, which GitHub publishes to https://mozilla.github.io/oghliner/, is too basic. It doesn't contain any images nor CSS files, so there's no guidance about where to put those files in the source directory structure. Nor are there other common files, like favicon.ico or robots.txt.

Nor does it even explain the Oghliner project or point to more info about it. I might not expect that info in a template that gets created by a command-line tool, but I would expect it in a template that is itself published as an app, which is the case here.

By contrast, compare to the app/ subdirectory of the Yeoman gulp-webapp template (mkdir my-app && cd my-app && yo gulp-webapp), which contains:

apple-touch-icon.png
favicon.ico
fonts
images
index.html
robots.txt
scripts
scripts/main.js
styles
styles/main.scss

Or the Angular template (mkdir my-app && cd my-app && yo angular), whose app/ subdirectory has:

404.html
favicon.ico
images
images/yeoman.png
index.html
robots.txt
scripts
scripts/app.js
scripts/controllers
scripts/controllers/about.js
scripts/controllers/main.js
styles
styles/main.scss
views
views/about.html
views/main.html

Or the template built into Ember (npm install -g ember-cli && ember new my-app), whose app/ subdirectory includes:

app.js
index.html
router.js
styles
styles/app.css
templates
templates/application.hbs

We should flesh out the Oghliner template app with a set of files that is more representative of the apps that users are going to build. Also, since the template app is itself published by GitHub as an app, we should include content that is useful to users who browse that site.

Perhaps we can use GitHub's Automatic Page Generator to generate an attractive basic page for the template, per Instantly Beautiful Project Pages. And then add content similar to the content in the README (or a basic description of the project along with a link to the README for further info).

When deploying an app, it should be possible to specify a message for the commit to the gh-pages branch. And when auto-deploying from Travis, the auto-deploy script should derive that message from the one for the commit to master that triggered the deployment.

The after_success script is executed for each job (e.g. each Node.js version specified in .travis.yml), which means auto-deploy is executed multiple times.

The deploy command currently hides your GitHub token when auto-deploying via Travis by not specifying a logger if it detects that a GitHub token has been specified in an environment variable. But the gh-pages submodule has a silent option specifically for this use case, so we should use it.

We need to figure out how to test the configure command. I confess to having no good idea how to do this.

The command depends on the GitHub and Travis APIs, so perhaps we could stub those.

Or I guess we could create a GitHub user specifically for testing, then encrypt that user's username and password for Travis. That still wouldn't allow us to test 2FA, nor perhaps some other aspects of the configuration flow, but it could test many aspects. However, it would be a live test, dependent on the status of the GitHub and Travis API endpoints.

When the configure command doesn't find your repository in Travis, it tells Travis to sync it and then waits for the sync to complete (periodically polling Travis to find out the status of the sync). But if Travis is already syncing your repository when configure tells it to sync, then the travis.users.sync.post call throws { message: 'Sync already in progress. Try again later.' }, and the command doesn't catch that error, so it simply reports the error and dies:

Checking the status of your repository in Travis…
I didn't find your repository in Travis. Syncing Travis with GitHub…
{ message: 'Sync already in progress. Try again later.' }

It should instead catch that error and call the travisAwaitSyncing polling function in that case, just as it does in the case of the repository not being found.

The Oghliner template should be available via Yeoman, so you can use that tool to bootstrap an app.

The "config" task, which configures auto-deploy via Travis, should be a command-line utility rather than a task, since you only ever need to run it once per app.

Once we start testing the commands (#22, #34, #35, #36), we should add a Travis build status badge to the README.

Oghliner's output is quite simplistic, just plain text without any special formatting. By contrast, other popular command-line tools, like Yeoman (npm install -g yo && yo), use color, boldness, and other stylistic formatting to clarify the text they output to the console. (Yeoman even prints ASCII art, although that's a bit extraneous.)

We should evaluate the output of the various Oghliner commands for opportunities to use formatting to make them clearer.

• Use gutil.log to log messages (#131)
• Print colored warnings and errors (#131)
• Clarify output of configure (#148)
• Print large file warning in yellow (#151)
• Print all suggestions in blue (#156)
• Clarify output of bootstrap (#160)
• Clarify output of integrate (#161)
• Clarify output of offline (#162)
• Clarify output of deploy (#163)

sw-precache sets its maximumFileSizeToCacheInBytes to 2MiB by default, which breaks offlining of large image files, even if you explicitly include them via file globs. I understand the reasoning behind avoiding the caching of large files, especially if they aren't actually necessary for your app to function. But it does complicate and undermine the promise of Oghliner: that you can point it at your app's assets, and it'll ensure they're available offline.

Instead of simply excluding large files, we should warn the user when offlining them and give them the opportunity to exclude them from the cache, f.e. by allowing entries in fileGlobs to exclude files instead of merely including files. Then a user who is warned about a large file can choose to cache it anyway by doing nothing or explicitly exclude it to prevent it from being cached, f.e. by using a leading exclamation point to negate the match à la minimatch:

{
    "fileGlobs": [
        "!images/my-large-file.png",
    ]
}

We might even let users explicitly include a file to suppress the "large file" warning:

{
    "fileGlobs": [
        "images/my-large-file.png",
    ]
}

This recent build on Travis failed a hash check for the multiparty module:

make: Entering directory /home/travis/build/mozilla/oghliner/node_modules/travis-encrypt/node_modules/ursa/build' CXX(target) Release/obj.target/ursaNative/src/ursaNative.o CXX(target) Release/obj.target/ursaNative/src/asprintf.o SOLINK_MODULE(target) Release/obj.target/ursaNative.node COPY Release/ursaNative.node make: Leaving directory/home/travis/build/mozilla/oghliner/node_modules/travis-encrypt/node_modules/ursa/build'
npm ERR! Linux 3.13.0-40-generic
npm ERR! argv "/home/travis/.nvm/versions/node/v0.12.7/bin/node" "/home/travis/.nvm/versions/node/v0.12.7/bin/npm" "install"
npm ERR! node v0.12.7
npm ERR! npm v2.11.3

npm ERR! shasum check failed for /tmp/npm-2066-5cbdf385/registry.npmjs.org/multiparty/-/multiparty-2.2.0.tgz
npm ERR! Expected: a567c2af000ad22dc8f2a653d91978ae1f5316f4
npm ERR! Actual: 6055a8df1315811def83ca6585d46cd476845953
npm ERR! From: https://registry.npmjs.org/multiparty/-/multiparty-2.2.0.tgz
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR! https://github.com/npm/npm/issues

npm ERR! Please include the following file with any support request:
npm ERR! /home/travis/build/mozilla/oghliner/npm-debug.log

The command "npm install" failed and exited with 1 during .

Your build has been stopped.

A rebuild succeeded, so you won't see that error on Travis now, but here's the log file:

log.txt

Presumably this is a rare build failure, and nothing to be concerned about, but filing it here just in case it happens more frequently, and we should dig into it further.

We should release the template files to the public domain, so web developers who use them to bootstrap an app can choose their own license for the app. In some cases, this will require replacing template code that we obtained from other sources (like the GitHub Pages generator). The headers for files in the public domain are at the bottom of this page: https://www.mozilla.org/en-US/MPL/headers/.

We need to figure out how to test all the commands. deploy and configure might be harder, because they interact with servers; but offline at least should be doable, so let's start there.

c19f3ff introduced gulp.env usage to gulpfile.js, but gulp complains: "gulp.env has been deprecated. Use your own CLI parser instead. We recommend using yargs or minimist." So I guess we should use yargs (or minimist) after all. Note that cli.js uses commander, but that seems like overkill and a mismatch for this use case.

Travis auto-deployment requires Git to be configured with the name and email address of the user who will commit the deployed files to the gh-pages branch, and while the name can be arbitrary, the email address has to be the one associated with the user whose token is being used to authorize the deployment.

But the configure task doesn't configure Travis to configure Git with this info. It should do so.

For example, in my .travis.yml file, I manually configure Travis to configure Git via a section like the following:

before_script:
  - 'git config --global user.name "Travis-CI"'
  - 'git config --global user.email "[email protected]"'

The configure task should determine the user's email address and use it to configure .travis.yml similarly.

Sometimes images are conditionally downloaded, particularly responsive images that use <img srcset> to make the browser determine which one of a set to load, as described in Responsive Images: If you’re just changing resolutions, use srcset..

I'm not sure how often they're used in apps generally, much less offline-able apps hosted on GitHub Pages, but they're mentioned in Introduction to Service Worker at least, so at least some folks have spent some time thinking about how to handle them when using Service Worker to cache app assets.

Since the promise of Oghliner is to simply cache your whole app offline, I'm loathe to complicate the interface with conditional/lazy caching of assets. But perhaps it would make sense to include some support for caching one of several variants of an image.

For example, we might let users specify assets that shouldn't be cached immediately but should be cached once requested, something like:

{
    "filesToCacheLazily": [
        "images/logo-lo-res.png",
        "images/logo-hi-res.png",
    ],
}

Then the worker can cache whichever variant the browser ends up requesting.

Before we do something like this, however, we should have better use cases for it, ideally in the field.

GitHub Pages processes output with Jekyll by default, but it isn't clear how likely it is for offline web apps to use this feature, and processing files with Jekyll can introduce unexpected behavior. It may also cause GitHub Pages to take longer to deploy your app than it otherwise would.

So the template should include a .nojekyll file to turn off Jekyll processing. And it should configure the gh-pages module to include that file when deploying to GitHub Pages.

Of course we should document this feature so folks know about it (and how to disable it after bootstrapping apps that are going to use Jekyll).

We should figure out how to test the bootstrap command. Like offline (#22), its effects are all local, so this should be straightforward.

Oghliner should support an app update user flow, so an app that uses Oghliner receives a message when an update is available and can prompt the user to update the app (if it chooses to do so, and at the appropriate time).

The flow would be optional, so an app that wants updates to be completely automatic and silent can simply ignore the message and never prompt the user to update the app, which will then update itself automatically whenever the service worker is updated by the browser (which is currently the default and only supported flow).

See sw-precache's demo service-worker-registration.js file for inspiration, but ideally we would not have apps add their custom logic to our own equivalent (service-manager.js) but rather expose the registration to their code in a way that makes it possible for them to listen for its events and handle them in their own script.

We need to figure out how to test the deploy command.

We can create a local Git repository and bootstrap an app in it to deploy, but then there's the question of how to test that deployment pushes it to its origin remote, which is normally on GitHub.

But deploy doesn't actually depend on GitHub itself, only on there being an origin "remote", which could presumably be another local repository. So it should be possible to test deploy without a dependency on a GitHub server.

We would just need to create two local repositories, one of which has the other as its origin; build the app in the first repository; and then test that the deploy command pushes the build output to the gh-pages branch in the origin repository.

The configure command assumes that you want to auto-deploy merges to the master branch on the origin remote.

But for projects with multiple contributors, it's common for a user to fork the original repo and clone their fork, such that the origin remote points to their fork, and then create an upstream remote for the original repo, as described in GitHub's Fork A Repo documentation.

And it's actually more interesting to configure auto-deploy for the original, upstream repo, since that's the one in which people will more often merge changes via pull requests and want those merges to trigger auto-deploy.

But it's hard to do that right now. You have to either temporarily rename your remotes so the original remote is named origin (which actually sounds like a better name for it, although it is unconventional) or clone the original remote and configure that clone.

We should make it easier to configure auto-deploy for the upstream repo.

For example, we could prompt the user to choose the remote for which to configure auto-deploy, selecting the upstream remote by default if one is defined.

(A related problem is that you can't configure a repo to auto-deploy both the original repo and a fork, since the encrypted GitHub auth token in .travis.yml is repo-specific, and there can be only one auth token in the Travis build environment variables. But that's a different issue.)

When integrating offlining/deployment into an existing app, the clunkiest part of the process is integrating the offline-manager.js script into the app. You have to copy the script from an obscure location in local or global node_modules/ into the appropriate place in the app's source tree:

> cp node_modules/oghliner/app/scripts/offline-manager.js app/scripts/

Then you have to add a <script> tag for it to the appropriate place in your HTML page(s) or page template(s):

<script src="scripts/offline-manager.js"></script>

By contrast, when bootstrapping an app from the template, this happens automagically.

We can't make integration as automagic as bootstrapping, but it can be easier than it is now.

First, we should add a command that copies offline-manager.js to the appropriate place without requiring the user to know its source location, something like oghliner *some-command* *destination*, i.e.:

> oghliner copy-offline-manager-to app/scripts/
Copying offline-manager.js to app/scripts/… done.

Second, we should consider auto-detecting the HTML page(s)/template(s) for the app and offering to auto-add the <script> tag to them, if we can guess the right location reliably enough to make it more helpful than a hindrance:

I've detected your app at app/index.html.  It needs to load the script offline-manager.js
in order to register the service worker that offlines your app.  I can add a <script> tag
that loads that script into app/index.html.

Add &lt;script tag to app/index.html (Y/n)?

Otherwise, we should at least tell the user they need to add the tag themselves and print it to the console so they know what to add:

Your app needs to load the script offline-manager.js in order to register the service
worker that offlines your app. To load the script, add this line to your app's HTML
page(s)/template(s):

    <script src="scripts/offline-manager.js"></script>

Perhaps we should do both of those things with a single command, f.e. integrate, which first prompts the user to specify the destination for offline-manager.js and then offers to add the <script> tag to the right file(s) if it thinks it can determine those reliably.

(An integrate command for existing apps, like its bootstrap counterpart for new apps, might eventually prompt the user to do other optional things, like configure Travis auto-deploy.)

Once #84 lands, we should add a code coverage badge using a service like https://coveralls.io/ or https://codeclimate.com/.

When developing a simple app that doesn't require a build step, it's tempting to put the app's files at the top-level of its repository, since that is the simplest possible directory layout:

• my-project/
  • gulpfile.js
  • index.html
  • script.js
  • style.css

And Oghliner can do a reasonable job of offlining such an app, since you can specify that rootDir is the current directory (./) and put the list of files to offline into fileGlobs.

But deployment is problematic, since the gh-pages submodule that Oghliner uses to deploy to GitHub Pages has a more limited mechanism for specifying the files to deploy, a src option that is a single minimatch string, and Oghliner doesn't currently expose it.

So if you deploy the current directory for such an app, then gh-pages will also deploy gulpfile.js and any other meta-file that is present in the directory even though it isn't actually part of the app itself.

Of course it's a good practice to put app files into a subdirectory like app/ and then build them into another directory like dist/ (even if "build" simply means copying the files). And the template does that by default.

But Oghliner should nevertheless support specifying the files to deploy at least to the extent that supports specifying the files to offline, i.e. via a set of file globs, to satisfy the "simple app" use case (and presumably others).

I don't know if we're compatible with Node.js 4, but when we are we should run tests both with 0.12 and 4.

This is an interesting request to be able to maintain a list of files to offline that is separate from the service worker: https://jakearchibald.com/2014/offline-cookbook/#comment-2189777576.

It's basically the appcache use case. I think i recall @marco-c suggesting that we could enable users of our various offlining tools to maintain a list of assets in a separate file. We'd still need to modify the service worker somehow on each update, since service worker updates depend on that file changing; but at least we could minimize such changes.

On the other hand, perhaps the list of file globs in the build file is sufficient to satisfy users who want to maintain such a list, since it is essentially that list of files, and it can consist entirely of specific files; it doesn't have to use globbing characters.