xuset / planktos Goto Github PK

The website needs some tweaks to make it look a bit more legit.

Tweaks:

• Fix favicon
• Planktos logo should be in the top left
• Add a contact us section
• Remove unnecessary empty space between sections
• Add Google analytics (make sure Planktos bugs are tracked)
• Add a big "Download Planktos" link pointing to the GitHub page after the about section
• Make text larger
• Fix unreferenced google error

Concerns:

• Not enough information is displayed on the initial page
• Perhaps the about section could be formatted better (maybe tiles or something?)
• The link to our GitHub should be available from anywhere in the website
• Background image is not really relevant and takes up a large amount of space

The lib shouldn't require the service worker in order to function.

• The webpages should assign a downloader, not the service worker. (Use tab-elect or equivalent)
• The downloader shouldn't use the service worker url to determine the webseed url

When update() is called, it makes 4 http requests which makes calling update() frequently infeasible. The files requested everytime update() is called:

• planktos/root.torrent
• planktos/manifest.json
• planktos/install.js
• planktos/planktos.min.js

It is better to only request planktos/root.torrent initially, and if the torrent is new then request the remaining files.

We should have a Travis CI build tag, NPM tag, and maybe a donate button...

This should make us look like professionals!

It looks like updated content is not recognized, resulting in stale data being loaded, even if the server is serving new content.

Deleting locally stored content seems to fix the issue.

Snapshots should automatically be cleanup when they are stale or the cleanup functionality should be exposed in the api.

If the current SW gets a refresh page request, it initiates the download for the torrent. When the SW is bounced, the new SW re-issues the download request, because the state of which client was assigned the download is gone.

Hello,
can Planktos be used with external/existing torrent files?
Since AWS s3 offers a torrent interface naively (https://docs.aws.amazon.com/AmazonS3/latest/dev/S3TorrentRetrieve.html) by simply adding a ?torrent to an s3 request I would appreciate to use this functionality to serve files either over torrent/Planktos or out of the s3 bucket.

The current way the downloader is injected into web pages leaves a lot to be desired. This issue is more of a tracking issue than one that provides a solution.

When a new page is opened, it must be able to download torrents by including the downloader code. This is currently accomplished by replacing the initial request with one of the injections found in ./lib/injection, the injection then makes a request for the actually requested content and displays it.

Some problems with this, iframes should not used form html pages, but there is no way to know definitively if the requested file is a html file (looking at the file extension only works if the extension is present). Using document.documentElement.innerHTML = text to replace the injectors html with the requested files html is a hack and can hopefully be improved. Ideally the improved injector would be less hacky, and be able to stream the requested content.

One potential improvement would be: if the requested file was html then edit the contents to include the downloader during planktos.fetch()

If only one file is passed to the planktos cli then a bad torrent is generated. The files section of the bad torrent is something like:

  "files": [
    {
      "path": "planktos",
      "name": "planktos",
      "length": <length>,
      "offset": 0
    }
  ]

When it should be:

  "files": [
    {
      "path": "planktos/<sha256 sum>",
      "name": "<sha256 sum>",
      "length": <length>,
      "offset": 0
    }
  ]

This discrepancy is likely releated to create-torrent treating the creation differently when it is a single or multi-file torrent

What do we need to accomplish before we releasing an MVP?

• Blocking auto-updates
• Clean up old snapshots
• Polish up website
• Create a blog (which uses Planktos)
• Write a blog post about integrating Planktos into an existing blog

Version 1.0.5 of run-parallel-limit was just published.

This version is covered by your current version range and after updating it in your project the build failed.

run-parallel-limit is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

Larger websites should be sharded into multiple torrents such that the files grouped in an individual torrent have high locality relative to each other. This makes it likely that a user visiting a specific portion of the website will need files from a torrent that has already been downloaded.

If we view the website as a graph, with files corresponding to nodes and links to edges, this optimization corresponds to finding the minimally connected subgraphs larger than some size.

To Do:

• Add a section about avoiding downtime (it is severely underplayed currently)

When I was trying to integrate Planktos into our landing page I ran into a problem: the static files are relative to the root directory, but the actual website is relative to /planktos/. Planktos cannot handle websites like this, because it assumes that the service worker is going to be relative to the root directory.

Version 3.4.0 of mocha just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As mocha is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

Opera should work with planktos out of the box since it has all the APIs planktos uses, but it may require a few changes to the code. Once Opera is passing the tests, it should be added the ci tests, and Opera support should be listed in the gh project pages.

The CLI should have it's own unit tests

If the service worker is active, a new tab is opened, and the service worker logs messages the messages will end up in the first tab. The logging should be to the correct tab.

Currently, each file is downloaded in the order it appears in the torrent. This can cause problems like large files being downloaded first before they're needed while the much needed index.html is the last file in the torrent and the last to be downloaded.

WebTorrent allows the priority of a given piece range to be changed such that pieces with a higher priority are downloaded before those with lower priority. So when a file is requested, a message should be sent to the web page that is downloading the torrent to instruct it to raise the priority of the requested file's pieces.

As we start trying to accelerate development on the project, and potentially get interest from other people, it will be necessary to have the unit testing framework set up.

Setting these values correctly helps the browser and possible end user detect what type of file is being read.

The ~~~service worker~~~ fetch() method should set the Content-Type header for responses that are for files in the torrent.

Also when blobs are created we should also construct the blob with the correct mime type.

The very helpful mime npm can do the mime type detection

The landing page should be served over planktos if the browser supports it. Also a live demo section should be added that informs the viewer that the page was downloaded from peers.

When an update is found a new snapshot is created, the snapshot is downloaded from scratch despite the high likely that older snapshots already contain some of the needed data. This should be fixed by iterating over old snapshots and seeing if the old and new snapshot have any pieces in common by comparing the piece's hash. If the pieces are identical then copy the piece from the old snapshot's chunk store to the new.

In previous discussions we have concluded that a simpler public interface would be beneficial and include the following methods:

• constructor
• update
• fetch
• getFile

The remaining public methods should be removed from the public interface.

At this point update() should also be modified to remove the old snapshots once the new snapshot is loaded in.

Related to #45

Planktos should use WhatWG ReadableStreams in order to allow content to be streamed from IndexedDB and into the fetch event's respondWith. This will greatly speed up page load times and allow for video/audio streaming.

Reading from indexeddb-chunk-store with a nodejs stream while the file is not fully downloaded was added in #28 which is the bulk of the work.

What is needed for streaming is:

• Convert nodejs streams to web streams
• Use web streams when available for respondWith
• Add support for byte range requests #6 (only needed for video and audio streaming)

Open a new tab and look at the clients registered to the SW.

Version 3.31.0 of create-torrent was just published.

This version is covered by your current version range and after updating it in your project the build failed.

create-torrent is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

• Integrate Planktos into the build process for the landing page
• Ensure that Planktos errors are captured in analytics
• Fix script fetching errors

The service worker should not contain the fetch() code or other lib related code. It should just contain the bare minimum while the bulk of the logic is contained within the lib.

To further improve the customization, the service worker should use importScripts instead of require. This allows the user to customize the sw without having to run a bundler like browserify. Doing this will drastically shrink the sw file to < 1kB so the minify is no longer necessary and can be removed from the build script.

Blocked by #35

The public API is quite simple; however, it is not clear what our contracts to the user are.

For example:

• getSnapshot not only gets the latest snapshot, but it also seeds the file if it wasn't being seeded before
• getSnapshot can only be called after update
• fetch and getFile have the same side effects and constraints as getSnapshot

Some of this may be a problem with our API; however, documentation is necessary regardless of whether or not these are issues.

The CLI currently does not conform to the expected user interface. Specific concerns:

• Installing the library should be one command
• Bundling files into a torrent should be a separate command

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

• Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

Your Greenkeeper Bot 🌴

Updating while the service worker has clients or files are being downloaded will likely break things currently. This issue needs investigating and fixing.

Previous Discussion in #53

Terminology: Intercept mechanism refers to the way in which a request is intercepted e.g. service worker or tag editing. Intercept handler refers to the piece of code which decides how an intercepted request should be handled.

To Do:

• Add a folder for different implementations of intercept handlers (for right now it should only contain default.js)
• Separate out the intercept handler from sw.js into default.js (service worker specific code should remain)
• Replace the Planktos fetch interface with something that is agnostic to intercept mechanism (right now it accepts a FetchEvent)
• Add a flag for selecting which intercept mechanisms to use
• Add a build phase in the CLI that calls the tag editor
• Inject the Planktos install script into every JS file during build phase
• Create a standard for modifying resource tags (webtorrent/webtorrent#879 (comment))
• Handle editing of all tags (<audio>, <base>, <iframe>, <img>, <link>, <object>, <picture>, <source>, <track>, <video>)

Website:

• It was unclear that active users of the website were seeders (question asked: why is this better than serving your website with S3)
• A reader wanted an example
• Need to clarify how it actually works
• Tagline should not use the word "seeders"

Scratch Notes:

It seems that we need to ground the readers of the website a bit better. We will have to pitch both WebTorrent and Planktos.

A place to keep thoughts and track our plans for Planktos.

The CLI should have a flag that allows the user to specify a custom tracker.

Passing the array/string that minimist generates into the opts for create-torrent will accomplish this.

As a side note, if a boolean false is passed into this option then create-torrent includes no trackers in the torrent meta. This is useful when testing so we are not bombarding trackers with our announces every time we run a test. The tests don't use trackers so there really isn't a point in announcing to them.

Pressing:

• Use Planktos
• Fix Planktos

Content:

• Remove the word "seeders"
• Explain different use cases
• Make the website sound less anti-user
• Put more content on the main page

Experience:

• Put an icon selector tab in the about section
• Animate the down arrow

We need to integrate PermaTorrent into Planktos.

If the intercepted http request has the RANGE header we should only return the bytes in the specified range as well as any associated response headers. The main reason to support this is video and audio seeking; with each seek the browser sends a new request for the selected range.

onFetch in sw.js will have to be modified and it will probably be a good idea to modify getFileBlob in index.js to support specifying the start and end bytes.

The Content-Type attribute should be set Snapshot.fetch

Currently, a request for GET / will be treated the same way as GET /index.html. The problem is that this only works for the root directory e.g. GET /a/ will not be treated as the same as GET /a/index.html. This needs to be fixed so it works for all directories. My guess for the best place to add the necessary code would be in somewhere index.js:getFileBlob()

Using the service worker to intercept HTTP requests has many benefits; however, for our initial use case, blogs and small websites, it is not as well suited as modifying resources in the HTML. Thus, we need to investigate tag editing and decide on an approach that allows us serve this initial use case and scale to more complex use cases as well.

Benefits of Service Worker:

• Intercepts all network requests
• Better API
• Doesn't require manipulating user code
• Works even if the web server is down (for static websites)
• Loads the index page over HTML after the SW is installed
• Early in the adoption curve

Benefits of Tag Editing:

• Initial website load can be done over WebTorrent (except for index file)
• Videos can be streamed in browsers other than Chrome
• Doesn't require including a script in the index file

It should be noted that these approaches are not mutually exclusive; I think it makes sense to enable the service worker and build-time tag editing by default. However, the code for both of these systems are completely unrelated, so they should be separated into different repositories.

Version 3.3.0 of mocha just got published.

This version is covered by your current version range and after updating it in your project the build failed.

As mocha is “only” a devDependency of this project it might not break production or downstream projects, but “only” your build or test tools – preventing new deploys or publishes.

I recommend you give this issue a high priority. I’m sure you can resolve this 💪

Your Greenkeeper Bot 🌴

The documentation for specifying a web seed and recursive files should be in the README.

Integrate uglify-js into the build process. If the inline source maps that browserify generates cannot be included in the minified js then it is probably best to have two build processes, one for production and the other for debug.

Instead of creating a project that (in my view) looks like (without advanced features) ZeroNet, why not contribute with them?
https://zeronet.io/