burtonator / polar-bookshelf

Polar is a personal knowledge repository for PDF and web content supporting incremental reading and document annotation.

Polar Bookshelf

Polar is personal knowledge repository which supports advanced features like incremental reading, annotation, comments, and spaced repetition. It supports reading PDF, EPUB, and the web content.

Features

• PDF support We have first-class PDF support thanks to PDF.js. PDFs work well when reading content in book format or when reading scientific research which is often stored as PDF.

• EPUB support Add any EPUB and we also capture web content to EPUB/

• Captured Web Pages Download HTML content and save them as offline documents which can be annotated.

• Pagemarks Easily keep track of what you're reading and the progress of each document.

• Text Highlights Highlight text in PDF and web pages.

• Area Highlights Capture a region of the page as a highlight which can be a chart, figure, infographic, etc.

• Portable Run across any platform. Linux, MacOS, and Windows supported.

Downloads

Packages for Windows, MacOS, and Linux are available on the downloads page.

We also have a CHANGELOG available if you're interested into what went into each release.

Discussion

We have both a Discord group and Reddit group if you want to discuss Polar.

If it's a very technical issue it might be best to create a Github Issue.

Contributors

This project exists thanks to all the people who contribute.

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

License

Polar is distributed under the GPLv3.

PDF.js is available under the Apache License. Electron is released under the MIT License. Rest of the code is MIT-licensed.

polar-bookshelf's Issues

Write up basic design of a sidebar for viewing annotations in a document

Make my ReactSummernote.tsx changes OSS ...

Redesign the capture page to function like a browser.

Use this sample app as a starting point:

https://github.com/hokein/electron-sample-apps/tree/master/webview/browser

While the page is loaded make the page grey via a filter and disable click navigation while it loads.

Once it's finished loading... capture it and create the PHZ file and then load it off disk.

Consider using another key binding library

https://github.com/madrobby/keymaster

https://dmauro.github.io/Keypress/

https://github.com/RobertWHurst/KeyboardJS

Key bindings might not be working after initial document capture.

Setup tslint

We need to setup tslint with a decent profile to keep our typescript tidy.

I'm sure it will spot some issues with my existing code as I'm running pretty fast.

Validate all Polar functionality on Windows.

I'm still not sure about all of the Windows functionality. I think all the key bindings should work and that MacOS and Linux work fine but we need to install it on Windows and make sure everything works there.

Feature: Kanban view for which books to read would be great!

https://webix.com/kanban/

Use quality fonts in the create flashcard app...

I love these fonts

https://react-bootstrap.github.io/components/progress/

No meta charsets in capture

// FIXME: don't allow meta charset and other ways to set the charset within the
// HTML file as we are ALWAYS UTF-8 since we're sending it to the caller
// in JSON.  For example, if the encoding is ISO-8859-4 from teh original
// source , and sent to use that way, we would store it that way but
// re-represent it as UTF-8 which is then invalid.

will-navigate isn't being fired when clicking URLs. URL not opened in main browser.

This is a big problem because it's a security issue. The URLs being loaded aren't safe when nodeIntegration and other settings are enabled in Chrome.

Consider some sort of protection / guard that allows only file: URLs and URLs running on localhost.

Contribution instructions / help tasks

We're having people suggest how to help out but not all of them are coders.

We should create a page to allow people to help out:

• make a donation
• participate in the forum
• test windows builds
• provide build feedback
• write documentation
• help with marketing (where should we post, post a review)
• post your feedback and suggestions for polar

Opening saved .phz doesnt work

System: Windows 10 64bit
Used polar-bookshelf-setup-1.0.0-beta22.exe from releases folder

Steps to reproduce:
open app,
capture web page, example https://en.wikipedia.org/wiki/Anki_(software)
close app, open again
click File -> Open, choose captured web page from .polar/stashes

stash.zip

Feature: implement text row joining for the new highlighter like we had for the old version.

Right now we join 'rows' in the old highlighter so that the highlights are across multiple rows of text and we get one large text highlight instead of independent rows.

I think this style is better but we might also want to solicit feedback.

Where is the documentation?

Where is the documentation on how to set this up with an Electron project? I am trying to load a blob into this viewer and can't find any documentation on how to even setup Polar Bookshelf.

EDIT: My blob resides in Electron's main process, so I am expecting something along the lines of creating a new BrowserWindow, but whatever Polar Bookshelf is when you import it and then PolarBookshelf.loadurl(blob)?

Implement 'sandbox' on iframes

I think for now we just want to disable scripts:

https://www.w3schools.com/tags/att_iframe_sandbox.asp

I think want to disable everything but allow:

allow-top-navigation allow-same-origin

... this should basically enable all the features we want and prevent any issues with accidental scripts being added.

Feature: Basic text and area highlight colors.

It would be nice to support different colors for highlights.

We should support just three for now I think. red, yellow and blue.

One thing I'm worried about though is having them conflict with pagemark colors and if we implement new features I haven't though of before we might not be able to tell the difference between annotation types in some documents.

Feature: It should be possible to add high level notes and comments to the document

We want per page notes but it would be to have overview notes too. At the per document level.

These could be shown at the top of the document but also in the repository viewer.

Remove 'legacy' highlighter system when used in pdf.js

We want to migrate to the new version completely but the old version works on pdf.js where the new version doesn't. Just fix the new version to support row joining like the old version.

The other major problem is that the positioning doesn't work properly for some reason.

Implement support for ad blocking

https://github.com/quanglam2807/electron-ad-block

This package uses the brave ad blocking system so it's probably decent.

We would need to integrate it and if it works REALLY well we could enable it by default but for now we don't have a configuration system to allow users to enable/disable it.

Evaluate MobileDocKit for use in Polar. Looks interesting

https://bustle.github.io/mobiledoc-kit/demo/

Implement shields.io badging for README

add pricing (free), license (GPL), github stars, and a dicord button.

Captured page is empty

(Windows 10)
Steps to reproduce:

open app, File -> Capture web page
insert http://www.faz.net/aktuell/ (german newspaper)

produces empty page, .phz which is about 3kb

log file:
127.0.0.1-1532024565294.txt

URL takes forever to download.

https://www.wired.com/story/guerrilla-wikipedia-editors-who-combat-conspiracy-theories/

Allow for customizable keyboard shortcuts

I believe Unity by default maps Ctrl + Alt + T to open a Terminal window. Would be nice to be able to change the shortcut within Polar Bookshelf instead of having to change it within Ubuntu's system settings.

Get 32 bit builds working for Windows.

Implement PROPER spectron tests of various browser infrastructure components.

We currently don't actually test electron components because Spectron is a REAL pain to work with.

I think the workaround is going to be to set a window.test_results value and just read that from mocha.

this way there's a standard way for any Electron code to update the test results and we can test more functionality of the electron core.

Content capture only works from console. Not in UI

Resizing PHZ windows causes the iframe to have multiple scrollbars.

Feature: Temporarily disable pagemarks and restore them without changing on-disk representation.

There are times where you want to update an annotation but don't want to delete the pagemark.

Feature: capture image of text that I highlighted / annotated so that it can be sync'd to Anki

Generate mockups of proposed final UI including flashcards, sidebar nav, and repository page.

This way I can get feedback from other people.

Migrate to Electron 3.0

They are on Beta 3 and this gets us to Chrome 66.

Migrate to Typescript and start cleaning up code

Feature: Text selection on Electron in pdf.js is horrible.

The text selection is excellent in Firefox but is broken in pdf.js.

I created a bug for it and posted it to pdf.js but I don't think they're going to be proactive about fixing the problem.

We might be able to build our own text selection system which just prevents updating the selection outside of the main textArea.

Polar processes not terminating properly when the main window closed

Machine:
XPS 13, i7-8550U @1.8GHz
16GB Memory
512GB NVMe SSD
OS:
Ubuntu 18.04.01 running kernel 4.17.14

Issue:
When closing application after opening a file, the polar processes are not terminated properly. Application cannot be opened again until the processes are killed manually. (See attached screenshots)
If the application is closed without opening any files, then all polar processes terminate properly.

Steps to Reproduce:
Open Application: spawns 4 polar processes (Screenshot_1)
Open any pdf or phz document; spawns additional polar process (Screenshot_2)
Close application (close box or File/Exit): additional book process is terminated, however, the original 4 polar processes are not terminated preventing the application from being reopened until these processes are manually killed. (Screenshot_3)

There are no log files created and no OS exceptions thrown

Screenshot_1-3.zip

Amp URLs can't be fetched and break capture.

It's disabled for now but AMP URLs can't be fetched.

Implement flashcard data input (web components)

We mostly have this finished now and really just need to migrate them to web components.

Use device emulation and scale to resize windows?

This might be a valid solution but I'm not sure I can change it on the fly...

Ability to change the title of a document

Investigate using preload scripts and service workers instead of interceptStreamProtocol for content caching.

It might be possible to implement our caching via service workers and inject them into a session via preload.

This would be a more standardized way of implementing this in Electron and probably better supported.

https://developers.google.com/web/fundamentals/primers/service-workers/

https://github.com/electron/electron/blob/master/docs/api/session.md

Area highlights not rendered deterministically within PHZ files.

Reload after reload shows it rendered in different positions.

Spectron Testing currently broken.

We get an error about box-layout because we're on 3.0 beta. Hopefully it will be fixed soon and we can re-validate Windows

Feature: Informational pagemark type.

• there needs to be a 3rd pagemark type, IE informational.. bascially table of
  content and appendix pages. They should not have a full pagemark over them.
  and/or the pagemark should just be inactive. If the pagemark obsctures / blocks
  the UI then the table of contents won't be usable but at the same time I don't
  want the ToC to count against our progress percentage.

Disjoint highlighting in HTML

Right now we don't support the unified highlights like we do in PDF.js

Need to clean this up so that the text on the same lines are all the same as well as the breaks between them removed.

We must use the protocol object for each session to inject cached content.

Right now we tried to use 'partition' to automatically make a session and bind it to a window but then our requests no longer use request interceptors because a new session is made.

If we create a session manually then we can then get the protocol object and add our own interceptor.

If we use what we currently are doing though it works for default sessions but not new/custom sessions.

https://github.com/electron/electron/blob/master/docs/api/session.md

The 'hidden' profile doesn't always include images

https://medium.com/@maybekatz/introducing-npx-an-npm-package-runner-55f7d4bd282b

The 'headless' profile works better but it still fails here.

Audit TODO and FIXME issues with code.

Since this started off as a proof of concept I took notes in places where things are less than idea.

These are denoted by TODO and FIXME.

We should resolve these and/or move them to Github issues.

Some of them are probably very complex. Some are probably easier and could be resolved quickly.

Fix final issues with resizing pagemarks.

There are still some final issues with pagemarks and resizing them. I think it's an issue of vertical or horizontal intersection. We need more tests for these broken edge cases.

More Typescript conversion.

We still have to step through some of our Javascript code and convert it to TS...

This is a good area for new developers to take a look at pieces of the code and convert them slowly.