mr55p-dev / pagemail

Pagemail

Pagemail is a tool for saving and rembering links.

## Environment

• PM_ENV: dev, stg or prd
• PM_LVL: DEBUG, INFO or WARN
• PM_TEST_USER: user id to store session for
• PM_DB_PATH: path to sqlite3 file

pagemail's Issues

Docker compose config is duplicated

Can combine the base config into one file and then just override in the other ones using extends keyword

There is no way to reset a password

Logs requests for static assets

Need to disable request logging for static assets

User has to refresh before page preview will be loaded

The user has to refresh after adding a link before their page preview is rendered.
This is generally poor UX. It would be easier to just do the preview rendering synchronously, or can
add a callback delay to refresh when (if) the preview is found

If a user has already added a page then just update the date modified

So duplicates should just move the page to be the most recent (maybe alter the date created instead)

pagination is broken on the pages view

Looks like all pages are loaded immediately, on scroll does nothing atm

Create readability microservice

Separate service accessible on the pagemail network only
Can accept requests for following actions (stubbed):

• Check readability of a page
• Create readability job for a page (returns job id)
• Check job status (by job id)

Create new AWS infrastructure

• S3 bucket for readings
• IAM permissions for accessing polly

Ensure that all auth-based activities are cryptographically secure

Rolling own auth feels like a bad idea, lets get some proper management around the identifiers that
are being generated

Dependency on SQLX

SQLX is being used to manage all queries, while it has some advantages, SQLC is more suitable for
this use case of simple but lots of queries. Also will allow a declarative table structure which
will make implementing e2e tests a little easier.

Display log in with google on sign up screen

Need to make a background worker for all "unknown" preview states

Migrate to postgres

We are now a little beyond the model that sqlite3 supports with a single db consumer - it's time to
consider how to migrate to postgres or something similar.

Google sign in popup not working on safari desktop

Create new fields in database

Pages:

• readable: bool NOT NULL DEFAULT false
• reading_status: string NOT NULL DEFAULT unknown
• reading_job_id: string
  Users:
• has_readability: BOOL NOT NULL DEFAULT false

Migrations are not running properly

There is an error connecting during the build step, so the image is run on the actions docker runner

Shortcut response is html

PostPage is not respecting the content type header specified

Login form email autofills as username

Needs to be specifically email

Remove structs with a log field

Some of the structs in the app still have a log field in them. This can now be deleted

Use a proper session manager

the current session manager implementation was fine for now, but in fact a session store backed by
a database of kind (or at least a file...) needs to be implemented

handrolling is still an option (just a sqlite file in a docker volume is entirely doable) but it
might be easier to admit defeat and switch to gorilla sessions

Accounts update form does not work

The "subscribed" checkbox is not valid. Can rework the form so that each input update sends a request individually.

Mail is not coming out

Since merging hack and slash (#58) there has been no mail coming out via SES.
Need to investigate and fix

Password reset flow

iOS shortcut is not set to correct URL

Current iOS shortcut refers to the older /shortcut api, which is no longer available.
Needs to be re-pointed and deployed

Shortcut token should be a hashed value

[URGENT] Add page POST request returns HTTP 405 (Method not allowed)

When adding a page via dashboard, returns http405 method not allowed. Logged at 2024-05-08 22:07,
check container logs for info

Create articles and readings tables

Update the existing schema to add new tables:

Articles

Stores the pages text content extracted via readability service

Readings

Stores info for an articles reading (status, file location, progress and errors)

UI can present article text for users (and request caching can allow for offline capability down the line)
Should surface option to refresh an articles content (make sure to disallow spamming, keep requests to once per hour)
Track the progress of audio playback

Readability

There's a lot of work done on readability already. Can some of this be reused?
Now that we are using docker compose it'll be possible to host readability service in a separate
container and built it as a part of this project still. This allows using typescript anyways.

Create a "request readability" form

Form should be submitted and generate an email to me
Admin users should have access to grant access to readability

No way to unlink an identity platform

There should be a view on the accounts page to unlink id providers

403 when accessing shortcut url

Shortcut token does not seem to be recognised at the /pages/shortcut endpoint, probably because the wrong auth middleware is attached

There are actually zero tests

There are extremely few tests which makes debugging quite hard.
Need to identify some of the key API routes which are suitable for testing and implement tests
Can use net/http/httpmock for full end to end testing done locally.
Need a tool for mocking the DB

Add a footer

With contact info

Verify users

Index page title is "pagemail - "

Page title is not dropping hyphen in wrapper logic

cross-region aws access and deployment

new gen ai voices are only available in us-east-1
need a terraform deployment in that region as well as polly access for the ec2 instance in that region.
The pagemail solution must be able to presign links for that region as well, so the session will have to be either cross region or have two

Bcrypt the pages token

shortcut tokens should be hashed not just passwords, as this is another piece of access-granting info

OAuth with google needs implementing

Need to figure out how to implement the sign in with google from the previous site into this one

• Configured to fetch credential from sign in succesfully
• Added a /login/google endpoint to handle the request
• Creates a new user if one does not already exist
• Written DB migration to add an auth table so users can map multiple auth methods to one account
• Need to rewrite all the user queries which handle auth

try to figure out if the sqlc driver supports having multiple query files all contribute to
the same pakage.

Also, is it better to just have the db connection stored in (*Router).Conn and instantiate
dbqueries.New as and when it is needed?

Allow users to maintain multiple shortcut tokens

For having shortcuts on multiple devices

Add Articles page

Articles page is the hub for readability articles:

• Ready to read section: shows pages which already have a reading generated
• Create reading section: requests a new reading for the given compatable pages

Test issue

Create a playback page

Playback page for an article should start playing the audio of a reading which has already been generated

Fix production environment not being used in production

Alter the existing changes such that we are checking if env != production rather than env===production, so we will use the correct one by default and can debug the issue in stage

Create a release when deployment to prod succeeds

There's an action for it just put it in parallel with the deploy to prod call in the push tag workflow

AWS resources are not provisioned

Resources in AWS are currently hand deployed. We can move them into terraform very easily as AWS
auth is already done in github actions. Consider looking at atlantis
for PR based deployments.

Create synthesize endpoint

Synth endpoint needs connection to AWS via node aws sdk to polly.
Job endpoint should be able to lookup job status, error message and S3 location.

Script to run db migrations on server

• Create a backup
• Upload migration files
• Run dbmate for the relevant env

OR

• Create an ad-hoc docker image
• Include migrations and then mount the db.
• Run dbmate
• remove image
• can do this all in a GitHub action workflow, and include one for migrating up/down

There are no error messages at the login or sign up screens

Fix function call for preprod environment

Firebase ID token has incorrect "aud" (audience) claim. Expected "pagemail-2bc26" but got "pagemail-preprod". Make sure the ID token comes from the same Firebase project as the service account used to authenticate this SDK. See https://firebase.google.com/docs/auth/admin/verify-id-tokens for details on how to retrieve an ID token.

Search

Users need to be able to search for their pages (duh) on the dashboard