srid / zulip-archive

Zulip Archive viewer (statically generated HTML)

zulip-archive

Zulip archive viewer written in Haskell, using the rib static site generator.

Demo

Prerequisites

First, install the Nix package manager:

bash <(curl https://nixos.org/nix/install)

Optionally, enable the Nix cache if you would like to speed up local builds:

# If you do not already have cachix, install it:
nix-env -iA cachix -f https://cachix.org/api/v1/install
# Enable nix cache for rib
cachix use srid

Zulip API Key

Get your API key for zulip (instructions here). Note: you are looking for your API key, and not a bot's API key.

Running

First, create a configuration file, adding your Zulip site settings (including the API key):

cp config/config.example.dhall config/config.dhall
vim config/config.dhall

NOTE: You can also pass the contents of the config file in the environment variable ZULIP_ARCHIVE_CONFIG which takes precedence over the config file. This can be used to setup GitHub Pages workflow (see ./.github/workflows).

To build and run the site:

bin/run

Go to http://localhost:7004 to view your generated site.

Manual update

User uploaded files are unavailable via the API. They must be manually exported (needs admin rights), and copied to ./site/user_uploads.

zulip-archive's Issues

Build docker image

This would really speed up the daily Publish CI jobs.

Steal workflow YAML from https://github.com/srid/TheMotteDashboard

Support custom instance logos

Not sure if it is possible to get via Zulip API, but it would be nice to be able to show instance logo on top of archive page, together with custom favicon in tab card.

Support user_uploads

For example, images are uploaded in this topic: /haskell/selective-functor-paper-example.html

And it currently renders as:

Some old messages seem to be missing

At https://funprog.srid.ca/mtl/, there are no messages shown, even though this message was sent nearly two months ago.

Convert zulip links to archive links

For example #narrow/stream/201385-Haskell should be rewritten to /haskell/. Likewise for topic and message links.

Add Structured Markup to Pages

This would help display posts as rich results in search result pages:
https://developers.google.com/search/docs/guides/intro-structured-data
https://schema.org/docs/schemas.html

The few relevant types I can think of:

• Breadcrumbs: https://developers.google.com/search/docs/data-types/breadcrumb
  https://schema.org/BreadcrumbList
• Article: https://developers.google.com/search/docs/data-types/article
  https://schema.org/Article
• Or the DiscussionForumPosting Article-subtype: https://schema.org/DiscussionForumPosting
• Organization(for site name, logo, url to this repo, etc): https://schema.org/Organization

Should use rib's HTTP server

cf. #21 (comment)

Stream blacklist

Allow some streams to be blacklisted. Which means they should not have their HTML generated, and be excluded from the main index.

For eg., #haskell-beginners (because it is now locked and unused).

Emoji reactions

Gravatar support

From https://funprog.srid.ca/haskell/zulip-archive-looking-for-maintainer.html

Add Sitemap File

A sitemap of all the threads would help crawlers and improve search engine discovery. You can point crawlers to the sitemap using the Sitemap: <full-url> directive in a robots.txt file, or by sending them a ping. A sitemap can have a maximum of 50,000 URLs. Beyond that, you need to generate a sitemap index file that points to the different sitemaps.

Not sure if this is a rib issue or a zulip-archive issue, but you could leverage this package I published recently to build the sitemaps and sitemap index: https://hackage.haskell.org/package/sitemap-gen

OGP - "description" should be plain text

The description field is currently set to a HTML value, as such it doesn't seem to appear in link preview.

Zulip API can return the original Markdown format. Setting apply_markdown to true in /api/get-messages will get the original format. But we need the HTML as well, so maybe this API should be invoked twice.

Once both the formats are stored in the Message object (and consequently messages.json) we can use the markdown format in OGP description.

Reuse Rib.Extra.OpenGraph

srid/rib#118

cc @prikhi

Add proper logging

Using https://github.com/kowainik/co-log

Deal with late topic renaming

https://funprog.srid.ca/general/late-renaming-of-topics-can-break-archive-threads.html

Check topic slug uniqueness

No two topics should have the same slug.

Support arbitrary Zulip instances through custom configuration file

Custom configuration through YAML/TOML/Dhall would make it easy to run archive on arbitrary Zulip instance.

Revamp this project using Ema and Tailwind 3.0

https://ema.srid.ca/guide/helpers/blaze

Also revamp the design of the generated site. Make it awesome.

Code blocks don't follow site width

This is mostly visible on phones:

We probably want them to be shown in smaller, scrollable frame, similarly to Zulip itself.

Schedule periodic fetch of messages

GitHub Pages support

Create a Actions workflow that automatically deploys generated .html files to GitHub Pages (in a separate repo).

Should probably use docker image of zulip-archive (built via nix).