sw-yx / ssg

Svelte Site Generator. why try this? because sapper has a lot of setup, and isn't great at pipelining data:

Aug 2020 Update - This is now a deprecated project

I was planning to do a rewrite round about now, but there are enough people working on the static sites problem in Svelte that I don't wish to contribute to the noise. Please check out these other projects.

• https://elderguide.com/tech/elderjs/
• https://github.com/junglejs/junglejs
• more on the way

I will be exploring moving swyx.io to one of these projects to support them.

---

Why SSG?

Watch this 20 min talk: https://youtu.be/JEH0V-zNGUk

What is this repo?

this is a monorepo for a Sapper Site Generator and related packages. Check README's inside /packages for more info on each. In particular:

• /packages/ssg is the primary package. Please click thru and read this to get started.
• /packages/source-yaml (status: active)
• /packages/source-devto (status: active)
• /packages/source-remark (status: inactive)

We also have an examples folder but its kinda inactive right now.

Project Roadmap

I have been dragging my feet on SSG mostly because I wish someone else was working on it rather than me (it will never make money and I don't have time for it).

However the conversations I have daily at work always bother me and make me wish there was a better solution out there. Because nobody is working on exactly what I want, I'm forced to be the one to do it :( despite not having a better usecase for it than my own blog.

I am not at all optimistic this will ever be a wise venture so this is literally for shits n giggles until someone more serious than me picks this up. Because I'm not serious about it, it has an even higher chance of failure than the abysmal chance it had :(. at least i hope somebody else comes along and picks this up.

Other attempts at a Svelte Site Generator

• Sapper
• https://github.com/jakedeichert/svelvet/issues/31
• https://github.com/PKlknr/svatic

SSG v0

whatever i have now, the Sapper-based slow monstrosity. I can add:

• an incremental build api
• zero config filestructure based markdown
  • default ejectable theme
• preprocessed typescript and postcss
• DEBUG=*

SSG v1

What I want SSG to launch with is:

• 0kb of JS for a page that doesnt use it
• visual editor? use netlifycms? fork sapper studio?
• incremental builds
• ludicrous speed. this probably means not using Sapper, and possibly using Rust/WASM.
• ejectable themes
• preprocessed typescript and postcss
• Zero config:
  • RSS
  • filestructure based markdown
• ship your own docs in CLI
• proper docsite
• DEBUG=*
• Differential Dataflow

open questions:

• dev mode?
• no clientside routing?
• graphql?
• 404 search
• react import path?
• tailwind?
• graphql support?
• support 11ty's templating (mustache etc) and exact api?

SSG v2

I actually wish i could work on this earlier, but Netlify doesn't support datastores now so there's no point:

• serverless serving if updates are found (so you get instant refresh)
• smart clientside routing? need to learn a lot more about this
• service worker layer?

TODO: check if there's a way to see if there's a new version of a page updated

• nested ssgconfigs so you can section up sites?

What else?

ssg's Issues

Ejecting `rollup.config.js` forces us to eject `client.js`, `server.js` and `service-worker.js`

I think the title is pretty clear. ^^

@sw-yx, I read you are looking for feedback and help so I will start by dropping everything I can in issues as I am trying to build my personal site with ssg. I don't know how much I will find so let me know if it gets too crowded. ;)

[help] Logo

ssg needs a logo before launch

theres a small chance that ssg becomes framework agnostic, so we may not want to tie to svelte too much, altho ofc svelte will forever be first class

[v1] notes for ssg rewrite

The primary way ssg will work:

1. ReadConfig
   • Read in all plugins and markdown and site global info
   • Create plan to create indexes
2. Refresh
   • Pull indexes, run thru all plugins, create desired manifest
   • Pull existing files/create old Manifest (skipped in DEV)
3. Plan (skipped in DEV)
   • Diff desired vs old manifest
   • generate list of pages (and other files) to generate
4. Build (in DEV - only build on demand)
   • Pull data slices
   • Generate HTML, JS, CSS and JSON
5. PostBuild (n/a in DEV)
   • FullData => RSS
   • Serverside Redirects?

Concepts:

• Site global info is readable anywhere. changing it forces complete rebuild.
• We break up data fetching into (cheap) data index and (expensive) data slices. We memoize data slice calls and attach them to data Index in a FullData object, with which we can do final builds like for RSS.
• ssg can generate pages, images, and misc files - we just call these "files"
• plugins can:
  • contribute indexes
  • map from index they own to files
  • map from FullData index to files
  • map from indexes to new index? -> eg how to generate tags/categories

Features to design for:

• shadowed/eject templates and webpack config
• We want programmatic page creation
• We want incremental builds
• We want a nice dev mode
• We want redirects -> clientside as well as serverside
• We want typescript/postprocessing/postcss
• We want RSS (dataslices)
• we can stop and rerun any stages if misconfigured
• REPL for data
• Filesystem routing

nonfeatures

• No js in main page - just has mount points for svelte
• no clientside routing
• no functions/api routes

===

https://github.com/sw-yx/ssg-prototype

notes on kirby api

from conversaiton with limitlessloop today

swyx
#2697

swyxToday at 4:45 PM
have you looked at react-static?

limitlessloopToday at 4:47 PM
No, I was originally put off when I tried React so I haven't really gone back to it. I found Svelte and Vue much easier to get to grips with. But maybe I should take another look at it.

swyxToday at 4:47 PM
well, its a valid feeling. just trying to help you find the right tech, as someone who spends every day in this space
i agree graphql is annoying overkill much of the time

limitlessloopToday at 4:49 PM
Thanks. I really appreciate it. I've been back and fourth looking for the right tool for building static sites. Hard to find the right balance of things.
I'll check out react-static. So far VuePress has been my favourite. Components are automatically registered, lots of support for markdown, not much to configure, support for PostCSS, Less, Sass etc.
Just lacks and easy way to get at content.

swyxToday at 4:52 PM
i'm surprised to hear that
you should give this feedback to the vuepress team
(i dont personally use it so i dont know.. but still)

limitlessloopToday at 4:53 PM
Yeah I've reached out to the creator to see if I can get their thoughts on it.

swyxToday at 4:54 PM
meanwhile i am working on my own version
https://npm.im/ssg
but nowhere near ready for you to try it lol

limitlessloopToday at 4:55 PM
Well if I can be of any help let me know. I'll certainly take a look at it regardless.

swyxToday at 4:56 PM
well i think the better i understand usecases the better i can make this thing
i want to make it really fast, taking the best lessons ive had from every SSG i've used

limitlessloopToday at 4:57 PM
Sounds really interesting and promising. Svelte/Sapper is a great thing to build it from.
Have you ever used Kirby? Although it's not a ssg, it's by far the best flat file CMS I've ever seen/used. And I'm not even talking about the admin panel, just the way it allows you to create views and reference content from files.

swyxToday at 4:59 PM
nope
tell me more about how it allows you to create views and reference content from files?
this $pages thing you linked doesnt really help

limitlessloopToday at 5:01 PM
For example by default templates look to content files with the same name for their content.

swyxToday at 5:01 PM
example?

limitlessloopToday at 5:01 PM
That means creating a page is as simple as creating a template with the same name as its content counter part.

swyxToday at 5:01 PM
foo.template.js -> foo.md?

limitlessloopToday at 5:01 PM
Yes

swyxToday at 5:01 PM
if its a 1:1 relationship why do you even need a template
templates are for many:1

limitlessloopToday at 5:03 PM
So you can keep template markup separate from content I guess. And also I'd argue that idea that templates are for the many. In a lot of my projects which are simpler static sites as apposed to apps and a lot of the pages are fairly unique in their layout, so a one to one relationship is simpler and easier to manage. It's "flatter" by nature.

swyxToday at 5:04 PM
ok. any other features that you feel strongly about?
also can you link to an example repo with this filestructure
so i can see this in action

limitlessloopToday at 5:06 PM
The other features I really enjoy are 1) ability to reference data from text files using their API and filter it, manipulate it, traverse it. 2) Kirby has a really nice system for building the backend CMS UI using what they call blueprints. I'll send some repos over.
https://github.com/getkirby/starterkit
GitHub
getkirby/starterkit
Kirby's demo site is the easiest way to get started with Kirby. - getkirby/starterkit

In particular these folders
https://github.com/getkirby/starterkit/tree/master/content
GitHub
getkirby/starterkit
Kirby's demo site is the easiest way to get started with Kirby. - getkirby/starterkit

https://github.com/getkirby/starterkit/tree/master/site/templates
GitHub
getkirby/starterkit
Kirby's demo site is the easiest way to get started with Kirby. - getkirby/starterkit

https://github.com/getkirby/starterkit/tree/master/site/blueprints
GitHub
getkirby/starterkit
Kirby's demo site is the easiest way to get started with Kirby. - getkirby/starterkit

swyxToday at 5:09 PM
for 1), that feels like just "lodash-like utilties". for 2) taking a look
blueprints are indeed very nice. i will try to steal this
sorry, i mean the templates
blueprints arent my concern right now but i do think netlifycms kinda works like that
ok, here it is
https://github.com/getkirby/starterkit/blob/master/site/templates/home.php
GitHub
getkirby/starterkit
Kirby's demo site is the easiest way to get started with Kirby. - getkirby/starterkit

limitlessloopToday at 5:13 PM
Yeah blueprints would be least of my concerns too.
Its worth trying Kirby if you can. You only need to install MAMP and download the starter kit, it's pretty easy to get setup.

swyxToday at 5:14 PM
nahh its ok you've given me a good enough idea
my MAMP days are years ago

limitlessloopToday at 5:15 PM
I like how you can put numbers before folders and files to give them an order but the number is removed.
So posts are automatically order by date in the file name but the date is removed for the url

swyxToday at 5:16 PM
what if you really wanted it there

limitlessloopToday at 5:17 PM
I think there is a way to keept it there by adding another prefix but I've never had a need to.
Yeah my MAMP days are over too, but I wonted to refresh my memory about Kirby so I installed it just to try it again.
Im tired of configing my machine for PHP when I never use it anymore

swyxToday at 5:21 PM
oh well. this is helpful. but for my thing im not even there yet.. i'm still at the "should i use sapper or fork it" stage

limitlessloopToday at 5:22 PM
Yup I understand.
I wish you all the best.

swyxToday at 5:24 PM
thanks!

limitlessloopToday at 5:24 PM
Let me know how you get / what you decide

reload

rerun index data fetch when ssg.config.js is reloaded

preserve json-ld scripts by default in rollup config

When trying to add json-ld scripts to svelte:head, we get this issue:

[svelte-preprocess] Error transforming 'ld+json'.

I have documented my solution in this stackoverflow. It is ugly and I would be surprised there is not a better one. If anyone has a better solution it is welcome. In the meantime I suggest at least adding

// rollup.config.js
// ...
import sveltePreprocess from 'svelte-preprocess';

const preprocess = sveltePreprocess({
  preserve: ['ld+json'],
  // ...
});

export default {
  client: {
    plugins: [
      svelte({
        preprocess,
        // ...
      }),
  },
  server: {
    plugins: [
      svelte({
        preprocess,
        // ...
      }),
    ],
  },
};

to the default rollup config.

Thoughts?

use nprogress by default

preloading sveltejs/sapper#968

`ssg-dev` doesn't detect new markdown files

ssg is running via ssg dev.

I have a content/ folder with markdown files.

I type touch content/new.md. The server logs content/new.md changed. rebuilding...

new.md is not returned by this.ssgData({}) inside route preload functions.

I kill the server and re-run ssg dev and now new.md is returned by this.ssgData({})

important svelte issues

• component mktplace sveltejs/svelte#1070
• awesomelist https://github.com/ryanatkn/awesome-svelte-resources

How to parse markdown?

How should one parse markdown content of a content folder like you do in your website?

I was guessing it had something to do with marked dependency in your website -- but I was able to uninstall it and your site still built, so it seems to be happening somewhere else ...

squash output mode

squash output mode

[note to self] importance of lazifying in dev mode

https://twitter.com/wesbos/status/1234593086380150786

[feature] programmatic page creation

So I've spent a little bit of time trying to see if I could make ssg work like Gatsby for page creation (with the createPage node API hook).

My approach is currently

1. Fetch data from my CMS in createIndex and cache it
2. Use this.ssgData in the preload function of my [slug].svelte component

The main problem with this approach is that the route matching is dependent on file structure. My [slug].svelte component can only match routes at level 1. If I want sub-pages I need to create a component at [one]/[two].svelte. The problem is I would like to be able to create an indefinite number of nested routes on my external CMS without having to worry about how many sub-folders I created in the source code.

Another problem (arguably) that is obvious with the way Sapper create files on export is it only creates files for routes that can be crawled from the index page. We cannot create 'hidden' pages or landing pages for example.

Is it something you are interested to handle with ssg @sw-yx ?
For now, ssg seems to be focused on having your data along with your source code. If we want to enable programmable page creation it seems we are going to diverge from Sapper's core principles in major ways.

Error: Cannot find module '/home/bart/src/ssg/ssg.config.js'

Afer installing ssg with yarn add ssg I get the following error when trying to run yarn ssg dev:

ssgConfig file ssg.config.js doesnt exist, continuing as regular sapper app
> Cannot find module '/home/bart/src/ssg/ssg.config.js'
Require stack:
- /home/bart/src/ssg/node_modules/ssg/dist/cli-ssg.js
- /home/bart/src/ssg/node_modules/ssg/dist/cli.js
- /home/bart/src/ssg/node_modules/ssg/ssg
Error: Cannot find module '/home/bart/src/ssg/ssg.config.js'
Require stack:
- /home/bart/src/ssg/node_modules/ssg/dist/cli-ssg.js
- /home/bart/src/ssg/node_modules/ssg/dist/cli.js
- /home/bart/src/ssg/node_modules/ssg/ssg
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:966:17)
    at Function.Module._load (internal/modules/cjs/loader.js:859:27)
    at Module.require (internal/modules/cjs/loader.js:1028:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at readSSGConfig (/home/bart/src/ssg/node_modules/ssg/dist/cli-ssg.js:78:21)
    at /home/bart/src/ssg/node_modules/ssg/dist/cli.js:92:27
    at Generator.next (<anonymous>)
    at fulfilled (/home/bart/src/ssg/node_modules/ssg/dist/cli.js:5:58)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

It's supposed to be zero-config right?

Cannot read property 'pubdate' of undefined (ssg/dist/coreData.js:80:35)

After yarn add ssg && touch ssg.config.js:

[bart@karuna ssg]$ yarn ssg dev
yarn run v1.21.1
warning package.json: No license field
$ /home/bart/src/ssg/node_modules/.bin/ssg dev
> Cannot read property 'pubdate' of undefined
TypeError: Cannot read property 'pubdate' of undefined
    at /home/bart/src/ssg/node_modules/ssg/dist/coreData.js:80:35
    at Array.sort (<anonymous>)
    at Object.<anonymous> (/home/bart/src/ssg/node_modules/ssg/dist/coreData.js:79:18)
    at Generator.next (<anonymous>)
    at fulfilled (/home/bart/src/ssg/node_modules/ssg/dist/coreData.js:5:58)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

sapper export notes

• sveltejs/sapper#859
• sveltejs/sapper#869
• sveltejs/sapper#856

Example doesn't run

> [email protected] dev /home/justin/dev/blog
> ssg dev

SSGDEBUG ssgconfig ssg.config.js
> ssgConfig file ssg.config.js missing
Error: ssgConfig file ssg.config.js missing
    at readSSGConfig (/home/justin/dev/blog/node_modules/ssg/dist/cli-ssg.js:24:15)
    at /home/justin/dev/blog/node_modules/ssg/dist/cli.js:82:27
    at Generator.next (<anonymous>)
    at fulfilled (/home/justin/dev/blog/node_modules/ssg/dist/cli.js:5:58)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] dev: `ssg dev`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/justin/.npm/_logs/2020-01-13T02_32_31_535Z-debug.log```

Sapper viability on Netlify

Hi @sw-yx. Thanks for your efforts trying to get Sapper working better with the Netlify platform. Would you say it's viable to use Sapper for a small to medium-sized site or PWA currently? Would something like your Gatsby JAMstack app be doable with Sapper, or would the issues you've detailed be major hurdles? Would your SSG be needed?
In SSG's readme, you also indicated "Netlify doesn't support datastores"? Any further details on that?

Would you recommend against trying to build something non-trivial with only Svelte, no Sapper?

---

I'm really liking Svelte's "less code" philosophy, and would love to get something deployed with Sapper, without having to switch to a generator that takes a React, Vue, or Angular approach.

Sapper seems to have been targeted at SSR, but after working through some of the hurdles with that, using Amplify and Firebase tools, I'm came across Netlify's dynamic JAMstack pitch and "one workflow" philosophy, which seemed like a better setup.

I think Sapper team is getting the message around a desire for static deploys + serverless functions over SSR? Have you seen this thread and fork?

On the Netlify side, I'd love to see a Sapper configuration soon, or even just a template.
I'm seeing that the Netlify CLI does detect Sapper projects, so I'm hoping to see the pairing gain traction. I'd be happy to sound off in Netlify Community or in Sapper's issues if you think that could help.

Unable to access index from getDataSlice in custom plugin; "not found" error

I can't seem to write a getDataSlice function that is aware of the index. Below is my attempt, based on the devto plugin, but I've probably missed something simple.

Even if this worked, I'm not sure how I would access the index if I were writing a one-off plugin in exports.plugins, so I wonder: would you be open to including the requested index as a parameter in the call from readConfig.getDataSlice to the getDataSlice plugin function?

const { slugify } = require('./helpers')
const fs = require('fs')
const matter = require('gray-matter')
const normalMD = require('markdown-it')('commonmark')
  .enable('table')
  .use(require('markdown-it-attrs'))
  .use(require('markdown-it-footnote'))

module.exports = function(options) {
  let opts = Object.assign({
    folders: false,
    section: '',
    md: normalMD,
  },
  options || {})
  if (typeof opts.folders === 'string') opts.folders = [opts.folders]

  let allPosts = []

  async function createIndex(idx) {
    idx.ssgCoreData.filter(o => {
      o.type = o.metadata.type || o.shortFilePath.split('/')[0]
      if (o.type.indexOf('.') > -1) o.type = 'page'
      return !opts.folders || (opts.folders.indexOf(o.type) > -1)
    }).forEach(o => {
      if (!o.metadata.slug && o.metadata.title) o.metadata.slug = slugify(o.metadata.title)
      o.slug = o.metadata.slug || o.slug
      o.href = o.metadata.href || [
        (opts.section || o.metadata.section), // section (music)
        (o.metadata.type || o.type),          // type (song)
        o.slug                                // slug (the-earth-and-the-eagle)
      ]
        .filter(Boolean)
        .join('/')
        .replace(/^page\//, '')

      allPosts.push(o)
    })
    return allPosts // <---------allPosts contains data
  }

  async function getDataSlice(slug, coreDataPlugin) {
    let item = allPosts.find(i => i.slug === slug) // <---------allPosts is empty
    if (item) {
      let rawData = await fs.readFile(item.filePath)
      let fileData = matter(rawData)
      fileData.html = opts.md.render(fileData.content)
      return fileData
    }
  }

  return {
    createIndex,
    getDataSlice,
  }
}

graphql layer?

https://github.com/khrome83/khrome.dev/tree/f36c4438b0c87a3190fedef88db676ff5913ce89/www/src/libs

[howto] use data from an external source

I tried fiddling around ssg to populate the mainIndex cache with imported data. I was quite successful at that but I can't really figure out how to getDataSlice properly.

I guess my real question is what is your mental model for createIndex, getDataSlice and plugins?
I checked the source code but I couldn't really grasp the gist.