Delta builds for very large sites about react-static

[tannerlinsley]

That's great to hear! This is a really cool idea, too. As far as only updating specific posts, that would probably need some tweaking in the architecture. But it could be done. Basically, it would involve a few things:

• Keeping the dist folder around in your project
• Not cleaning the dist folder on build
• A special update CLI command or option in the config
• A special getRoutesUpdate callback or something similar to fetch the data for the updated posts and return which routes should be updated.

Some other thoughts:

• The bundle would probably need to be generated again

[D1no]

@mattyfresh something obvious that comes to mind is watchmen: https://facebook.github.io/watchman/

Concepts

You can query a root for file changes since you last checked, or the current state of the tree
You can subscribe to file changes that occur in a root

[mattyfresh]

@D1no Good call! Already digging through their source code. I have a super basic implementation that I threw together that just forks the react-static build command, does not rm the DIST directory, and filters for new posts via a JSON file of writtenPaths that were built on the last build.

const routesToUpdate = config.routes.filter(route => (writtenPaths.indexOf(route.path) === -1))

Then you can just pass that to writeRouteComponentsToFile which is convenient. This would not work for updating an existing post though, and I'm sure we would want more control like build by certain post-type or some other criterion.

Thanks for the suggestions! This seems like it should be doable.

[derekperkins]

That works for file based systems. How would you handle a dynamic site querying a CMS like Contentful? It'd be pretty trivial to filter on their api for posts modified since lastBuildDate, which would then get pushed through getRouteProps.

[mattyfresh]

Thinking on this further, and thinking along the lines of what Gatsby does building a sort of dependency tree where I can know when related entities change, and therefore rebuild the affected pages. IE you have an Musician page that renders a list of Songs, if the structure of Songs changes, everything that depends on that entity ought to be rebuilt as well.

My coworker found this fairly well articulated here: https://stackoverflow.com/questions/17607287/changes-based-regeneration-by-static-site-generators

This is from 2013! Still seems like a custom solution needs to be created.

[tannerlinsley]

Yeah that relationship graph will almost always be custom and dependent on where the data ends up in your components. Derek's suggestion is the safest I believe. Detecting changes using a timestamp and programmatically regenerating any routes that use that data, whether that's a single page, a category list or a next post button.

…

On Wed, Oct 18, 2017 at 8:11 AM Matthew Padich ***@***.***> wrote: Thinking on this further, and thinking along the lines of what Gatsby does building a sort of *dependency tree* where I can know when related entities change, and therefore rebuild the affected pages. IE you have an Musician page that renders a list of Songs, if the structure of Songs changes, everything that depends on that entity ought to be rebuilt as well. My coworker found this fairly well articulated here: https://stackoverflow.com/questions/17607287/changes-based-regeneration-by-static-site-generators This is from 2013! Still seems like a custom solution needs to be created. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#53 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFUmCcK31Ju0pCNUepCrklJJ2KwhC8B7ks5stgcbgaJpZM4P8Qba> .

[tannerlinsley]

Has anyone thought any more on this subject?

[tannerlinsley]

I know this isn't necessarily a delta build strategy, but you should look at #96, which I just merged. :)

[mattyfresh]

Awesome! Been side tracked getting TypeScript to play nice in our proposed front end ecosystem (TS, React, React Static, Apollo Client). Took a quick look at the PR and I am excited to take another crack at rendering a large site for our POC.

I did manage to wire up the TypeScript build, but I am still using react-static v2.0.0. Hooking into webpack was pretty straightforward as mentioned here #8, later this week I could help create an example project if that helps!

Side Note: I obviously haven't ran the updated script yet, but when I was writing my own basic implementation I got bit by using .forEach (which has to throw a new function on the stack for each iteration). Using a for or for..of loop (if it doesn't make things completely confusing) where possible sped up my script considerably.

For reference:
https://coderwall.com/p/kvzbpa/don-t-use-array-foreach-use-for-instead

[tannerlinsley]

Yeah, normally I would use a loop, but in this case, the scaffolding loops only take a second or two to generate the routes. 99% of the time is actually spent in webpack and react SSR.

…

On Wed, Oct 25, 2017 at 9:38 PM Matthew Padich ***@***.***> wrote: Awesome! Been side tracked getting TypeScript to play nice in our front end ecosystem (TS, React, Apollo Client). Took a quick look at the PR and I am excited to take another crack at rendering a large site as for our POC. I did manage to wire up the TypeScript build, but I am still using react-static v2.0.0. Hooking into webpack was pretty straightforward as mentioned here #8 <#8>, later this week I could help create an example project if that helps! Side Note: I obviously haven't ran the updated script yet, but when I was writing my own basic implementation I got bit by using .forEach (which has to throw a new function on the stack for each iteration). Using a for or for..of loop (if it doesn't make things completely confusing) where possible sped up my script considerably. For reference: https://coderwall.com/p/kvzbpa/don-t-use-array-foreach-use-for-instead — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#53 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFUmCetEpzctMVFoOh1RyRAq9qOaJlAvks5sv_7FgaJpZM4P8Qba> .

[mikeplis]

Definitely interested in this issue. Pushing the onus onto the user to determine which routes need updating feels like a good idea. I made a fork that very naively implements a new update command that basically does the same thing as build, but calls getRoutesUpdate instead of getRoutes and doesn't delete the dist folder. I haven't had a chance to really test it out, but it seems like it could work, at least for certain use cases.

One thing I was wondering with this approach is would there need to be some way to pass arguments to the update command that the user can use to determine which routes need updating (e.g. a timestamp)?

[tannerlinsley]

After a lot of discussion on this topic both internally and externally, I think we've decided to take the direction of the library towards "fast immutable" builds. Delta builds introduce a few things that are difficult to prepare for:

• Data dependencies and relationships. Given RS's data agnostic approach, this would be almost impossible.
• Server or git related caching mechanisms. This is not RS's arena or responsibility.

We feel our efforts will be much better invested into producing faster builds for the library.