Modularizing jus about jus

[elingerojo]

About the PR.
Just to clarify. It will be a WIP: PR
I do not want this branch code to land in master yet. It is code ONLY to test the original Modularizing idea.

[zeke]

Underlying grand scheme: Maybe this can be used to give https://github.com/electron/electron.atom.io a post-ruby post-Jekyll future.

[jdormit]

Would this module replace the current file.js subclass functionality that pulls out metadata about each file type, or am I misunderstanding?

[zeke]

I think this would specifically be about replacing datafile.js.

Now that I look back at jus's source code, I see that the file-watching piece is really fundamental to every type, not just metadata files. I need think this through a bit more.. 🤔

[elingerojo]

Hi,

@zeke Are you thinking something like this fake README?

---

---

README.md

contextualizer

Extract all available data from files in a directory to one javascript object

Contextualize builds a context object from the data available in files in a directory for easy manipulation. It mirrors the directory structure and selectively extracts data depending on file type.

Usage

const contextualizer= require('contextualizer')
var ctx = {}
contextualizer(ctx, dir)
// ctx now contents dir structure AND all files data content

dir structure

.
|-- assets/
|   |-- photo.jpg
|   |-- style.css
|   `-- data.yml
|
|-- index.html
|-- README.md
`-- notes.txt

ctx content

// ctx = { assets: [
//            {type: "directory", ...}, ...
//                 { files: [
//                      {type: "image", name: "photo.jpg", ...}, ...
//         files: [
//            {type: "html", name: "index.html", ...}, ...

Example

If file data.yml has foo:"bar"

then:

console.log(ctx.assets.data.foo) // > "bar"

Dynamic async usage

const watcher = require('chokidar')
const contextualizer= require('contextualizer')
const options = {basedir: dir}
var ctx = {}
contextualizer(ctx, options) // initialize

// optional
// contextualizer().use(filetype, fileProcessor(ctx, metaCtx))

chokidar.watch(dir)
  .on('add', (filename) => contextualizer(ctx).add(filename))
  .on('change', (filename) => contextualizer(ctx).update(filename))
  .on('unlink', (filename) => contextualizer(ctx).delete(filename))

contextualizer(ctx).done( () => {
  // code that runs once ctx is available again (ctx in not available 
  // while there is any file still being contextualized)
  // ...
})

API

bla...bla...bla

License

MIT bla...bla...

[elingerojo]

I think it can be done if we break the contextualize process in two parts.

1- The modular one that create a context where it does just the basic stuff. Things that are not much opinionated so it can be popular on npm.
It will include the file.js and folder files but with just the basics.
And for the brave developers something like contextualizer().use(filetype, fileProcessor(ctx, metaCtx)) so they can extend and add their "special sauce" to their App. This metaCtx will have dir, filename, flags and options, etc., everything that does not go in ctx but is needed to extend it.

2- The jus part that take the above context as a pre-context and complete it to suit. Maybe by rewriting the fileProcessors to look like the ones jus already have.

This is kind of writing some module code that will not end up being used by jus but needed so others can have a starting point.

Approach

• Test the idea in jus as it is right now with a branch called something like split-ctx.
• If concept proven possible, then develop the future README.md of the contextualizer module (or whatever we call it).
• Then... If the README looks like a winner, then start with the "Modularizing jus". IMHO there is no point in start coding and getting something too complex that ends up not been popular.
• In worst case, the split-ctx branch could end up merging with master as an improvement and not everything is lost.

[elingerojo]

Ideas for 'proof of concept' of the contextualizer module

First steps

• does not include chokidar (jus does)
• can process single files, not necessarily arrays of files. This make it directly compatible with chokidar add, change and unlink events
• has decontextualize inside that is the contextualize inverse. It will work like a 'perfect' undo
• file updates are handled like two sequential events, unlink plus add

Later steps

• includes chokidar to process entire directories directly

Once proven, continue to the README "litmus test"

[zeke]

@elingerojo this is awesome! I love the approach you've outlined here, and that looks like a great README.

Do you want to give this a try?

PS I invited you to join the github org a while back. You should be able to see the invitation by visiting https://github.com/jus

[elingerojo]

Invitation, great!
@zeke, Thanks.

---

@ALL
I just finished a small step.

I have a branch named alpha-file-self-ctx in my fork https://github.com/elingerojo/jus.git

• There is no more centralized contextualize. It has been distributed allover each file classes.
• Now there is a decontextualize step for updates and deletes. Now there is no debris left behind.
• Passes all test
• I have already tested live locally with:
  1. jus.js.org
  2. zeke.sikelianos.com
     ...and nothing has been broken

It is a temporary first step because I am already working in the second step but you guys can take a look at it if you feel curious.

Let me know if you prefer me to open a branch in this repository for your convenience because I am planning to have at least 3 to 4 small branch releases before I submit a PR to master.

[zeke]

Let me know if you prefer me to open a branch in this repository for your convenience because I am planning to have at least 3 to 4 small branch releases before I submit a PR to master.

Opening a branch on jus/jus would be slightly more convenient for me because then we can more easily 🎾 the code back and forth if needed, but do whatever is more comfortable for your workflow.

[zeke]

There is no more centralized contextualize. It has been distributed allover each file classes.

Why? It seems there is now a lot of duplicated code.

[elingerojo]

Opening a branch on jus/jus would be slightly more convenient...

Will do

Why? It seems there is now a lot of duplicated code.

No duplicated code (that I am aware of).
It is just spread allover.

The common code goes in File class only (so it is not duplicated at all)
The particular code for each "fileType" goes in its own class only

The idea behind is that each developer will extend only what they need.

• The module will have File class and each "fileType" class with the bare minimum to produce a ultra simple context object that reflects the directory structure and minimum data

• The developer's application (like jus) will have also the File class and the "fileType" classes that he chooses to extend, with only the code particular to the application (and any code to override the default if needed) but will not need to duplicate code that already is in the module's classes

Note:
There is some "like-duplicated" code for asynchronous reasons.

When changed to contextualize per file, the order of some contextualization matters. For instance, if you add a page then an image or first the image and then the page you end up with problems on "who" does the "same directory" contextualization.

Is it the page class or the image class ?

The answer is both. Because it needs to be the last one that does the contextualization.

For this reason, you end up with very "similar" code (not duplicated) in both classes.

This only happens to the app developer, it does not happen in the module.

[elingerojo]

Why? It seems there is now a lot of duplicated code.

Re reading your question, now I get it.

You are right. It seems a lot of "duplicated" because it is double the code.

Now there is deContextualize for update and delete operations.

• Instead of always "rebuilding" all the context in response to a change, now it is incrementally (and "decrementally") updated

• And now, you can add more sophisticated context like aggregated relations and update with confidence that you do not leave debris behind (as long as you write the inverse code also)

[elingerojo]

Created a new branch called alpha-dynamic-class, it is a step toward isolating the meta data finder

• still pass all tests

• still nothing broken

• now there are two groups of file types

  1. basic types, all those included in files directory inside the module (at present it is only one, type datafile )
  2. custom types, all those included in the files directory inside the App (like in jus case, the directory ./lib/files with the usual 7 types, datafile, image, layout, ... unknown )

• the custom file types override the basic types if files directory path is given

I feel confident that the module could include chokidar so it can be removed from jus but need some feedback on the "contextualizer" module API

@ALL
What is your opinion on the following assumptions?

• There are two groups of "contextualizer" module users

  1. standard users
     • that they get just what is already included in the module context object with meta data from YAML and JSON files mirroring directory structure (they use the module "out of the box")
     • that they do not extend to other file types
  2. advanced users
     • that they extend the classes at will (this is jus case)
     • that they look for the benefit of a meta data finder/extractor tested framework that is extensible. I am taking about having a directory named files that have classes named after the file itself, etc.., a asynchronous design based on chokidar, etc.

• The advanced users

  1. may only need to add two or three file types and not the full blown set of "possible" types.
  2. appreciate a balance between "convention over configuration". This means, that will like to have control over how each type is handled but do not want to "break the squeezing mechanism". I am talking about not "exposing" the basic contextualizing but letting them have control of particular contextualizing for each type.

@zeke
The last point is the underlying reason for "duplicated code". If we keep the contextualizing in one place, we loose the "single responsibility principle" from the file type point of view. This way, the user could add as many types as he/she wants and all types will be independent (and each one will be responsible of its own particular contextualizing).

Note:
I modified the mocha test call because a "memory leak" appeared when moving require-dir inside Context class constructor. So "still pass all test" is not totally true 😄

[zeke]

Nice work, @elingerojo. Do you want to open a PR to make it easier to have a discussion about the branch you're working on?

I was thinking chokidar should be a dependency of contextualizer, not jus.

Here's what I'm picturing:

Each "filetype" should be a module that exports an object which defines two functions: check and parse. Here's an example of what a Sass parser could look like:

// parsers/sass.js

module.exports = {

  // calls back with a boolean indicating whether this parser should be used to parse the given file.
  check: function (filename, callback) {
    const extension = path.extname(filename).toLowerCase()
    const allowedExtensions = ['.scss', '.sass']
    return callback(null, allowedExtensions.includes(extension))
  }, 

  // works its magic then calls back with a metadata object
  parse: function (filename, callback) {
      // this example is synchronous, but ideally all parsers would be fully async 
      // to keep the memory footprint low
      const output = sass
        .renderSync({data: fs.readFileSync(filename, 'utf8'), indentedSyntax: true})
        .css
        .toString('utf8')
      return callback(null, output)    
  }
}

Then you tell the contextualizer which plugins to use:

const contextualizer = require('contextualizer')
contextualizer.use(require('./lib/parsers/sass'))
contextualizer.use(require('./lib/parsers/js'))
contextualizer.use(require('./lib/parsers/yml'))

// `watch` should take the same options object as chokidar

contextualizer.watch(sourceDir, {})
    .on('context', (context) => {
      // this is the context object containing all contextualized files
      // this event is emitted when the contextualizer finishes parsing all files in the source directory
      // and it's also emitted any time a file is added, removed, or changed.
    })

[elingerojo]

Do you want to open a PR to make it easier to have a discussion about the branch you're working on?

Will do.

I was thinking chokidar should be a dependency of contextualizer, not jus.

Yes. Me too.

Before I take a look at the check/parse let me show you the new commits of the branch I am working on.

---

contextualizer

• still pass all test
• still nothing broken (jus works as usual)
• now contextualizer could be extracted as an independet module

Usage

module.exports = function jus (sourceDir, targetDir) {
  sourceDir = sourceDir || path.normalize(process.cwd())
  targetDir = targetDir || path.normalize(tmp.dirSync().name)
  const emitter = ee()

  var options = {}
  options.typesDir = path.resolve(__dirname, './files')
  options.typer = require('./typer')
  options.ignored = ignored(sourceDir, targetDir)

  // TODO: Evaluate targetDir API
  // Is targetDir relevant to contextualizer or is it an advanced option to write files?
  contextualizer(sourceDir, targetDir, options)
    .on('adding', (file) => {emitter.emit('file-add', file)})
    .on('updating', (file) => {emitter.emit('file-update', file)})
    .on('deleting', (file) => {emitter.emit('file-delete', file)})
    .on('contexting', (files) => {emitter.emit('squeezing', files)})
    .on('contexted', (ctx) => {emitter.emit('squeezed', ctx)})

  process.nextTick(function() {
    emitter.emit('started')
  })

  return emitter
}

Options

• Custom file type dir - The files dir for custom file type classes
• Typer function - Custom function that returns a custom file type from a given filename
• All chokidar options - Simple wrapper to pass options directly to chokidar

Questions:

How to handle targetDir? Is it relevant to meta data finder/extracter?

Ideas:

• could be handled inside options. This will imply changes in File/BaseFile classes to handle only two arguments (one of them: options). Maybe this could be good because will open a way to send arbitrary parameters/arguments to custom file types that could handle more sophisticated contextualizing.

How to handle unknown files that not were ignored? Is unknown type relevant to meta data finder/extracter?

• It only adds "weight" to context object when not used.

Ideas:

• maybe should add logic to ignore unknown files by default. Something that depends on the presence of unknown in files/. It could be ' ignore extras ' instead of ' catch all ' policy.

[elingerojo]

About check / parse

Sounds great!

Do you prefer to purse this instead of the stand alone metadata finder?

They are different scopes with different use cases.

For instance, the check part will be spread all over and now it is "centralized" in createFile() (in master... and still "centralized" in typer() in my branch)

I have no problem going either direction.

[zeke]

Do you prefer to pursue this instead of the stand alone metadata finder?

Yeah I think I kind of do.

I imagined a scenario this morning: You could point this module at a directory full of images and use it to get image width, height, colors, geolocation, etc. Would be pretty useful for building an image viewer with Electron...

[elingerojo]

@zeke

I imagined a scenario this morning: You could point this module at a directory full of images and use it to get image width, height, colors, geolocation, etc.

You can test drive this