New Runner demands same config for all reporters about purescript-spec

[felixSchl]

We could use something like:

_run
  :: ∀ e r
   . { timeout :: Maybe Int, slow :: Int | r }
  -> Spec (RunnerEffects e) Unit
  -> Producer Event (Aff (RunnerEffects e)) (Array (Group Result))

What did you have in mind that is not possible right now?

[owickstrom]

Sorry, I think I was a bit unclear. It's not the Config type passed to _run that is the problem - it's the c and s type parameters in this signature that makes it problematic:

run'
  :: ∀ c s e
   . Config
  -> Array (BaseReporter c s (Eff (RunnerEffects e)))
  -> Spec (RunnerEffects e) Unit
  -> Eff  (RunnerEffects e) Unit

Both c and s are used for all BaseReporters in the array. This means that one cannot use multiple reporters (I think...):

main = run [specReporter, xunitReporter { some = config } do 
  ...

... as specReporter and xunitReporter have different config and state types. What I'm thinking is if we can remove the c and s type parameters, hiding those specific types in each reporter. The reporter config is easy to just close over, but the state is passed around. Maybe StateT could be more handy for this case?

[felixSchl]

Ah yeah, that's something I was thinking about when writing the code, actually, I remember now.
I think no matter how you skin this in the end that list needs to be homogeneous, so even if we were to use StateT or ReaderT, you'd end up in the same position. I think the only way to hide this would be to use existential types somehow

[felixSchl]

Actually it was a lot easier than I thought, you were right, closing over config is the way to go, no need to keep it in base reporter.

[owickstrom]

The removal of c looks good, solves the part with configs. Still troublesome with the state though. I have an idea with extensible records, trying it now. :)

[owickstrom]

Extensible records will not be a good solution to this, it seems. :/

Also, very loose thought that I'm exploring: maybe Reporters can be functions of type Producer ... -> Producer ..., and thus can be combined using function composition instead of being passed as an array.

[owickstrom]

Just to give you a hint of where I'm going with this, here's what I've got right now. Will have to rewrite the reporters:

type TestEvents e = Producer Event (Aff e) (Array (Group Result))

type Reporter e = TestEvents e -> Effect (Aff e) (Array (Group Result))

-- Run the spec, report results and exit the program upon completion
run''
  :: ∀ e
   . Config
  -> Reporter (RunnerEffects e)
  -> Spec (RunnerEffects e) Unit
  -> Eff  (RunnerEffects e) Unit
run'' config reporter spec = void do
  runAff onError onSuccess (P.runEffect (reporter (_run config spec)))

  where
    onError :: Error -> Eff (RunnerEffects e) Unit
    onError err = do withAttrs [31] $ logShow err
                     Process.exit 1

    onSuccess :: Array (Group Result) -> Eff (RunnerEffects e) Unit
    onSuccess results = if successful results
                        then Process.exit 0
                        else Process.exit 1

Not sure if the Reporter type will make composition possible, or if it has to be Producer ... -> Producer, but in the context of run'' it should return an Effect.

[felixSchl]

Would be cool to have the code pushed to look at, looks interesting

[owickstrom]

Yeah, I'll push to the PR branch once it's in a better state. Probably in about 12 hrs.

[owickstrom]

Not entirely done yet, but I've pushed some stuff (8b043bc), and would love some feedback.

Reporters now have type Pipe a a (Eff e) (Array (Group Result)), and compose using regular >->. The only trouble is the summary thing, but I thought we might implement that as a separate reporter that one can put in the end of the chain. Something like specReporter >-> summaryReporter. What do you think?

Silly example of how reporters compose:

-- composing reporters (interleaved effects will cause messy console logs)
main = run (specReporter >-> dotReporter { width: 80, slow: 800 }) do
  ...

[felixSchl]

Personally, I feel reporters should only be reporting (consume) but now they appear like transformers (pipes), where in your example it looks as if specReporter was transforming input and then passing it to dotReporter. For example, we could have done this using parTraverse: 8b043bc#diff-6284d4c18f32617e31426063da6476aeL172 but I hadn't thought of that at the time.

[owickstrom]

OK, yeah it might be a bit confusing with >->. Can't find parTraverse anywhere though? tee in Haskell pipes could be something, if that can be built in purescript-pipes?

[felixSchl]

Yeah can and should be built into purescript-pipes, not sure why it's not. I must have missed it. Here's parTraverse: https://pursuit.purescript.org/packages/purescript-parallel/2.1.0/docs/Control.Parallel#v:parTraverse. We'd probably be more interested in parTraverse_ though

[owickstrom]

OK, so the tee version could be something like:

main = run (tee specReporter >-> tee (dotReporter { width: 80, slow: 800 }) >-> printSummary) do
  ...

Do you have anything in mind regarding parTraverse_? Not sure how it would fit in.

[owickstrom]

Any thoughts on tee? It seems we're very close to closing this issue. Just gotta agree on this last bit. 😃

[felixSchl]

Sorry! Is the final >-> printSummary up there a requirement to satisfy the type of the first argument to run?

[owickstrom]

No problem at all! :) Yeah, I was thinking the argument to run should be an Effect instead of a Pipe, and that printSummary could take a Pipe with return type Summary and return an Effect. Sounds reasonable?

[felixSchl]

Depends on what we want out of it. A summary could be considered optional. We could also remove 90% of our reporter code and just make sure our tap reporter is sound and then plug in one of the many existing reporters in. Would be less fun, but has other advantages.

[owickstrom]

Less code to maintain is quite nice. :) Maybe just skip the reporters argument for run, and provide a run' which takes a Consumer, for those that want to plug something specific into it?

[felixSchl]

Here is how purescript spec is currently visualised in my head:

    run!
      |          .- reporter 1 -> ()
      |         /
    event | report -> ()
      |         \
      v          `- reporter n -> ()
   summarize

I think were you are going is:

    run!
      |
    event | tee report 1 | tee report n | summarize

Is this correct? Having it visually in front of me it does look appealing. How would you fold over the list of reporters or do we do a single tee and combine all reporters (back to parTraverse?)

    run!
      |
    event | report | summarize
             /  \
   reporter 1    reporter n

[owickstrom]

The implicit tap reporter could be a bit on the dull side if you just run pulp test, though.

[felixSchl]

Maybe just skip the reporters argument for run, and provide a run' which takes a Consumer, for those that want to plug something specific into it?

That sounds good to me

[owickstrom]

The mocha stuff already has its own custom runner anyway, so nothing to worry about there.

[owickstrom]

The last diagram looks like in my head. 👍 Not sure what you mean with fold over the list of reporters? I think tee (report 1) >-> tee (report n) will pipe to all reporters properly, right?

[owickstrom]

and run could be run' (tee tapReporter >-> discard) where discard would be some function just ignoring the return value and returning an Effect.

[owickstrom]

Hand-wavy, but yeah...

[felixSchl]

Yeah, let's do this :) I suppose sth like fold (>->) $ tee <$> reporters

[owickstrom]

OK, but that would leave us with the type problem again, having differently typed reporters in an array. Leaving the teeing to the user solves it. 👍

[owickstrom]

I'm leaning towards keeping the reporters for now, and a run that takes a reporter (or teed chain of reporters), and always ending them with a printSummary. The only hustle for users then is rewriting run [reporter1, reporter2] to run (reporter1 >-> tee reporter2).

If someone wants full control they could use run' and not have the implicit printSummary. The tap output might be one of those cases?

Is that OK?

[felixSchl]

I don't think you would have an issue with differently typed reporters in an array if we close over the reporter options.

I'm leaning more towards not exposing >-> as it requires users to have to learn how pipes works. I agree on having a low level run' though

[owickstrom]

You are a genius. Arrays do work, and we'll not have to expose pipes. Users won't even have to change anything in their tests if they used run [ specReporter ] ... . 👍 👍 👍