Assert vs Must vs Chai about dendritic

[kaievns]

There's no suggestion of there being bugs in assertion libraries. I think there's a miscommunication. When I say that bugs might hide in thing.must.have.ownKeys(['foo', 'bar']) it's not that I suspect the implementation is faulty.

Ah, my bad, i did misunderstand you there. I actually agreed about this. It is actually usually even worse and looks something like 'smth.smth.smth.must....'

To anyone who heard of the Demeter's law this thing smells bad. And that is partly a reason why i slightly favour the 'expect' syntax. It is somewhat functional and doesnt fall on its face on undefineds.

Although coming to your original question which one produces a more bug free result, i think that both assert and should will fail equally. It just the should will fail with 'undefined doesnt have property named "must"'. Which sucks, but hey it is javascript, i shouldnt really complain :)

[moll]

For custom error messages, there is the message arg in the constructor (https://github.com/moll/js-must/blob/master/doc/API.md#Must). Only available through the function-syntax though:

var demand = require("must")
demand(undefined, "The undefined undefineds").be.undefined()

I don't think the Demeter thing applies, or should apply, to reading properties out of nested objects/types. If it did, denormalization would the "solution". There is reason behind calling out modifying internals of a parent object through its nested objects, but for reading, I wouldn't raise it.

[kaievns]

I don't really have a strong inclination here tbh. I often find myself going for the expect as I find it has both, assert and should parts in it. but that's more of a habit than a real preference.

one thing about asserts that bugs me a bit is that it's a bit clunky and inconsistent. i feel like expect(smth).to.eql(smth) is cleaner than assert.deepEqual(smth1, smth2), it's just flows with the BDD way of wording things out nicer than asserts in my opinion.

but, as i said i don't really care as long as it is consistent throughout a project.

make it configurable?

[nwinch]

I prefer fluent assertions, and I prefer must. One thing to consider is that by promoting and using must we're also helping promote one of our own. That's a nice thing to do :)

[davidbanham]

It's certainly a nice thing to do, but the core of the decision has to be what's the best solution. There are plenty of my own projects we don't use because they're not the best thing for the job.

The extra syntactic magic in assertion libraries is just more places for bugs to hide. What exactly does .toExist test again? Is it falsiness or does it test typeof === 'undefined'? It's errors in memory or understanding like that that lead to a green test suite with a broken codebase. That's a really hard class of bug to recover from.

assert.equal(typeof 'foo', 'string'); is completely unambiguous. Test code should be the dumbest, most pedestrian code you ever write. It should be absolutely unimpeachable in its function and its intent. It's okay for it to be verbose, it's okay for it to be clunky. It's never okay for it to be obtuse.

[kaievns]

that's a great opinion. and i'm fully agreed with what it says, but i differ a bit in terms of interpretation.

1. I wouldn't worry about stability of a 1000+ stars assertion library. I am vastly more concerned about my own code than the assertion library i use. I know that in 99.9999% of cases it will my fault.
2. asserts while dead simple are a bit limited and tend to obscure the intent. like in your case it will fail saying array is not equal string, which is not what you're actually testing there.
3. Asserts can be chunky. We had equal and eql since Java 1.4. .deeplyEqual makes me cringe a bit, like it was the first assertion library that person used in their life.

Let me summarize it the old fashioned way with pros and cons:

asserts

Pros

• Dead simple
• Available out of the box
• Used since 1999

Cons

• Limited in terms of granularity
• Tend to obscure the intent
• Can be hard to read as english like text

Shoulds

Pros

• Very compact syntax
• Expansive and granular, allows to specify intent clearly
• More readable than asserts, although can feel weird

Cons

• Pollutes the prototype
• Explodes when called on undefined

Expect

Pros

• IMHO the most readable of the three (as english like code)
• Expansive and detailed assertions
• Follows the BDD spec syntax guidelines
• Doesn't pollute the prototype
• Doesn't explode on undefines
• Used by majority of devs in different languages

Cons

• Mostly hated by assert people and creates tension
• Can be seen as a bit verbose

That's basically my opinion on those three. Although I'm not a huge fun of expect and it took me some time to get over it. I find it to be practical and stable.

Overall I prefer to see:

expect(smth).to.be.a('string');
expect(smth).to.eql({bla: "blah"});

than

assert.equal(typeof smth, 'string');
assert.deeplyEqual(smth, {bla: "blah"});

In my opinion the first one is a bit more compact, more clearly describes the intent, and it follows the BDD conventions.

again, my opinion is not that strong, the differences are rather marginal to me. but, i see this sort of reasoning a lot. when rspec converted from should to expect, we spent months arguing about it back and fourth. in the end the expect syntax won by a small margin and for pretty much reasons I have just described.

i just want to stress out that a lot of people feel rather strongly about this sort of things. and i'm fully respectful to that. hence, i suggest that this would be configureable. i don't really see it worth locking people in one way or another

[nwinch]

It's certainly a nice thing to do, but the core of the decision has to be what's the best solution. There are plenty of my own projects we don't use because they're not the best thing for the job.

I did not imply we should be using personal projects just because we made them. If we are to use an assertion library, it would be great to use a well built and tested one by a colleague where which we can help promote and improve the product. Using the best tool for the job has always been clear, and I haven't suggested otherwise.

The extra syntactic magic in assertion libraries is just more places for bugs to hide. What exactly does .toExist test again? Is it falsiness or does it test typeof === 'undefined'? It's errors in memory or understanding like that that lead to a green test suite with a broken codebase. That's a really hard class of bug to recover from.

I haven't experienced an issue like this before?

I get the point you're making, but I don't think we should avoid assertion libs. Be wary of issues like this, sure. But I've yet to have an issue with Must along the lines of what you're describing.

[Nicktho]

I'd like to point out that issue #38 may affect the assertion library or lack there of we go with.

[nwinch]

Possibly. must supports assertions on promises out of the box.

[kaievns]

so in order to work on #37 we need to come to a conclusion here. i recon there are two main issues.

1. which style of assertions do we prefer to use, assert, should or expect?
2. which exact assertion library we use for that?

[kaievns]

@davidbanham clearly prefers assert am i correct? but how much of the actual applications code is he going to write?

also @nwinch i understand wants to use must over chai

i kind of don't care that much about it. but my opinion is that chai is a pretty much an industry standard at this point and must doesn't provide any significant value on top of chai. i don't mind the switch, but i think we need to acknowledge that we are stepping off the mainstream with this and every next person we hire will have to go through this. also we need to think about our clients who we will eventually hand over the product. would they be thrilled about the change as well?

also i'm slightly inclined towards the expect syntax. again because it is a standard BDD syntax and used outside of javascript context. which makes sense to me in conjunction with mocha bdd environment.

@s-taylor was kind of quiet about the whole thing. and @lyntco was left out of the discussion completely

so can we all voice out our preferences here in maybe like:

1. I prefer - blah
2. I don't mind - blah
3. i don't like - blah

[nwinch]

Yeah, I think we have more votes For than Against using an assertion lib, so as not to block progress I think we should use it.

Regarding which one, I prefer the should style of assertions. Which must supports. I like the fluent style it promotes and it is more readable to me. Both should and expect styles are very similar though, I'd say should wins out for me purely on a personal preference.

I agree that chaijs is used more widely than must. More github stars, which implies more projects with a wider reach, and most likely more contributors. That is something we should strongly consider when weighing up speed of adoption.

I think must is technically a little stronger in the battle, as it has a few things - such as promise support - baked in. Also this. We've also used this on many projects so far with positive results so there is a level of familiarity that exists.

@MadRabbit We need to be careful about using "the community does X" argument style here, as it's relative to each developer. I agree that we should wisely consider what our perception of the community is doing, though for all of us, that will be different. That said, I do see chaijs used quite a bit in my travels around the open source country.

[s-taylor]

I prefer Must, but I only have tape as a point of reference, I've never used chai.
Must supports promises out of the box which I think is a pro if we're using promises everywhere.
I don't think the learning curve on must is high, yes it's slightly different to Chai, but it's not hard to pick up.
It also recognises the work of one of our devs (Andri who wrote it) and promotes more of our code which is nice.

All that being said, if everyone wants Chai it wouldn't bother me either.

[kaievns]

@nwinch absolutely. the point of this is not to define what a community does but to make sure that we are making a decision with a clear understanding of options.

So, I personally don't mind the should syntax, and, I think that guy with a crouch camera knows what he does. so as long as we understand the consequences which include:

1. possible dissonance with the future employees
2. possible dissonance with a client
3. possible lack of community support comparing to chai

I'm happy with the must lib & should syntax.

@davidbanham thoughts?

[lyntco]

I've been using assert(node) when doing small things, and expect(chai) when larger ish things.
I guess the nice thing about knowing the person who wrote must is that we might not need as much community support because we can just ask him XP I enjoyed must while I was on Ordermentum, but in the end my opinion is not so strong on any.
assert and expect probably come close mainly because I have used them the most, but learning to do a unfamiliar thing is always okay to me.

[nwinch]

We have some +1s for must and some "happy/don't mind", so this seems to have nosed it's way to the head of the pack.

[kaievns]

ok, so, it seems that must + should syntax is the thing that will make most of us either happy or work as an acceptable solution. lets wrap it up at that

[moll]

❤️

To beat a dead horse, I never wanted to write a testing library. I just had to after seeing the following gems in other libs. 😇

Quoting myself from a few years back:

I didn't create Must.js merely because of the property access design issue, although I do consider that to be such an awful decision that it makes me wary of other design choices of those projects. There was also a Chai.js's eql bug1 that was open almost half a year on something so elementary as comparing an empty object with an empty array that drove me up the wall. Should.js couldn't handle objects with valueOf2 for a long time. Bugs can occur (though such really shouldn't in testing libraries which should be exemplars of proper systematic testing), but poor quality of basics and slow responses aren't good.

As for a single assert function vs higher level libs, I too like simplicity, but I think diffs are worth it. Can't get a "foo" must be "bar" with assert(a == "bar"). And once one goes down assert.(deep)equals, one's already in abstraction land and then it's just a matter of TDD vs BDD API. With a limited stdlib like in JS, the other built-in assertions of a testing lib are convenient, too.

[kaievns]

@moll just out of curiosity why didn't you submit a PR with a fix instead of building a whole new library?

also, just for the future reference. what is your preferred assertion style at the moment, assert, should or expect?

[moll]

Redesigning APIs in backwards incompatible way isn't as easy as a quick PR. Chai is still with its property access years later. For small bugs, I'm all for PRs. Poor or not, design taste is a leadership problem not solvable by a PR.

In JavaScript, I find the RSpec/OO style (a.must.be.a.number()) and BDD style for the test harness (describe and it) to be most convenient.

[davidbanham]

If we're using a lib, I agree that must is a better choice than chai or should.

Let's unpack the readability concern further.

describe('fromulgator', () => {
  it('should flibber the gibbets', () => {
    return gabble.fromulgate(gibbets)
    .then(gibbets => {
      assert(gibbets.flibbered, 'The gibbets should be flibbered');
      assert.deepEqual(gibbets, gibbetFixture, 'The gibbets should be flibbered in the exact way we expected');
    });
  });
});

describe('fromulgator', () => {
  it('should flibber the gibbets', () => {
    return gabble.fromulgate(gibbets)
    .then(gibbets => {
      gibbets.flibbered.should.be.truthy();
      gibbets.should.eql(gibbetFixture);
    });
  });
});

I argue that the addition of specific assertion error messages to assert makes it much more readable than the should syntax alone. It doesn't look like must allows custom messages.

> must(true).be.false('OHAI THERE LOOK AT ME')
AssertionError: true must be false

Actually thinking about this further the fix for that is just an inline comment.

describe('fromulgator', () => {
  it('should flibber the gibbets', () => {
    return gabble.fromulgate(gibbets)
    .then(gibbets => {
      gibbets.flibbered.should.be.truthy(); // The gibbets should be flibbered
      gibbets.should.eql(gibbetFixture); // The gibbets should be flibbered in the exact way we expected
    });
  });
});

Leaving the above in place for posterity, but I'm willing to declare that the assertion lib style and the raw-assert style are at least as readable as each other. If anyone wants to mount a case for expect/should being more readable than assert I'm open to it.

Definitely agree that the diffed output from must's .eql is likely to be far more immediately useful than the output from .deepEqual. It's a benefit, but I think the size of that benefit is limited. When I'm TDDing with that test I'm likely to have something like:

console.log('gibbets is', JSON.stringify(gibbets, null, 2));

Before my assertions. Which, yes, is gross, but its lifetime is pretty limited.

@davidbanham clearly prefers assert am i correct? but how much of the actual applications code is he going to write?

Most of it already, and into the future hopefully more than I currently am! That's not actually the point, though. The choice here isn't made solely upon personal preference. A vote on who prefers using what doesn't get us the answer we're looking for. It's about which direction is going to result in fewer bugs over the lifetime of this application, and the applications built with this application.

The discussion seems to have moved away from that core premise. I have asserted (seewhatIdidthere) that assert is less prone to bugs due to:

1. Simpler API
2. No dependency tree

The counterargument appears to be summarisable into:

Must is less prone to bugs than assert since the improved developer experience of diffed output and richer types of assertions makes people more willing to write more tests.

Is there anything that I'm missing?

[kaievns]

i think you're missing the text in the it(.... block, that is where you supposed to put some docs about what this test is all about.

i recon that is a fundamental difference between TDD and BDD. the first does

function blah() {
  assert.equal(one, another, "what it all means");
}

and the second one does this

it("does this thing because of that", () => {
  one.must.eql(another);
});

the difference is that TDD docs the assertions, and BDD docs the behavior and the assertions are just to verify the behavior. as rule of thumb a good it block in BDD has just one assertion line.

so, technically speaking you're trying to do TDD within BDD, which imho defies the point

[davidbanham]

Completely agree that that text should communicate what your test is trying to achieve. If you're Doing It Right, that should be sufficient to also explain your assertions. Often, however, you find yourself needing to add some extra explanation for a specific assertion like "I'm checking X here because it implies the existence of Y and I can't check Y directly because of Z"

Anyway, I think the readability/explanatory text thing is a dead issue. The BDD style does an excellent job of explaining intent and code comments can fill in any little gaps that it leaves.

I want to focus the discussion on what I think is the core question. "Which direction results in fewer bugs?"

[kaievns]

i think, and that was my position from the very beginning, that there is no real difference between assertion styles when it comes to making sure that things are tested and the app is shipped with a lower probability of an explosion at a runtime.

the only difference between those three I think is the "aesthetics" of it. namely, if i need to work on this project tomorrow, how long it will take me to stare into the screen before I'll be confident to make changes?

that is why i prefer to be in a consistent BDD mindset. because it is fairly standard and can be grasped by another developer quickly.

for example tape + asserts is a fine option for small cases. but as the complexity grows you have to hack things around as both tape and asserts are limited. and as there is no standard way of hacking them, you end up with something different every time. and then you hate yourself everytime when you have to return to a project 6 months later, because all deps are moved on and half of your things explode, and you don't remember why you wrote them this way or another

[davidbanham]

I disagree that there's no real difference.

I have asserted (seewhatIdidthere) that assert is less prone to bugs due to:

1. Simpler API
2. No dependency tree

Also:

for example tape + asserts is a fine option for small cases. but as the complexity grows you have to hack things around as both tape and asserts are limited. and as there is no standard way of hacking them, you end up with something different every time. and then you hate yourself everytime when you have to return to a project 6 months later, because all deps are moved on and half of your things explode, and you don't remember why you wrote them this way or another

We're not talking about tape here, just asserts. Can you provide an example of the kind of hacking you're referring to with asserts?

[nwinch]

Let's move the BDD/TDD discussion to Flowdock and continue the core question discussion here. I'll kick it off over there.

[kaievns]

i kind of disagree with your disagreement

1. assertion libs are normally very well tested, and i'd be infinitely more concern about bugs in my own code
2. from coding perspective require("assert") and require("chai") are pretty much the same
3. considering the size of any ORM or babel, an assertion lib isn't going to make a dent in your deps footprint

as for hacking asserts, i used to quite often add assert.includes, because the alternative

assert(list.indexOf(smth) !== -1);

almost physically hurts

[kaievns]

@nwinch i think TDD vs. BDD is the right context for this discussion. without it, this doesn't make much sense, then either one of this is just an == on steroids.

i think doing TDD style assertions within a BDD spec is a blasphemy, and hence the choice depends on this context

[nwinch]

Yep for sure, I can see the relevance and overlap, though I had interpreted Dave's message here:

I want to focus the discussion on what I think is the core question. "Which direction results in fewer bugs?"

As vanilla assert vs assertion libs - is that correct @davidbanham?

Apologies if I've misinterpreted, I don't want to throw a spanner in the works, just thought we could split concerns if there are concerns worth splitting. Please carry on if I had that wrong.

[davidbanham]

i kind of disagree with your disagreement

Brilliant! Let's hash it out.

assertion libs are normally very well tested, and i'd be infinitely more concern about bugs in my own code

There's no suggestion of there being bugs in assertion libraries. I think there's a miscommunication. When I say that bugs might hide in thing.must.have.ownKeys(['foo', 'bar']) it's not that I suspect the implementation is faulty. My concern is that not every developer on the project will have an intimate understanding of what exactly ownKeys does and that differing or incorrect assumptions about that may introduce bugs into our application.

considering the size of any ORM or babel, an assertion lib isn't going to make a dent in your deps footprint

Which is part of the reason we use neither!

assert(list.indexOf(smth) !== -1);

assert(!_.includes(list, smth));

Lodash triggers the same objection in me as assertion libraries, btw. Really rich API that lends itself to misunderstandings about implementation details. I let that one slide due to the amount of additional utility is brings being worth the potential for confusion. If we can prove the same with assertion libs I'm sold.

[davidbanham]

You've interpreted my statement correctly, Nath. Nikolay disagrees, though, and I think his disagreement is reasonably founded. Let's keep the discussion all in one place.

[davidbanham]

Forget it. Let's roll with Must. I think that lodash point has tipped me over the edge. We're trading one potentially confusing API for another. Assert can also be confusing in different ways. If people are this keen on BDD style assertions let's just go for it. I think vanilla assert still has the edge, but this discussion has convinced me that the gap is small enough that it's not a big deal.

[kaievns]

Totally agreed. We always can awk/sed this later

[kaievns]

I agree that LoD can be dismissed in this case as 'must' is an Object property and future calls are self referential. I was using it as a reference to why constructions like 'smth.must.be.smth' might feel weird and bug prone

[davidbanham]

This one isn't LoD for me. The concern is that I'm a very stupid man with a short attention span for documentation. When it comes to test code, I'd rather have 5 lines of very pedestrian code that have a very obvious purpose than 5 characters of clever API call.

If I come across an assertion library (utility library, etc) I'm very unlikely to ever go and read all the docs for its API. Chances are I'm going to assume what it does based off the name, maybe have a quick skim of the docs and get bored.

I'm happy to use those kinds of libraries and helpers in my application code, because if I do something boneheaded I know my tests will pick it up, along with all the other stupid shit I did.

I'm reluctant to use Clever Things in my test code. If I've done something dumb in my tests... that bug is at least going to a human tester and probably to production.