Unable to collect code coverage when using `ChannelsLiveServerTestCase` about channels

[carltongibson]

Ok, well I can't see anything in obvious... there's a lot going on in the repo so I guess trying to reduce it would be the next phase.

I can see if I can find it next time I'm looking this area.

[carltongibson]

Thanks for the report. I'm not immediately sure what the problem is. Have you explored coverage's concurrency option?

[kierangillblueberry]

I have. It wasn't clear to me which Daphne used so I tried every concurrency option in their docs. I also adding coverage directly to sitecustomize.py as recommended in the docs but did not make any headway.

There are more details in the reproduction's README.md.

[carltongibson]

I'd imagine you'd need a multi-value like multiprocessing,thread 🤔

[kierangillblueberry]

I just tried every combination of the concurrency options available to me:

multiprocessing,greenlet,thread
multiprocessing,greenlet
multiprocessing,thread
greenlet,multiprocessing
greenlet,thread
thread,multiprocessing
thread,greenlet

I am still unable to collect coverage. gevent and eventlet weren't dependencies of anything I am using so I didn't try them.

[kierangillblueberry]

@carltongibson I've created a smaller reproduction here: https://github.com/kierangillblueberry/code-coverage-ChannelsLiveServerTestCase_small

• README.md
• tests.py
• views.py
• run LiveServerTestCase tests
• runChannelsLiveServerTestCase tests

[carltongibson]

Thanks @kierangillblueberry 👀

[kierangillblueberry]

Hey @carltongibson, following up on this. I can also take a stab at this fix but I'm not sure where to start. Any pointers would be helpful! Also happy to hop on a call if you think it'd be helpful.

[carltongibson]

Hi @kierangillblueberry, I haven't had a chance to sit down with it to be honest. First pass would be to see if we can pick down exactly what's happening.

[carltongibson]

parallel = True

Q: does coverage work running with just a single process (I guess yes)?

[johanneswilm]

@carltongibson @kierangillblueberry Did you make any headway with this?

I am running into the same issue. switching from django/tornado to django-channels decreases the coverage by a lot as coverage no longer can pick up the correct percentage [1]. I used multiprocessing,thread already when running with tornado.

[1] fiduswriter/fiduswriter#1282 (comment)

[carltongibson]

@johanneswilm No, not yet. There are a couple of related issues with the live server test case. It needs someone to sit down with it, and really work out what's going on. (I've been, and am, busy elsewhere and haven't had that time yet.)

[johanneswilm]

@carltongibson @kierangillblueberry I spent around a day adding print statements various places in my virtual environment... and it looks to me like I have gotten it to work. What was needed was to update coverage to a recent version (7.4.4) and then to enable the sigterm coverage option. ChannelLiveServerTestCase creates a separate daphne process for each test. LiveServerTestCase just creates a single process for an entire test class. I don't know why there is a difference (it creates other problems for me), but at the end of the process, both test classes call the terminate() method on the process, which is equivalent to issuing a sigterm on the process. I have not looked into why this isn't an issue for LiveServerTestCase as it also terminates the process after running tearDownClass(), but in the case of ChannelLiveServerTestCase this was what was lacking in order to make each process write down its findings to disk.

@kierangillblueberry BTW - you don't need a PostgreSQL db for ChannelLiveServerTestCase - you can just a disk-writen sqlite db instead.

I have put it all together in this PR: kierangillblueberry/code-coverage-ChannelsLiveServerTestCase_small#1

I assume that the two of you will figure out if any changes are needed either here or in coverage. At the very least I would document this behavior to not have others waste time on this.

[carltongibson]

@johanneswilm Good digging. Yes, that makes sense. At the least a docs section about collecting coverage here would help.

(I need to dig in a bit more to think about whether there's anything to change.)