Strange issue with screenshotting dynamic websites about cdp

[mafredri]

Thanks for the sample code, I tried running it but I can't spot the problem. To me, the Chrome screenshot looks better / more correct (dehtml seems to render incorrectly):

What should I be looking for? PS. Using Chrome Beta (66) on macOS.

[s3rj1k]

@mafredri bottom page, for me site header is doubled
proper footer should look like this
https://prnt.sc/isg5t0

using Chromium 65.0.3325.146 on Linux

[mafredri]

Hmm, I can't see the doubled site header, but it looks like pikabu.ru uses infinite scrolling. Most likely that has something to do with the footer looking incorrect.

E.g. maybe just as Chrome sets the page viewport, the site starts loading more content.

Not sure how to deal with that, maybe:

• Try to disable JavaScript, if possible
• Analyze page source and disable infinite scrolling (e.g. replace JS function)
• Use network interception before taking the screenshot and block network requests

Maybe this also classifies as a headless bug? Might be worth reporting to the headless mailing list and/or reporting on the chromium bug tracker.

[s3rj1k]

can you provide example on "Use network interception before taking the screenshot and block network requests" ?

[s3rj1k]

lol, with Google Chrome 66.0.3359.33 beta everything is superb
still would be nice to test with block network requests

[mafredri]

Heh 😅, updating Chrome often helps (esp. when using headless).

You can see some example uses of interception in #40, #36, #32. Although, some of those might be a bit outdated since Chrome changed the network interception API (I believe #40 at least is up-to-date).

But most likely you just need to begin interception, there's probably no need to "reply" to (e.g. continue) them.

[s3rj1k]

@mafredri can I just use c.Page.StopLoading(ctx) instead of interception?

[mafredri]

I'm not an expert, but judging from the documentation I don't think it does anything to prevent future network requests.

[s3rj1k]

hmm, that could be true, still has to be enough for quick screenshot after stopping page load.

that for your support, closing this one. 👍