asyncio.exceptions.LimitOverrunError about yacron

[gjcarneiro]

From what I could gather here, this might be due to the fact that this subprocess that yacron is trying to run is generating a line that is over 65k long.

I'll have to think the best way to handle it. In the mean time, you might want to avoid having this process generate such long lines. Definitely a bug, though, thanks for the report!

[mohnishkodnani]

Hi,
I saw the fix, thanks a lot. I had a question, is it possible to not ignore the line but get more than the chunk by specifying a bigger limit. In our case we have some very big JSON coming as response.

[gjcarneiro]

But what are you doing with that output?.. If you send to Sentry, it will ignore it, for the same reason, too big. If you send as email, I'm not sure, but if it's big it might get into similar problems.

If you just want to log it to the container's stdout, you can specify captureStdout: false and yacron will just let it pass through to stdout unchanged.

While it's technically possible to change the line limit, I'm not quite convinced it is really that important...

[mohnishkodnani]

We actually parse that output and then store fields in a database, that's our main application.

…

On Wed, Nov 10, 2021 at 9:00 AM Gustavo J. A. M. Carneiro < ***@***.***> wrote: But what are you doing with that output?.. If you send to Sentry, it will ignore it, for the same reason, too big. If you send as email, I'm not sure, but if it's big it might get into similar problems. If you just want to log it to the container's stdout, you can specify captureStdout: false and yacron will just let it pass through to stdout unchanged. While it's technically possible to change the line limit, I'm not quite convinced it is really that important... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#56 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABW4SVC5LKXJNRH25GQBMO3ULKQJ7ANCNFSM5HWIJZTA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[gjcarneiro]

Parse that output from where? You ask yacron to send as email and you parse the output from an email?

OK, fine, if you insist, I'll reopen to add a config option for this limit. Seems a convoluted use case, but fine, it's not that hard to make the limit configurable.

[mohnishkodnani]

yacron runs a job ( this is a python job too ), this job queries a service, fetches the json and parses it and puts in a DB.
If I run the job from outside yacron it works fine, but when invoked within yacron, I get this error.

[gjcarneiro]

Sure, but then it sounds like printing the json to stdout is just for debugging. In which case ignoring the very long line should have no effect in the ability of the script to do its job of parsing the json and storing in the DB.

My claim is that it is possibly ignoring this line is fine. I'll add the config option when I have more time, but it is possible that you are assuming that the job fails when in fact it succeeds?