Won't run on machine with more than 20 CPU cores about jmx_exporter

[ihulsbus]

Hi, We are also experiencing this issue at random moments with javaagent 0.9 and 10 machine cores.

Am I correct to say that the switch to HTTPserv in version 0.10 should be a fix for this issue?

[rathers]

Pull Request #123 resolves this for my use case.

I don't fully understand why this works but it does seem to! The bit that confuses me is that the thread limit of 10 seems to kick in when the cores are >20, rather than >10. The server I'm testing this on has 12 cores plus hyper-threading so the OS presents 24 cores, so maybe the HT is creating the x2 difference.

Obviously this isn't a general fix but it works for my use-case so thought I'd contribute in case it's useful

[davidbirdsong]

seeing the same thing. I had to disable 15/30 cores to get scrapes to work.

2017-03-16 08:10:47.341:WARN:ipjsoeji.nio:Dispatched Failed! SCEP@18172791{l(/127.0.0.1:58044)<->r(/127.0.0.1:8013),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@6d5cdac1,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to io.prometheus.jmx.shaded.org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@4b7c54dd

re-enabling the cores and bumping max threads also fixes for me.

[jwdevs]

+1
The same for me. Version 0.8 does not work for machines with larger number of cores. Version 0.7 works fine.

[brian-brazil]

Can you try it with an updated version of jetty? The one we're using is fairly old, so this may be a fixed issue.

[jwdevs]

I can give it a try if you build it. :)

[jamespic]

This is still an issue on Jetty 9.4.2, although the error messages are more explicit:

java.lang.IllegalStateException: Insufficient threads: max=10 < needed(acceptors=3 + selectors=8 + request=1)

Jetty chooses how many acceptors and selectors to run depending on the number of cores available. To make this work in general, if we can't trust Jetty to decide its own thread pool size, we'd need to explicitly add connectors with the desired number of acceptors and selectors preconfigured.

[brian-brazil]

That is wonderfully broken behaviour.

[j03wang]

I've noticed this happening even with the 0.9 jars, but not consistently. Might there be additional issues for this? I'm trying a manual build with the updated version of Jetty in the mean time. Thanks!

[brian-brazil]

Could you try at head? We just made some changes that'll affect this.

[j03wang]

Yup--what I'm doing now. Will let you know if I continue to see the issue.

[bencoughlan]

Has this been resolved? I'm seeing this on the 0.8 prometheus engine, and it's got the version of jetty that you've rolled 0.9 back to.

[cfrantsen]

Should work, I have been running HEAD on multiple 16 core machines for a while now.

[rynbrd]

I'm experiencing this same bug, but on a machine that claims to have two cores. Currently running 0.9.

I'm working to upgrade to HEAD and will see if that fixes anything.

[rynbrd]

I'm running HEAD now but it had no effect. Still spewing these errors:

Jul 01 16:49:37 ip-10-32-208-106.us-west-2.compute.internal docker[13473]: 2017-07-01 16:49:36.314:WARN:ipjsoeji.nio:Dispatched Failed! SCEP@4b2c2644{l(/10.32.208.181:47724)<->r(/10.32.208.106:9242),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@37afadac,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to io.prometheus.jmx.shaded.org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@abea96f
Jul 01 16:49:37 ip-10-32-208-106.us-west-2.compute.internal docker[13473]: 2017-07-01 16:49:36.314:WARN:ipjsoeji.nio:Dispatched Failed! SCEP@8862399{l(/10.32.208.181:52106)<->r(/10.32.208.106:9242),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@23b2f704,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to io.prometheus.jmx.shaded.org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@abea96f
Jul 01 16:49:37 ip-10-32-208-106.us-west-2.compute.internal docker[13473]: 2017-07-01 16:49:36.314:WARN:ipjsoeji.nio:Dispatched Failed! SCEP@593a9d28{l(/10.32.208.181:49986)<->r(/10.32.208.106:9242),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@7c4c0f19,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to io.prometheus.jmx.shaded.org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@abea96f
Jul 01 16:49:37 ip-10-32-208-106.us-west-2.compute.internal docker[13473]: 2017-07-01 16:49:36.314:WARN:ipjsoeji.nio:Dispatched Failed! SCEP@63570cef{l(/10.32.208.181:47992)<->r(/10.32.208.106:9242),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@65e5b995,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to io.prometheus.jmx.shaded.org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@abea96f

[j03wang]

Yep--seeing this issue crop up again at HEAD.

[bencoughlan]

Hi ih16, It does indeed.

[anton-opsguru]

This issue is still easily reproducible on 20+ core machines on .
Hitting: java.lang.IllegalStateException: Insufficient threads: max=10 < needed(acceptors=3 + selectors=8 + request=1)
I am running on :

• Java 1.8_144
• jmx_prometheus_javaagent-0.10.jar
  Full stack trace:

2017-09-27 18:49:23.541:INFO::main: Logging initialized @7596ms to io.prometheus.jmx.shaded.org.eclipse.jetty.util.log.StdErrLog
2017-09-27 18:49:26.138:INFO:ipjsoejs.Server:main: jetty-9.4.z-SNAPSHOT
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.IllegalStateException: Insufficient threads: max=10 < needed(acceptors=3 + selectors=8 + request=1)
at io.prometheus.jmx.shaded.org.eclipse.jetty.server.Server.doStart(Server.java:414)
at io.prometheus.jmx.shaded.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:84)
... 6 more
FATAL ERROR in native method: processing of -javaagent failed

[bencoughlan]

Upgrade to the 1.0 engine, it'll sort everything Sent with BlackBerry Work (www.blackberry.com)

…

________________________________ From: anton-opsguru <[email protected]> Sent: 27 Sep 2017 20:37 To: prometheus/jmx_exporter <[email protected]> Cc: Ben Coughlan <[email protected]>; Comment <[email protected]> Subject: Re: [prometheus/jmx_exporter] Won't run on machine with more than 20 CPU cores This issue is still easily reproducible on 20+ core machines on . Hitting: java.lang.IllegalStateException: Insufficient threads: max=10 < needed(acceptors=3 + selectors=8 + request=1) I am running on : * Java 1.8_144 * jmx_prometheus_javaagent-0.10.jar Full stack trace: 2017-09-27 18:49:23.541:INFO::main: Logging initialized @7596ms to io.prometheus.jmx.shaded.org.eclipse.jetty.util.log.StdErrLog 2017-09-27 18:49:26.138:INFO:ipjsoejs.Server:main: jetty-9.4.z-SNAPSHOT Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) Caused by: java.lang.IllegalStateException: Insufficient threads: max=10 < needed(acceptors=3 + selectors=8 + request=1) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.Server.doStart(Server.java:414) at io.prometheus.jmx.shaded.org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:84) ... 6 more FATAL ERROR in native method: processing of -javaagent failed — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#122 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AY3BAaoy4pKPq1Q-k6HGZ4BGqjIaade5ks5smqQEgaJpZM4MemZR>.