GithubHelp home page GithubHelp logo

alibaba / arthas Goto Github PK

View Code? Open in Web Editor NEW
35.1K 1.1K 7.4K 48.7 MB

Alibaba Java Diagnostic Tool Arthas/Alibaba Java诊断利器Arthas

Home Page: https://arthas.aliyun.com/

License: Apache License 2.0

Java 79.79% Shell 1.27% Batchfile 0.37% JavaScript 7.43% CSS 0.01% HTML 1.28% Dockerfile 0.03% Roff 0.11% Groovy 0.01% ActionScript 0.01% C++ 0.19% SCSS 0.01% Vue 7.86% TypeScript 1.64%
java agent trace jvm arthas trouble-shooting diagnosis classloader alibaba

arthas's Introduction

Arthas

arthas

Build Status download maven license Average time to resolve an issue Percentage of issues still open Leaderboard

Arthas is a Java Diagnostic tool open sourced by Alibaba.

Arthas allows developers to troubleshoot production issues for Java applications without modifying code or restarting servers.

中文说明/Chinese Documentation

Background

Often times, the production system network is inaccessible from the local development environment. If issues are encountered in production systems, it is impossible to use IDEs to debug the application remotely. More importantly, debugging in production environment is unacceptable, as it will suspend all the threads, resulting in the suspension of business services.

Developers could always try to reproduce the same issue on the test/staging environment. However, this is tricky as some issues cannot be reproduced easily on a different environment, or even disappear once restarted.

And if you're thinking of adding some logs to your code to help troubleshoot the issue, you will have to go through the following lifecycle; test, staging, and then to production. Time is money! This approach is inefficient! Besides, the issue may not be reproducible once the JVM is restarted, as described above.

Arthas was built to solve these issues. A developer can troubleshoot your production issues on-the-fly. No JVM restart, no additional code changes. Arthas works as an observer, which will never suspend your existing threads.

Key features

  • Check whether a class is loaded, or where the class is being loaded. (Useful for troubleshooting jar file conflicts)
  • Decompile a class to ensure the code is running as expected.
  • View classloader statistics, e.g. the number of classloaders, the number of classes loaded per classloader, the classloader hierarchy, possible classloader leaks, etc.
  • View the method invocation details, e.g. method parameter, return object, thrown exception, and etc.
  • Check the stack trace of specified method invocation. This is useful when a developers wants to know the caller of the said method.
  • Trace the method invocation to find slow sub-invocations.
  • Monitor method invocation statistics, e.g. qps, rt, success rate and etc.
  • Monitor system metrics, thread states and cpu usage, gc statistics, and etc.
  • Supports command line interactive mode, with auto-complete feature enabled.
  • Supports telnet and websocket, which enables both local and remote diagnostics with command line and browsers.
  • Supports profiler/Flame Graph
  • Support get objects in the heap that are instances of the specified class.
  • Supports JDK 6+.
  • Supports Linux/Mac/Windows.

Online Tutorials(Recommended)

Quick start

Use arthas-boot(Recommended)

Downloadarthas-boot.jar,Start with java command:

curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar

Print usage:

java -jar arthas-boot.jar -h

Use as.sh

You can install Arthas with one single line command on Linux, Unix, and Mac. Copy the following command and paste it into the command line, then press Enter to run:

curl -L https://arthas.aliyun.com/install.sh | sh

The command above will download the bootstrap script as.sh to the current directory. You can move it any other place you want, or put its location in $PATH.

You can enter its interactive interface by executing as.sh, or execute as.sh -h for more help information.

Documentation

Feature Showcase

Dashboard

dashboard

Thread

See what is eating your CPU (ranked by top CPU usage) and what is going on there in one glance:

$ thread -n 3
"as-command-execute-daemon" Id=29 cpuUsage=75% RUNNABLE
    at sun.management.ThreadImpl.dumpThreads0(Native Method)
    at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:440)
    at com.taobao.arthas.core.command.monitor200.ThreadCommand$1.action(ThreadCommand.java:58)
    at com.taobao.arthas.core.command.handler.AbstractCommandHandler.execute(AbstractCommandHandler.java:238)
    at com.taobao.arthas.core.command.handler.DefaultCommandHandler.handleCommand(DefaultCommandHandler.java:67)
    at com.taobao.arthas.core.server.ArthasServer$4.run(ArthasServer.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@6cd0b6f8

"as-session-expire-daemon" Id=25 cpuUsage=24% TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at com.taobao.arthas.core.server.DefaultSessionManager$2.run(DefaultSessionManager.java:85)

"Reference Handler" Id=2 cpuUsage=0% WAITING on java.lang.ref.Reference$Lock@69ba0f27
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.Reference$Lock@69ba0f27
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

jad

Decompile your class with one shot:

$ jad javax.servlet.Servlet

ClassLoader:
+-java.net.URLClassLoader@6108b2d7
  +-sun.misc.Launcher$AppClassLoader@18b4aac2
    +-sun.misc.Launcher$ExtClassLoader@1ddf84b8

Location:
/Users/xxx/work/test/lib/servlet-api.jar

/*
 * Decompiled with CFR 0_122.
 */
package javax.servlet;

import java.io.IOException;
import javax.servlet.ServletConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;

public interface Servlet {
    public void init(ServletConfig var1) throws ServletException;

    public ServletConfig getServletConfig();

    public void service(ServletRequest var1, ServletResponse var2) throws ServletException, IOException;

    public String getServletInfo();

    public void destroy();
}

mc

Memory compiler, compiles .java files into .class files in memory.

$ mc /tmp/Test.java

retransform

Load the external *.class files to retransform/hotswap the loaded classes in JVM.

retransform /tmp/Test.class
retransform -c 327a647b /tmp/Test.class /tmp/Test\$Inner.class

sc

Search any loaded class with detailed information.

$ sc -d org.springframework.web.context.support.XmlWebApplicationContext
 class-info        org.springframework.web.context.support.XmlWebApplicationContext
 code-source       /Users/xxx/work/test/WEB-INF/lib/spring-web-3.2.11.RELEASE.jar
 name              org.springframework.web.context.support.XmlWebApplicationContext
 isInterface       false
 isAnnotation      false
 isEnum            false
 isAnonymousClass  false
 isArray           false
 isLocalClass      false
 isMemberClass     false
 isPrimitive       false
 isSynthetic       false
 simple-name       XmlWebApplicationContext
 modifier          public
 annotation
 interfaces
 super-class       +-org.springframework.web.context.support.AbstractRefreshableWebApplicationContext
                     +-org.springframework.context.support.AbstractRefreshableConfigApplicationContext
                       +-org.springframework.context.support.AbstractRefreshableApplicationContext
                         +-org.springframework.context.support.AbstractApplicationContext
                           +-org.springframework.core.io.DefaultResourceLoader
                             +-java.lang.Object
 class-loader      +-org.apache.catalina.loader.ParallelWebappClassLoader
                     +-java.net.URLClassLoader@6108b2d7
                       +-sun.misc.Launcher$AppClassLoader@18b4aac2
                         +-sun.misc.Launcher$ExtClassLoader@1ddf84b8
 classLoaderHash   25131501

vmtool

Get objects in the heap that are instances of the specified class.

$ vmtool --action getInstances --className java.lang.String --limit 10
@String[][
    @String[com/taobao/arthas/core/shell/session/Session],
    @String[com.taobao.arthas.core.shell.session.Session],
    @String[com/taobao/arthas/core/shell/session/Session],
    @String[com/taobao/arthas/core/shell/session/Session],
    @String[com/taobao/arthas/core/shell/session/Session.class],
    @String[com/taobao/arthas/core/shell/session/Session.class],
    @String[com/taobao/arthas/core/shell/session/Session.class],
    @String[com/],
    @String[java/util/concurrent/ConcurrentHashMap$ValueIterator],
    @String[java/util/concurrent/locks/LockSupport],
]

stack

View the call stack of test.arthas.TestStack#doGet:

$ stack test.arthas.TestStack doGet
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 286 ms.
ts=2018-09-18 10:11:45;thread_name=http-bio-8080-exec-10;id=d9;is_daemon=true;priority=5;TCCL=org.apache.catalina.loader.ParallelWebappClassLoader@25131501
    @test.arthas.TestStack.doGet()
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        ...
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:451)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1121)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

Trace

See what is slowing down your method invocation with trace command:

trace

Watch

Watch the first parameter and thrown exception of test.arthas.TestWatch#doGet only if it throws exception.

$ watch test.arthas.TestWatch doGet {params[0], throwExp} -e
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 65 ms.
ts=2018-09-18 10:26:28;result=@ArrayList[
    @RequestFacade[org.apache.catalina.connector.RequestFacade@79f922b2],
    @NullPointerException[java.lang.NullPointerException],
]

Monitor

Monitor a specific method invocation statistics, including the total number of invocations, average response time, success rate, and every 5 seconds:

$ monitor -c 5 org.apache.dubbo.demo.provider.DemoServiceImpl sayHello
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 109 ms.
 timestamp            class                                           method    total  success  fail  avg-rt(ms)  fail-rate
----------------------------------------------------------------------------------------------------------------------------
 2018-09-20 09:45:32  org.apache.dubbo.demo.provider.DemoServiceImpl  sayHello  5      5        0     0.67        0.00%

 timestamp            class                                           method    total  success  fail  avg-rt(ms)  fail-rate
----------------------------------------------------------------------------------------------------------------------------
 2018-09-20 09:45:37  org.apache.dubbo.demo.provider.DemoServiceImpl  sayHello  5      5        0     1.00        0.00%

 timestamp            class                                           method    total  success  fail  avg-rt(ms)  fail-rate
----------------------------------------------------------------------------------------------------------------------------
 2018-09-20 09:45:42  org.apache.dubbo.demo.provider.DemoServiceImpl  sayHello  5      5        0     0.43        0.00%

Time Tunnel(tt)

Record method invocation data, so that you can check the method invocation parameters, returned value, and thrown exceptions later. It works as if you could come back and replay the past method invocation via time tunnel.

$ tt -t org.apache.dubbo.demo.provider.DemoServiceImpl sayHello
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 75 ms.
 INDEX   TIMESTAMP            COST(ms)  IS-RET  IS-EXP   OBJECT         CLASS                          METHOD
-------------------------------------------------------------------------------------------------------------------------------------
 1000    2018-09-20 09:54:10  1.971195  true    false    0x55965cca     DemoServiceImpl                sayHello
 1001    2018-09-20 09:54:11  0.215685  true    false    0x55965cca     DemoServiceImpl                sayHello
 1002    2018-09-20 09:54:12  0.236303  true    false    0x55965cca     DemoServiceImpl                sayHello
 1003    2018-09-20 09:54:13  0.159598  true    false    0x55965cca     DemoServiceImpl                sayHello
 1004    2018-09-20 09:54:14  0.201982  true    false    0x55965cca     DemoServiceImpl                sayHello
 1005    2018-09-20 09:54:15  0.214205  true    false    0x55965cca     DemoServiceImpl                sayHello
 1006    2018-09-20 09:54:16  0.241863  true    false    0x55965cca     DemoServiceImpl                sayHello
 1007    2018-09-20 09:54:17  0.305747  true    false    0x55965cca     DemoServiceImpl                sayHello
 1008    2018-09-20 09:54:18  0.18468   true    false    0x55965cca     DemoServiceImpl                sayHello

Classloader

$ classloader
 name                                                  numberOfInstances  loadedCountTotal
 BootstrapClassLoader                                  1                  3346
 com.taobao.arthas.agent.ArthasClassloader             1                  1262
 java.net.URLClassLoader                               2                  1033
 org.apache.catalina.loader.ParallelWebappClassLoader  1                  628
 sun.reflect.DelegatingClassLoader                     166                166
 sun.misc.Launcher$AppClassLoader                      1                  31
 com.alibaba.fastjson.util.ASMClassLoader              6                  15
 sun.misc.Launcher$ExtClassLoader                      1                  7
 org.jvnet.hk2.internal.DelegatingClassLoader          2                  2
 sun.reflect.misc.MethodUtil                           1                  1

Web Console

web console

Profiler/FlameGraph

$ profiler start
Started [cpu] profiling
$ profiler stop
profiler output file: /tmp/demo/arthas-output/20211207-111550.html
OK

View profiler results under arthas-output via browser:

Arthas Spring Boot Starter

Known Users

Arthas has more than 120 registered users, View All.

Welcome to register the company name in this issue: #111 (in order of registration)

Alibaba Alipay Aliyun Taobao ICBC 雪球财经 顺丰科技 贝壳找房 vipkid 百度凤巢 有赞 科大讯飞 智联招聘 达美盛

Derivative Projects

Credits

Contributors

This project exists, thanks to all the people who contributed.

Projects

  • bytekit Java Bytecode Kit.
  • greys-anatomy: The Arthas code base has derived from Greys, we thank for the excellent work done by Greys.
  • termd: Arthas's terminal implementation is based on termd, an open source library for writing terminal applications in Java.
  • crash: Arthas's text based user interface rendering is based on codes extracted from here
  • cli: Arthas's command line interface implementation is based on cli, open sourced by vert.x
  • compiler Arthas's memory compiler.
  • Apache Commons Net Arthas's telnet client.
  • async-profiler Arthas's profiler command.

arthas's People

Contributors

0xflotus avatar allan-qlb avatar anjia0532 avatar beiwei30 avatar bohrqiu avatar dependabot[bot] avatar earayu avatar fatpandac avatar garenchan avatar hearen avatar hengyunabc avatar hollowman6 avatar jamesxuhaozhe avatar joecqupt avatar kylixs avatar loongs-zhang avatar pandaapo avatar qxo avatar ralf0131 avatar roger3581321 avatar sheepblueblue avatar sjha2048 avatar superheizai avatar wangji92 avatar winson-huang avatar xenoamess avatar xudaotutou avatar xuzhiyiyoh avatar yanyir avatar yx9o avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

arthas's Issues

Call for document translation

The current document is written only in Chinese.
I am planning to translate them into English.

Any volunteer who is willing to help on this?

Table of content Assignee Status
README @ralf0131 DONE
Installatoin @Hearen DONE
Quick start @Hearen DONE
Advanced usage @Hearen DONE
Commands @Hearen DONE
Commands/dashboard @Hearen DONE
Commands/thread @Hearen DONE
Commands/jvm @Hearen DONE
Commands/sysprop @Hearen DONE
Commands/getstatic @Hearen DONE
Commands/sc @Hearen DONE
Commands/sm @Hearen DONE
Commands/dump @Hearen DONE
Commands/jad @Hearen DONE
Commands/classloader @Hearen DONE
Commands/redefine @Hearen DONE
Commands/monitor @Hearen DONE
Commands/watch @Hearen DONE
Commands/trace @Hearen DONE
Commands/stack @Hearen DONE
Commands/tt @Hearen DONE
Commands/options @alfredzouang DONE
Commands/Basic commands @Hearen DONE
User cases
Release note @Hearen Done #138
Questions and Answers
CONTRIBUTING @Hearen Done #139

mvn clean install 失败

mac下执行mvn clean install报错:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.taobao.arthas.core.view.ObjectViewTest
Tests run: 19, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec <<< FAILURE!
testObjectTooLarge(com.taobao.arthas.core.view.ObjectViewTest)  Time elapsed: 0.004 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<... exceeds size limit:[100]> but was:<... exceeds size limit:[ 100, try to specify -M size_limit in your command, check the help command for more.]>
        at org.junit.Assert.assertEquals(Assert.java:115)
        at org.junit.Assert.assertEquals(Assert.java:144)
        at com.taobao.arthas.core.view.ObjectViewTest.testObjectTooLarge(ObjectViewTest.java:244)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
        at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
        at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)


Results :

Failed tests:   testObjectTooLarge(com.taobao.arthas.core.view.ObjectViewTest): expected:<... exceeds size limit:[100]> but was:<... exceeds size limit:[ 100, try to specify -M size_limit in your command, check the help command for more.]>

jad/watch/reset类失败

sc可查到类,但jad/watch/reset都失败,错误日志:

01 2018-09-18 09:46:52.510 ERROR [as-command-execute-daemon:arthas] [] [] [] jad: fail to decompile class: com.xx.xx.serialize.Transformer
java.lang.InternalError: class redefinition failed: invalid class
  at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method) ~[na:1.8.0_66]
  at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144) ~[na:1.8.0_66]
  at com.taobao.arthas.core.advisor.Enhancer.enhance(Enhancer.java:393) ~[arthas-core.jar:na]
  at com.taobao.arthas.core.command.klass100.JadCommand.processExactMatch(JadCommand.java:111) [arthas-core.jar:na]
  at com.taobao.arthas.core.command.klass100.JadCommand.process(JadCommand.java:97) [arthas-core.jar:na]
  at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82) [arthas-core.jar:na]
  at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18) [arthas-core.jar:na]
  at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111) [arthas-core.jar:na]
  at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108) [arthas-core.jar:na]
  at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:370) [arthas-core.jar:na]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
  at java.lang.Thread.run(Thread.java:756) [na:1.8.0_66]

连接失败

Connecting to arthas server... current timestamp is 1536904847
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host

日志内容

Arthas server agent start...
Arthas start to bind...
01 2018-09-14 14:04:01.241 ERROR [arthas-binding-thread:arthas] [] [] [] Error during bind to port 3658
java.lang.ExceptionInInitializerError: null
	at com.taobao.arthas.core.shell.ShellServerOptions.<clinit>(ShellServerOptions.java:29) ~[na:na]
	at com.taobao.arthas.core.server.ArthasBootstrap.bind(ArthasBootstrap.java:82) ~[arthas-core.jar:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
	at com.taobao.arthas.agent.AgentBootstrap.bind(AgentBootstrap.java:158) [arthas-agent.jar:na]
	at com.taobao.arthas.agent.AgentBootstrap.access$000(AgentBootstrap.java:15) [arthas-agent.jar:na]
	at com.taobao.arthas.agent.AgentBootstrap$1.run(AgentBootstrap.java:125) [arthas-agent.jar:na]
Caused by: java.lang.NullPointerException: null
	at java.io.Reader.<init>(Reader.java:78) ~[na:1.8.0_162]
	at java.io.InputStreamReader.<init>(InputStreamReader.java:72) ~[na:1.8.0_162]
	at com.taobao.arthas.core.util.IOUtils.toString(IOUtils.java:134) ~[arthas-core.jar:na]
	at com.taobao.arthas.core.util.ArthasBanner.<clinit>(ArthasBanner.java:33) ~[na:na]
	... 9 common frames omitted
Arthas server port binding failed! Please check $HOME/logs/arthas/arthas.log for more details.
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.taobao.arthas.agent.AgentBootstrap.bind(AgentBootstrap.java:158)
	at com.taobao.arthas.agent.AgentBootstrap.access$000(AgentBootstrap.java:15)
	at com.taobao.arthas.agent.AgentBootstrap$1.run(AgentBootstrap.java:125)
Caused by: java.lang.ExceptionInInitializerError
	at com.taobao.arthas.core.shell.ShellServerOptions.<clinit>(ShellServerOptions.java:29)
	at com.taobao.arthas.core.server.ArthasBootstrap.bind(ArthasBootstrap.java:82)
	... 7 more
Caused by: java.lang.NullPointerException
	at java.io.Reader.<init>(Reader.java:78)
	at java.io.InputStreamReader.<init>(InputStreamReader.java:72)
	at com.taobao.arthas.core.util.IOUtils.toString(IOUtils.java:134)
	at com.taobao.arthas.core.util.ArthasBanner.<clinit>(ArthasBanner.java:33)
	... 9 more

Error: Unable to access jarfile /root/.arthas/lib/3.0.4/arthas/arthas-core.jar

按照教程启动 arthas
选择程序进程号后
控制台报错:
Calculating attach execution time...
Attaching to 21106 using version 3.0.4...
Error: Unable to access jarfile /root/.arthas/lib/3.0.4/arthas/arthas-core.jar
attach to target jvm (21106) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.

排查:
/root/.arthas/lib/3.0.4 目录下没有 arthas 文件夹
/root/ 文件夹下没有 logs 文件夹

uname - a:
3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a:
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.5.1804 (Core)
Release: 7.5.1804
Codename: Core

java -version:
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

请问我应该如何启动?

【Arthas问题排查集】谁调用了System.exit/System.gc?

我们有时候可能会遇到这样的问题,进程莫名其妙的退出了,或者是发生了GC,通过日志或者是其他办法发现是有人调用了System.gc/System.exit,但是确不知道是谁干的。

如何找出这个罪魁祸首呢?一般来说,可以通过一段Btrace脚本来解决

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;

@BTrace
public class TraceSystemGCCall {
  @OnMethod(
    clazz="/.*/",
    method="/.*/",
    location=@Location(
      value=Kind.CALL,
      clazz="java.lang.System",
      method="gc"
    )
  )
  public static void onSystemGC() {
    jstack();
  }
}

类似这样的脚本(不保证能正常执行啊。。)经常容易写错,导致各种问题,有没有更好的办法呢?

今天我们来分享下,如何通过Arthas排查这类问题。

这里我们假设你已经了解下载,安装,启动Arthas的步骤。

第一步,由于java.lang.System是JDK自带的类,Arthas默认关闭了对JDK类的自带类的增强,需要通过options命令打开。

$ options unsafe true
 NAME    BEFORE-VALUE  AFTER-VALUE
-----------------------------------
 unsafe  false         true

第二步,使用stack命令,观察谁调用了java.lang.System#gc

$ stack java.lang.System gc
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 81 ms.
stack java.lang.System gc

然后,等待触发即可!

$ stack sun.management.MemoryImpl gc
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 27 ms.
stack sun.management.MemoryImpl gc
thread_name=RMI TCP Connection(5)-30.5.50.222;id=29;is_daemon=true;priority=9;TCCL=sun.misc.Launcher$AppClassLoader@14dad5dc
    @sun.management.MemoryImpl.gc()
        at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
        at sun.reflect.GeneratedMethodAccessor22.invoke(null:-1)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
        at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
        at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
        at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
        at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at javax.management.StandardMBean.invoke(StandardMBean.java:405)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1470)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1311)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1403)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:832)
        at sun.reflect.GeneratedMethodAccessor29.invoke(null:-1)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(AccessController.java:-2)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$255(TCPTransport.java:683)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$2/1638002256.run(null:-1)
        at java.security.AccessController.doPrivileged(AccessController.java:-2)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

这里有点偷懒,测试用的是jconsole的performace gc这个功能,背后调用的是sun.management.MemoryImpl的gc方法。效果是一样的。

那如果这个命令很长时间都不执行,我们不可能一直在那里等着呀,这种时候怎么办呢?

这时,我们可以使用arthas的批处理模式, 将arthas的命令写成一个脚本来执行。

具体方法是,第一步,准备一个脚本,例如system_gc.as,扩展名任意,这里为了便于记忆,采用了.as的扩展名,内容如下:

➜  tmp cat system_gc.as
options unsafe true
stack java.lang.System gc -n 1

其实就是把上面的两条命令放在一个文件里面而已。务必注意这里的-n,表示这个命令只执行一次,然后退出。如果不指定,Arthas 2.0会一直卡在那里,永远不退出,Arthas 3.0默认执行100次才会退出。在写批处理脚本的时候,请务必加上-n参数。

然后就可以执行这个脚本了, 默认脚本执行的结果会输出到标准输出,我们可以通过重定向,将输出持久化,最后通过最后一个&放到后台运行。

./as.sh -b -f system_gc.as 59863 > system_gc.out 2>&1 &

然后,在合适的时候去查看文件的结果即可。

cat system_gc.out

这种情况还有一个问题,就是如果我们的ssh session断了,这个进程就会退出。

如何保持后台运行呢?这里介绍2种办法:

  1. nohup让这个进程不响应SIGHUP信号
nohup ./as.sh -b -f system_gc.as 59863 > system_gc.out 2>&1 &
  1. 终端模拟类命令。screen或者tmux这类命令在线上其实非常有用,可以避免每次ssh连接断掉之后,原来连接中运行进程和数据丢失。具体用法如下:
sudo yum install -b current screen
screen
./as.sh -b -f system_gc.as 59863 > system_gc.out 2>&1

这时候可以去干别的事情了,即便网络断掉也没有问题。过一段时间之后,要查看之前的结果。如果连接断掉了,只需要输入以下命令:

screen -r

就可以恢复原来的session,并查看结果。

禁止掉数据回报功能

  1. 目前没有合适的服务器
  2. 不应该收集外部用户信息,除非用户同意
  3. 如果有部署服务器,可以考虑后续增加选项,让用户配置打开数据回报功能

监控jar包程序启动报错

[root@iz2ze3cbj61sjk0brcnz opt]# ./as.sh 
Arthas script version: 3.0.4
Found existing java process, please choose one and hit RETURN.
* [1]: 1376 xxx.jar
  [2]: 9137 xxx.jar
  [3]: 9163 xxx.jar
2
Calculating attach execution time...
Attaching to 9137 using version 3.0.4...
Error: Unable to access jarfile /root/.arthas/lib/3.0.4/arthas/arthas-core.jar
attach to target jvm (9137) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.
[root@iz2ze3cbj61sjk0brcnz opt]# ./as.sh debug 9137
Arthas script version: 3.0.4
Calculating attach execution time...
Attaching to 9137 using version 3.0.4...
Error: Unable to access jarfile /root/.arthas/lib/3.0.4/arthas/arthas-core.jar
attach to target jvm (9137) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.

序号跟进程号对不上

appdeploy@test1appsrv52vl:/app/arthas/bin$ ./as.sh
Arthas script version: 3.0.4
Found existing java process, please choose one and hit RETURN.
* [1]: 14787 /data/shell/../webapps/scp-devicemgmt-component-service-1.4.0.jar
  [2]: 16773 /data/shell/../webapps/scp-videointercom-app-service-1.4.0.jar
  [3]: 20357 /data/shell/../webapps/scp-visitor-app-service-1.4.0.jar
  [4]: 17158 /data/shell/../webapps/scp-accesscontrol-app-service-1.4.0.jar
  [5]: 22925 /data/shell/../webapps/scp-etl-component-service-1.4.0.jar
  [6]: 23246 /data/shell/../webapps/scp-searchengine-component-service-1.2.0.jar
  [7]: 19535 /data/shell/../webapps/scp-broadcast-app-service-1.4.0.jar
  [8]: 21839 /data/shell/../webapps/scp-map-app-service-1.4.0.jar
  [9]: 21137 /data/shell/../webapps/scp-mdm-app-service-1.4.0.jar
  [10]: 20754 /data/shell/../webapps/scp-communityplatformapi-component-service-1.4.0.jar
  [11]: 23323 /data/shell/../webapps/scp-security-app-service-1.4.0.jar
  [12]: 15195 /data/shell/../webapps/scp-usermgmt-component-service-1.4.0.jar
  [13]: 14044 /data/shell/../webapps/scp-iotbusconsoleapp-service-1.4.0.jar
  [14]: 24926 /data/shell/../webapps/scp-thirdpartyswitch-app-service-1.4.0.jar
  [15]: 12447 /data/shell/../webapps/gateway-server-1.4.0.jar
  [16]: 33440 /data/shell/../webapps/scp-parkinglot-app-service-1.4.0.jar
  [17]: 12065 /data/shell/../webapps/config-server-1.4.0.jar
  [18]: 17571 /data/shell/../webapps/scp-videogateway-app-service-1.4.0.jar
  [19]: 19108 /data/shell/../webapps/scp-patrol-app-service-1.4.0.jar
  [20]: 52581 /data/shell/../webapps/scp-imagemgmt-component-service-1.4.0.jar
  [21]: 17958 /data/shell/../webapps/scp-ismg-component-service-1.4.0.jar
  [22]: 21545 /data/shell/../webapps/scp-websocket-component-service-1.4.0.jar
  [23]: 19945 /data/shell/../webapps/scp-cardmgmt-app-service-1.4.0.jar
  [24]: 18348 /data/shell/../webapps/scp-informationrelease-app-service-1.4.0.jar
  [25]: 23789 /data/shell/../webapps/scp-realdatapush-component-service-1.4.0.jar
  [26]: 15983 /data/shell/../webapps/scp-businesscommon-app-service-1.4.0.jar
  [27]: 15599 /data/shell/../webapps/scp-event-app-service-1.4.0.jar
  [28]: 24559 /data/shell/../webapps/scp-guidepost-component-service-1.4.0.jar
  [29]: 13681 /data/shell/../webapps/scp-logtransfer-component-service-1.2.0.jar
  [30]: 25842 /data/shell/../webapps/common-djs-component-service-1.4.0.jar
  [31]: 24179 /data/shell/../webapps/scp-robot-component-service-1.4.0.jar
  [32]: 13236 /data/shell/../webapps/scp-logservice-component-service-1.2.0.jar
  [33]: 11768 /data/shell/../webapps/discovery-server-1.4.0.jar
  [34]: 16380 /data/shell/../webapps/scp-laddercontrol-app-service-1.4.0.jar
  [35]: 14397 /data/shell/../webapps/scp-apianalysis-component-service-1.4.0.jar

35
Calculating attach execution time...
Attaching to 14397 using version 3.0.4...

real	0m0.824s
user	0m0.752s
sys	0m0.060s
Attach success.
Connecting to arthas server... current timestamp is 1537185409
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
  ,---.  ,------. ,--------.,--.  ,--.  ,---.   ,---.                           
 /  O  \ |  .--. ''--.  .--'|  '--'  | /  O  \ '   .-'                          
|  .-.  ||  '--'.'   |  |   |  .--.  ||  .-.  |`.  `-.                          
|  | |  ||  |\  \    |  |   |  |  |  ||  | |  |.-'    |                         
`--' `--'`--' '--'   `--'   `--'  `--'`--' `--'`-----'                          
                                                                                

wiki: https://alibaba.github.io/arthas
version: 3.0.4
pid: 16380
timestamp: 1537185409166

$ sm  com.eg.egsc.scp.apianalysis.AnalysisServiceApplication
Affect(row-cnt:0) cost in 7 ms.
$ 

输入命令的时候连接被关闭

root@0584b8941065:/# ./as.sh 1 help
Arthas script version: 3.0.2
Calculating attach execution time...
Attaching to 1 using version 3.0.4...

real    0m0.101s
user    0m0.163s
sys     0m0.044s
Attach success.
Connecting to arthas server... current timestamp is 1536953910
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
  ,---.  ,------. ,--------.,--.  ,--.  ,---.   ,---.
 /  O  \ |  .--. ''--.  .--'|  '--'  | /  O  \ '   .-'
|  .-.  ||  '--'.'   |  |   |  .--.  ||  .-.  |`.  `-.
|  | |  ||  |\  \    |  |   |  |  |  ||  | |  |.-'    |
`--' `--'`--' '--'   `--'   `--'  `--'`--' `--'`-----'


wiki: https://alibaba.github.io/arthas
version: 3.0.4
pid: 1
timestamp: 1536953910961

$ Connection closed by foreign host.

在windows10环境中运行as.bat失败

Start arthas failed, exception stack trace:
com.taobao.middleware.cli.InvalidValueException: The value '' is not accepted by 'pid'
        at com.taobao.middleware.cli.impl.DefaultCommandLine.create(DefaultCommandLine.java:342)
        at com.taobao.middleware.cli.impl.DefaultCommandLine.getValue(DefaultCommandLine.java:267)
        at com.taobao.middleware.cli.impl.DefaultCommandLine.getOptionValue(DefaultCommandLine.java:91)
        at com.taobao.arthas.core.Arthas.parse(Arthas.java:40)
        at com.taobao.arthas.core.Arthas.<init>(Arthas.java:23)
        at com.taobao.arthas.core.Arthas.main(Arthas.java:86)
Caused by: java.lang.IllegalArgumentException: java.lang.NumberFormatException: For input string: ""
        at com.taobao.middleware.cli.converters.ConstructorBasedConverter.fromString(ConstructorBasedConverter.java:70)
        at com.taobao.middleware.cli.converters.Converters.create(Converters.java:52)
        at com.taobao.middleware.cli.impl.DefaultCommandLine.create(DefaultCommandLine.java:339)
        ... 5 more
Caused by: java.lang.NumberFormatException: For input string: ""
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:592)
        at java.lang.Integer.<init>(Integer.java:867)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.taobao.middleware.cli.converters.ConstructorBasedConverter.fromString(ConstructorBasedConverter.java:67)
        ... 7 more

arthas的版本号策略

采用类似 3.0.0 这样子的版本号。为了保证版本比较能正常工作,不得包含字母等。

目前as-package.sh 里是定义了一个 3.0,然后再取当前时间拼接起来,出来的版本号类似:3.0.20171103142340

为了保证后续的新的版本号比较能正常工作,后续发版本时,直接从 3.0.3 开始,本地开发时,拼接出来的版本号是 3.0.3 + date ,即类似 3.0.3.20171103142340

保证本地开发可以用到最新版本。

增加trace和watch的复合参数

目前trace可以看到耗时,watch能看到入参和出参,希望有个命令即能够看到入参出参,同时能看到耗时?这样可以更方便的定位到底是因为什么参数导致比较慢?

使用最新的包,watch命令报错

$ watch Demo$Counter value returnObj
No class or method is affected, try:

  1. sm CLASS_NAME METHOD_NAME to make sure the method you are tracing actually exists (it might be in your parent class).
  2. reset CLASS_NAME and try again, your method body might be too large.
  3. visit middleware-container/arthas/issues/278 for more detail
    $

No class or method is affected/Trace, watch等命令无法找到对应的类和方法

分为两种情况:

一、 先用sc或者sm搜索对应的类和方法,确认有结果,确认已经被JVM加载的

  1. 系统级别的类(即java.*)默认不能进行增强,需要增强是请参考这里的unsafe开关,增强系统类时请谨慎操作

    options unsafe true
    
  2. 检查下 $HOME/logs/arthas/arthas.log,有没有 ERROR c.t.arthas.core.advisor.Enhancer -the classloader can not load SpyAPI, ignore it. 的日志。如果有,则加载目标类的ClassLoader实现代码有问题,没能正确加载java.* package下面的类。可以尝试开启arthas.enhanceLoaders=java.lang.ClassLoader配置项 ,参考: #1596

  3. 构造函数是<init>,例如:watch demo.MathGame <init> '{params,returnObj,throwExp}' -v

  4. 默认情况下,enum类会被过滤掉,参考: #1677

  5. 不支持对 lambda 生成的动态类做增强,JVM自身限制

  6. 数组类、接口类、枚举类以及java.lang.Class/java.lang.Integer/java.lang.reflect.Method等类都是不能进行增强的

  7. 由Arthas自身的ClassLoader加载的类不能被增强

  8. $HOME/logs/arthas/arthas.log中查找有没有Method code too large的异常

  9. 存在该异常时,尝试用reset class_name命令对类进行恢复,再进行trace,watch等操作

二、用sc或者sm搜索对应的类和方法,结果是0时

  1. 查找内部类要用 $ 符号拼出正确的类名,比如sc outer-class$inner-class
  2. 连错了进程,如果之前连过别的进程,且没有shutdown退出,则下次连别的进程时,默认还会连到之前连的进程上。

Error: Could not find or load main class com.taobao.arthas.core.ArthasConsole

按照提示执行了 “curl -sLk http://arthas.io/arthas/install.sh | sh” 依然如此

`
[root@arthas]# as.sh 19711
Arthas script version: 3.0.2
Calculating attach execution time...
Attaching to 19711 using version 0.0...
Start arthas failed, exception stack trace:
java.lang.IllegalStateException: as.sh is too old to support web console, please run the following command to upgrade to latest version:
curl -sLk http://arthas.io/arthas/install.sh | sh
at com.taobao.arthas.core.Arthas.parse(Arthas.java:44)
at com.taobao.arthas.core.Arthas.(Arthas.java:23)
at com.taobao.arthas.core.Arthas.main(Arthas.java:86)
Error: Could not find or load main class com.taobao.arthas.core.ArthasConsole

real 0m0.224s
user 0m0.212s
sys 0m0.059s
Attach success.
Connecting to arthas server... current timestamp is 1537154714
Error: Could not find or load main class com.taobao.arthas.core.ArthasConsole
`

离线内网安装arthas正常,但是启动as.sh 错误

环境:

LSB Version:	:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID:	RedHatEnterpriseServer
Description:	Red Hat Enterprise Linux Server release 6.8 (Santiago)
Release:	6.8
Codename:	Santiago

arthas版本
arthas-packaging-3.0.0-RC-bin

JDK 版本

java version "1.8.0_71"
Java(TM) SE Runtime Environment (build 1.8.0_71-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.71-b15, mixed mode)

错误如下:

Calculating attach execution time...
Attaching to 30267 using version 0.0...
Start arthas failed, exception stack trace: 
java.lang.IllegalStateException: as.sh is too old to support web console, please run the following command to upgrade to latest version:
curl -sLk http://arthas.io/arthas/install.sh | sh
	at com.taobao.arthas.core.Arthas.parse(Arthas.java:44)
	at com.taobao.arthas.core.Arthas.<init>(Arthas.java:23)
	at com.taobao.arthas.core.Arthas.main(Arthas.java:86)
错误: 找不到或无法加载主类 com.taobao.arthas.core.ArthasConsole

java 10下面web console抛出异常

java.lang.NoClassDefFoundError: io/netty/util/internal/TypeParameterMatcher
        at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
        at java.base/java.lang.Class.forName(Class.java:374) ~[na:na]
        at io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:66) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:58) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.util.internal.TypeParameterMatcher.get(TypeParameterMatcher.java:42) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.util.internal.TypeParameterMatcher.find(TypeParameterMatcher.java:78) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.SimpleChannelInboundHandler.<init>(SimpleChannelInboundHandler.java:67) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.SimpleChannelInboundHandler.<init>(SimpleChannelInboundHandler.java:57) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.termd.core.http.netty.HttpRequestHandler.<init>(HttpRequestHandler.java:45) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.termd.core.http.netty.TtyServerInitializer.initChannel(TtyServerInitializer.java:50) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.termd.core.http.netty.TtyServerInitializer.initChannel(TtyServerInitializer.java:34) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.ChannelInitializer.channelRegistered(ChannelInitializer.java:68) ~[arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRegistered(AbstractChannelHandlerContext.java:141) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRegistered(AbstractChannelHandlerContext.java:127) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRegistered(AbstractChannelHandlerContext.java:120) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRegistered(DefaultChannelPipeline.java:1293) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRegistered(AbstractChannelHandlerContext.java:141) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRegistered(AbstractChannelHandlerContext.java:127) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRegistered(DefaultChannelPipeline.java:799) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:498) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:412) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:471) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:339) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:393) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145) [arthas-core.jar:3.0.1-RC-SNAPSHOT]
        at java.base/java.lang.Thread.run(Thread.java:844) [na:na]
Caused by: java.lang.ClassNotFoundException: io.netty.util.internal.TypeParameterMatcher
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) ~[na:na]
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) ~[na:na]
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) ~[na:na]
        ... 27 common frames omitted

arthas zip包的发放地址

有几个地方可以放

  1. maven central repository , 下载方便,发版本比较麻烦
  2. github release ,国内下载容易被墙,发版本容易,先打tag,再上传zip包即可
  3. alibaba.github.io/arthas 下面,即放在 gh-pages 分支下载,目前下载没问题,但是也有被墙的风险。另外,这样子发版本不规范

综合考虑还是把 zip 发到 maven仓库里比较好

去掉com.taobao.arthas.core.Arthas里反射调用 com.sun.tools.attach.VirtualMachine 的代码,改为直接调用

com.taobao.arthas.core.Arthas

增加一个依赖,因为里面带有com.sun.tools.attach.VirtualMachine api,打包 fat jar时,并不会打进去,只是编绎用:

        <dependency>
            <groupId>org.powermock</groupId>
            <artifactId>powermock-module-javaagent</artifactId>
            <scope>provided</scope>
            <optional>true</optional>
        </dependency>

不使用tools.jar,是因为jdk9之后没有这个jar包了。

【Arthas问题排查集】活用ognl表达式

前言

Arthas 3.0中使用ognl表达式替换了groovy来实现表达式的求值功能,解决了groovy潜在会出现内存泄露的问题。灵活运用ognl表达式,能够极大提升问题排查的效率。

ognl官方文档:https://commons.apache.org/proper/commons-ognl/language-guide.html

一个测试应用

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;

/**
 * @author zhuyong on 2017/9/13.
 */
public class Test {

    public static final Map m = new HashMap<>();
    public static final Map n = new HashMap<>();

    static {
        m.put("a", "aaa");
        m.put("b", "bbb");

        n.put(Type.RUN, "aaa");
        n.put(Type.STOP, "bbb");
    }

    public static void main(String[] args) throws InterruptedException {
        List<Pojo> list = new ArrayList<>();

        for (int i = 0; i < 40; i ++) {
            Pojo pojo = new Pojo();
            pojo.setName("name " + i);
            pojo.setAge(i + 2);

            list.add(pojo);
        }

        while (true) {
            int random = new Random().nextInt(40);

            String name = list.get(random).getName();
            list.get(random).setName(null);

            test(list);

            list.get(random).setName(name);

            Thread.sleep(1000l);
        }
    }

    public static void test(List<Pojo> list) {

    }

    public static void invoke(String a) {
        System.out.println(a);
    }

    static class Pojo {
        String name;
        int age;
        String hobby;

        public String getName() {
            return name;
        }

        public void setName(String name) {
            this.name = name;
        }

        public int getAge() {
            return age;
        }

        public void setAge(int age) {
            this.age = age;
        }

        public String getHobby() {
            return hobby;
        }

        public void setHobby(String hobby) {
            this.hobby = hobby;
        }
    }
}

public enum Type {
    RUN, STOP;
}

查看第一个参数

params是参数列表,是一个数组,可以直接通过下标方式访问

$ watch Test test params[0] -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 26 ms.
@ArrayList[
    @Pojo[Test$Pojo@6e2c634b],
    @Pojo[Test$Pojo@37a71e93],
    @Pojo[Test$Pojo@7e6cbb7a],
    ...
]

这里的-n表示只输出一次

查看数组中的元素

第一个参数是一个List,想要看List中第一个Pojo对象,可以通过下标方式,也可以通过List的get方法访问。

$ watch Test test params[0][0] -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 14 ms.
@Pojo[
    name=@String[name 0],
    age=@Integer[2],
    hobby=null,
]

$ watch Test test params[0].get(0) -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 14 ms.
@Pojo[
    name=@String[name 0],
    age=@Integer[2],
    hobby=null,
]

查看Pojo的属性

拿到这个Pojo可以,直接访问Pojo的属性,如age

$ watch Test test params[0].get(0).age -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 21 ms.
@Integer[2]

还可以通过下标的方式访问params[0][0]["age"],这个写法等效于params[0][0].age

$ watch Test test params[0][0]["name"] -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 53 ms.
watch failed, condition is: null, express is: params[0][0][age], ognl.NoSuchPropertyException: com.taobao.arthas.core.advisor.Advice.age, visit /Users/wangtao/logs/arthas/arthas.log for more details.

但这样会报错,这时候需要再加一个引号

$ watch Test test 'params[0][0]["age"]' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 25 ms.
@Integer[2]

集合投影

有时候我们只需要抽取对象数组中的某一个属性,这种情况可以通过投影来实现,比如要将Pojo对象列表中的name属性单独抽出来,可以通过params[0].{name}这个表达式来实现。 ognl会便利params[0]这个List取出每个对象的name属性,重新组装成一个新的数组。用法相当于Java stream中的map函数。

$ watch Test test params[0].{name} -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 56 ms.
@ArrayList[
    @String[name 0],
    @String[name 1],
    @String[name 2],
    @String[name 3],
    null,
    @String[name 5],
    @String[name 6],
    @String[name 7],
    @String[name 8],
    @String[name 9],
]

集合过滤

有时候还需要针对集合对象按某种条件进行过滤,比如想找出所有age大于5的Pojo的name,可以这样写

$ watch Test test "params[0].{? #this.age > 5}.{name}" -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 25 ms.
@ArrayList[
    @String[name 4],
    @String[name 5],
    @String[name 6],
    null,
    @String[name 8],
    @String[name 9],
]

其中{? #this.age > 5} 相当于stream里面的filter,后面的name相当于stream里面的map

那如果要找到第一个age大于5的Pojo的name,怎么办呢?可以用^$来进行第一个或最后一个的匹配,像下面这样:

$ watch Test test "params[0].{^ #this.age > 5}.{name}" -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 24 ms.
@ArrayList[
    @String[name 4],
]
Command hit execution time limit 1, therefore will be aborted.
$ watch Test test "params[0].{$ #this.age > 5}.{name}" -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 43 ms.
@ArrayList[
    @String[name 9],
]

多行表达式

有些表达式一行之内无法表达,需要多行才能表达,应该怎么写的?比如,假设我们要把所有Pojo的name拿出来,再往里面新加一个新的元素,在返回新的列表,应该如何写?可以通过中括号将多个表达式串联起来,最后一个表达式的返回值代表整个表达式的最终结果。临时变量可以用#来表示。

$ watch Test test '(#test=params[0].{name}, #test.add("abc"), #test)' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 28 ms.
@ArrayList[
    @String[name 0],
    @String[name 1],
    @String[name 2],
    @String[name 3],
    @String[name 4],
    @String[name 5],
    @String[name 6],
    @String[name 7],
    @String[name 8],
    null,
    @String[abc],
]

调用构造函数

调用构造函数,必须要指定要创建的类的全类名。比如下面的例子中,创建一个新的list,然后添加一个新的元素,然后返回添加后的list。

$ watch Test test '(#test=new java.util.ArrayList(), #test.add("abc"), #test)' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 37 ms.
@ArrayList[
    @String[abc],
]

访问静态变量

可以通过@class@filed方式访问,注意需要填写全类名

$ watch Test test '@Test@m' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 35 ms.
@HashMap[
    @String[a]:@String[aaa],
    @String[b]:@String[bbb],
]

调用静态方法

可以通过@class@method(args)方式访问,注意需要填写全类名

$ watch Test test '@java.lang.System@getProperty("java.version")' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 42 ms.
@String[1.8.0_51]

静态方法和非静态方法结合,例如想要获取当前方法调用的TCCL,可以像下面这样写:

$ watch Test test '@java.lang.Thread@currentThread().getContextClassLoader()' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 84 ms.
@AppClassLoader[
    ucp=@URLClassPath[sun.misc.URLClassPath@4cdbe50f],
    $assertionsDisabled=@Boolean[true],
]

访问Map中的元素

Test.n是一个HashMap,假设要获取这个Map的所有key,ongl针对Map接口提供了keys, values这两个虚拟属性,可以像普通属性一样访问。

$ watch Test test '@[email protected]' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 57 ms.
@KeySet[
    @Type[RUN],
    @Type[STOP],
]

因为这个Map的Key是一个Enum,假设要把key为RUN这个值的value取出来应该怎么写呢?可以通过Enum的valueOf方法来创建一个Enum,然后get出来,比如下面一样

$ watch Test test '@[email protected](@Type@valueOf("RUN"))' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 168 ms.
@String[aaa]

或者是下面这样,通过迭代器+过滤的方式:

$ watch Test test '@[email protected]().iterator.{? #this.key.name() == "RUN"}' -n 1
Press Ctrl+C to abort.
Affect(class-cnt:1 , method-cnt:1) cost in 72 ms.
@ArrayList[
    @Node[RUN=aaa],
]

附录: ognl内置的ognl的虚拟属性

  • Collection:
    • size
    • isEmpty
  • List:
    • iterator
  • Map:
    • keys
    • values
  • Set:
    • iterator
  • Iterator:
    • next
    • hasNext
  • Enumeration:
    • next
    • hasNext
    • nextElement
    • hasMoreElements

最后

欢迎在留言区分享你的牛逼用法,互相交流进步~

linux启动问题

Error: Unable to access jarfile /home/xxxx/.arthas/lib/3.0.4/arthas/arthas-core.jar
attach to target jvm (16085) failed, check /home/xxxx/logs/arthas/arthas.log or stderr of target jvm for any exceptions.

Demo进程是第4个,则输入4,再输入回车/enter。Arthas会attach到目标进程上,并输出日志;

操作这个步骤时出现的该错误?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.