BridgeLi's Blog

前几天响应公司的要求，系统日志接入公司的 ELK，按照中间件的同学要求之后，果然不出意外的遇到了问题，项目跑不起来了，控制台 catalina.out 打印日志如下：

Aug 16, 2018 10:02:21 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-apr-8144"]
Aug 16, 2018 10:02:21 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-apr-58144"]
Aug 16, 2018 10:02:21 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 28819 ms
Aug 16, 2018 10:02:23 AM org.apache.catalina.loader.WebappClassLoaderBase loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load com.alibaba.rocketmq.shade.io.netty.util.concurren
t.DefaultPromise$2.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate t
he thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1743)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:590)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:398)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:151)
        at java.lang.Thread.run(Thread.java:745)

Exception in thread "NettyClientWorkerThread_3" java.lang.NoClassDefFoundError: com/alibaba/rocketmq/shade/io/netty/util/concurrent/DefaultPromi
se$2
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:590)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:398)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:151)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise$2
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1858)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
        ... 4 more
Aug 16, 2018 10:02:23 AM org.apache.catalina.loader.WebappClassLoaderBase loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise$2.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1743)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:590)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:398)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:151)
        at java.lang.Thread.run(Thread.java:745)

Exception in thread "NettyClientWorkerThread_4" java.lang.NoClassDefFoundError: com/alibaba/rocketmq/shade/io/netty/util/concurrent/DefaultPromise$2
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:590)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:398)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:151)
        at java.lang.Thread.run(Thread.java:745)
Exception in thread "NettyClientWorkerThread_2" Exception in thread "NettyClientWorkerThread_1" java.lang.NoClassDefFoundError: com/alibaba/rocketmq/shade/io/netty/util/concurrent/DefaultPromise$2
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:590)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:398)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:151)
        at java.lang.Thread.run(Thread.java:745)
java.lang.NoClassDefFoundError: com/alibaba/rocketmq/shade/io/netty/util/concurrent/DefaultPromise$2
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:590)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:398)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:151)
        at java.lang.Thread.run(Thread.java:745)
Exception in thread "NettyClientSelector_1" java.lang.NoClassDefFoundError: com/alibaba/rocketmq/shade/io/netty/util/concurrent/DefaultPromise$2
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:590)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:398)
        at com.alibaba.rocketmq.shade.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:151)
        at java.lang.Thread.run(Thread.java:745)
Aug 16, 2018 10:02:28 AM org.apache.catalina.loader.WebappClassLoaderBase loadClass

很常见的一个错，java.lang.NoClassDefFoundError，找不到 com/alibaba/rocketmq/shade/io/netty/util/concurrent/DefaultPromise，但是不应该啊，因为什么都没有修改，之前都是跑的好好地，怎么会突然找不到这个类，莫名其妙，有同事说，是不是修改 jar 的版本之类的，不过我确实没有修改，只是引入了公司 ELK 相关的 jar 而已，而且看了一下这个类在 classpath 下是存在的，后来经同事提醒，这个地方可能不是真正的报错的地方，看一下 tomcat 的 localhost 日志，果然在发现了如下的报错：

java.lang.NoSuchMethodError: com.google.common.collect.ImmutableList.of(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Lcom/google/common/collect/ImmutableList;
        at com.weidai.middleware.log4j.flume.agent.EmbeddedAgentConfiguration.<clinit>(EmbeddedAgentConfiguration.java:159)
        at com.weidai.middleware.log4j.flume.agent.EmbeddedAgent.doConfigure(EmbeddedAgent.java:145)
        at com.weidai.middleware.log4j.flume.agent.EmbeddedAgent.configure(EmbeddedAgent.java:92)
        at com.weidai.middleware.log4j.flume.appender.FlumeEmbeddedManager$FlumeManagerFactory.createManager(FlumeEmbeddedManager.java:183)
        at com.weidai.middleware.log4j.flume.appender.FlumeEmbeddedManager$FlumeManagerFactory.createManager(FlumeEmbeddedManager.java:165)
        at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:112)
        at com.weidai.middleware.log4j.flume.appender.FlumeEmbeddedManager.getManager(FlumeEmbeddedManager.java:115)
        at com.weidai.middleware.log4j.flume.appender.FlumeDiamondManager.generateFlumeEmbeddedManager(FlumeDiamondManager.java:116)
        at com.weidai.middleware.log4j.flume.appender.FlumeDiamondManager.<init>(FlumeDiamondManager.java:80)
        at com.weidai.middleware.log4j.flume.appender.FlumeDiamondManager.getManager(FlumeDiamondManager.java:157)
        at com.weidai.middleware.log4j.flume.appender.FlumeAppender.createAppender(FlumeAppender.java:226)
        at com.weidai.middleware.log.LogConfigManager.addAsyncFlume(LogConfigManager.java:454)
        at com.weidai.middleware.log.LogConfigManager.doLogConfigure(LogConfigManager.java:180)
        at com.weidai.middleware.log.LogConfigManager.init(LogConfigManager.java:111)
        at com.weidai.middleware.log.LogContextListener.contextInitialized(LogContextListener.java:30)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5066)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1092)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1984)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at com.alibaba.ttl.TtlRunnable.run(TtlRunnable.java:47)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

也是一个很常见的错误：java.lang.NoSuchMethodError: com.google.common.collect.ImmutableList.of(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Lcom/google/common/collect/ImmutableList，公司封装的 jar，在 EmbeddedAgentConfiguration.java 中使用了 ImmutableList.of() 方法没有定义。然后看了提供这个 jar 的同事使用的 jar 是：

    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>18.0</version>
    </dependency>

而项目中，之前同事的使用的一个 jar，是：

    <dependency>
        <groupId>com.google.code.google-collections</groupId>
        <artifactId>google-collect</artifactId>
        <version>snapshot-20080530</version>
    </dependency>

然后替换之，果然解决，看了一下这个 jar 还以为是 Google 2018 年开源的一个工具类集合，感觉不应该啊，这个工具类是同事去年就引入了，我还有印象，当时还奇怪为什么是 snapshot 版，不过没有在意，这次仔细一下竟然 10 年前开源的 jar，也是厉害了。

其实这个问题不重要，重要的是学到了，今后在排查问题的时候不仅仅只看 catalina.out，有些问题很有可能隐藏在其他相关的 tomcat 的相关日志中，而且自己使用 tomcat 几年了，但是一直没有看过 tomcat 相关的学习资料，一般可能也就修改一下 server.xml，其实对 tomcat 并没有一个大体的了解，甚至不知道 tomcat 的日志都有哪些，都是干嘛的，今后要注意学习了，如果想在使用的过程中快速解决问题，什么东西都不能仅仅停留在一使用层面。