NoClassDefFoundError for Log4jLoggerFactory on hdp 2.5.3 when running the KafkaSpout in your topology? (how's that for a title?)

Created by Lester Martin
Mar 10, 2017
2 min read

Ephemeral Issue

NOTE: This is a corner-case blog post and really only useful for those who find this entry from a very specific Google search!!  Additionally, I'm filing a support case and expect the problem to be resolved long before it becomes a big issue for many.  But... there are those of us on the latest-greatest version and get to find this stuff out.  ;-)

The Problem (in a nutshell)

Storm in HDP 2.5.3 brings in version 1.6.6 of org.slf4j:log4j-over-slf4j instead of the required version 1.7.21.

The Backdrop

I'm running a Storm topology that uses storm-kafka and I run into the following error when running on HDP 2.5.3.

8487 [Thread-22-kafka-spout-executor[5 5]] ERROR o.a.s.util - Async loop died!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory
    at org.apache.log4j.Logger.getLogger(Logger.java:39) ~[log4j-over-slf4j-1.6.6.jar:1.6.6]
    at kafka.utils.Logging$class.logger(Logging.scala:24) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:38) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at kafka.consumer.SimpleConsumer.logger(SimpleConsumer.scala:38) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at kafka.utils.Logging$class.info(Logging.scala:75) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at kafka.consumer.SimpleConsumer.info(SimpleConsumer.scala:38) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:110) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]  
    at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:99) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:165) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:86) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:81) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:71) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at org.apache.storm.kafka.PartitionManager.<init>(PartitionManager.java:103) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]  
    at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:130) ~[cb1921bc05d611e7b83302d162ad3913.jar:?]
    at org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651) ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
    at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

A fellow Hortonworker, Ambud Sharma, helped me diagnosis this (ok... HE diagnosed this and I watched the master walk the maven dependency graph!) and it seems that ultimately I need version 1.7.21 (not the 1.6.6 ver identified in the stack trace above) of org.slf4j:log4j-over-slf4j which I tried to include in my project by adding the following to my pom.xml dependencies.

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>log4j-over-slf4j</artifactId>
            <version>1.7.21</version>
            <scope>compile</scope>
        </dependency>

Unfortunately, we determined that the storm-core module for HDP 2.5.3 includes 1.6.6 of this jar as seen in the (modified) ls output below.

[root@ip-172-xxx-xxx-95 ~]# ls /usr/hdp/current/storm-supervisor/lib/
log4j-over-slf4j-1.6.6.jar     ... deleted everything else for brevity ...

I replaced this jar with log4j-over-slf4j-1.7.21.jar and my topology runs fine.