Controlling logging functionality in hadoop
How to control logging functionality in hadoop? Hadoop uses default log4j.properties file for controlling logs. My use case is to control logs generated by my classes.
Hadoop daemons like JobTracker, TaskTracker, NameNode and DataNode daemon processes use log4j.properties file from their respective host node’s hadoop-conf-directory. The rootLogger is set to “INFO,console” which logs all message at level INFO to the console.
I trigger hadoop jobs using Oozie Workflow. I tried passing my custom log4j.properties file to the job by setting -Dlog4j.configuration=path/to/log4j.properties system property, but it is not working. Still, it takes log4j properties from the default one.
I am not supposed to touch default log4j.properties file.
I am using Oozie-v3.1.3-incubating, hadoop-v0.20 and cloudera CDH-v4.0.1.
How can I override the default log4j.properties file ?? or How can I control logs for my classes ??
1 Reply
rdaniels
Linode Staff
@soujanyabargavi Accomplishing this goal is a bit involved, and may require additional troubleshooting to achieve. That being said, there seem to be a couple of ways you can set custom Log levels in Hadoop, two of which are listed below.
- Custom levels in code
- Custom levels in configuration
Since Hadoop uses the Apache log4j via the Apache Commons Logging framework, you can find additional information on how to achieve custom log levels by reviewing the information in the below section in the Apache Logging manual.
I've also located some additional Hadoop focused resources that may help you troubleshoot your configuration.
Additional Resources
- https://stackoverflow.com/questions/22918720/custom-log4j-appender-in-hadoop-2
- https://www.edureka.co/community/30105/how-to-control-logging-functionality-in-hadoop
Hope this helps!