log4j and Scribe

Well I thought it should be fairly easy. But it turned out it's not that straightforward especially for a Java newbie like me. Basically we need to make a log4j appender for Scribe. Fortunately it's available from http://github.com/lenn0x/Scribe-log4j-Appender. The package contains scribelog4j.jar already. Just in case you want to build it yourself or change the referencing libraries, here are the instructions. The following were tested on CentOS 5.3 64-bit.

1. Download and extract log4j from  http://logging.apache.org/log4j/1.2/download.html

2. Download and extract Scribe 2.2 and Thrift from http://github.com/facebook/scribe/downloads and http://incubator.apache.org/thrift/download (the current version is 0.5.0, I am using 0.2.0 in this example.)

Build Thrift and install. Please refer to Step 14 in Installing & Running Scribe with HDFS support on CentOS article.

3. Download and extract slf4j (Simple Logging Facade for Java) from http://www.slf4j.org/download.html

4. Install latest ant if you haven't. See the previous post: Installing Latest Ant on CentOS. Thrift jar cannot be built using ant 1.6.5 from original CentOS repository.



5. Build libthrift.jar. From top Thrift directory:

[user@localhost:~/pkgs/thrift-0.2.0] cd lib/java
[user@localhost:~/pkgs/thrift-0.2.0/lib/java] ant

You should see libthrift.jar in the current directory.

6. Build libfb303.jar. From top Thrift directory:

[user@localhost:~/pkgs/thrift-0.2.0] cd contrib/fb303/java

Edit build.xml file.

Find "<classpath>" and change the value of "pathelement location" to where your libthrift.jar is.

My thrift is in $HOME/pkgs/thrift-0.2.0 directory. So I add a new property name on the top.
<property name="pkgs.dir" value="${user.home}/pkgs"/>


Also slf4j's api jar has to be added to classpath as well.

The pathelement location lines should look like the following:
<pathelement location="${pkgs.dir}/thrift-0.2.0/lib/java/libthrift.jar"/>
<pathelement location="${pkgs.dir}/slf4j-1.6.1/slf4j-api-1.6.1.jar"/>


Remove FacebookBase.java file. This has to be done or you will get "getStatus() in com.facebook.fb303.FacebookBase cannot implement getStatus() in com.facebook.fb303.FacebookService.Iface; attempting to use incompatible return type" error.

[user@localhost:~/pkgs/thrift-0.2.0/contrib/fb303/java] rm FacebookBase.java

Run ant.
[user@localhost:~/pkgs/thrift-0.2.0/contrib/fb303/java] ant

You should find libfb303.jar in build/lib directory

[user@localhost:~/pkgs/thrift-0.2.0/contrib/fb303/java] ll build/lib/
total 176
drwxr-xr-x 2 user user 4096 Sep 16 12:09 .
drwxr-xr-x 4 user user 4096 Sep 16 11:45 ..
-rw-r--r-- 1 user user 165162 Sep 16 12:09 libfb303.jar


7. Build scribe.jar. From top level scribe directory:

[user@localhost::~/pkgs/scribe] cd if

Edit scribe.thrift
Add the following line in the file.
namespace java scribe

Copy fb303 thrift directory here.
[user@localhost:~/pkgs/scribe/if] cp -r ~/pkgs/thrift-0.2.0/contrib/fb303/ .

And copy fb303's build.xml here as well
[user@localhost:~/pkgs/scribe/if] cp ~/pkgs/thrift-0.2.0/contrib/fb303/java/build.xml .

Edit build.xml to build scribe.jar. The file will look like the following. magenta text indicates the modifications for building scribe.jar.

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<project name="scribe" default="dist" basedir="..">
<!-- project wide settings. All directories relative to basedir -->
<property name="src.dir" value="if"/>
<property name="if.dir" value="if"/>
<property name="thrift_home" value="/usr/local"/>
<property name="pkgs.dir" value="${user.home}/pkgs"/>
<!-- temp build directories -->
<property name="build.dir" value="${src.dir}/build"/>
<property name="build.classes.dir" value="${build.dir}/classes"/>
<property name="build.lib.dir" value="${build.dir}/lib"/>

<!-- final distribution directories -->
<property name="dist.dir" value="/usr/local"/>
<property name="dist.lib.dir" value="${dist.dir}/lib"/>

<!-- make temporary and distribution directories -->
<target name="prepare">
<mkdir dir="${build.dir}"/>
<mkdir dir="${build.classes.dir}"/>
<mkdir dir="${build.lib.dir}"/>
<mkdir dir="${dist.dir}"/>
<mkdir dir="${dist.lib.dir}"/>
</target>

<!-- generate scribe thrift code -->
<target name="scribebuild">
<echo>generating thrift scribe files</echo>
<exec executable="${thrift_home}/bin/thrift" failonerror="true">
<arg line="--gen java -o ${src.dir} ${src.dir}/../if/scribe.thrift" />
</exec>
<move todir="${src.dir}">
<fileset dir="${src.dir}/gen-java/scribe">
<include name="**/*.java"/>
</fileset>
</move>
</target>

<!-- compile the base and thrift generated code and jar them -->
<target name="dist" depends="prepare,scribebuild">
<echo>Building scribe.jar .... </echo>
<javac srcdir="${src.dir}" destdir="${build.classes.dir}" debug="on">
<classpath>
<pathelement location="${pkgs.dir}/thrift-0.4.0/lib/java/libthrift.jar"/>
<pathelement location="${pkgs.dir}/slf4j-1.6.1/slf4j-api-1.6.1.jar"/>
<pathelement location="${pkgs.dir}/thrift-0.4.0/contrib/fb303/java/build/lib/libfb303.jar"/>

</classpath>
<include name="*.java"/>
<include name="${build.dir}/scribe"/>
</javac>
<jar jarfile="${build.lib.dir}/scribe.jar" basedir="${build.classes.dir}">
</jar>
</target>

<!-- copy the build jar to the distribution library directory -->
<target name="install" depends="dist">
<copy todir="${dist.lib.dir}">
<fileset dir="${build.lib.dir}" includes="scribe.jar"/>
</copy>
</target>

<target name="clean">
<echo>Cleaning old stuff .... </echo>
<delete dir="${build.dir}/classes/com"/>
<delete dir="${build.dir}/lib"/>
<delete dir="${build.dir}"/>
</target>
</project>

Run ant.
[user@localhost:~/pkgs/scribe/if] ant

You will see scribe.jar in build/lib directory.
[user@localhost:~/pkgs/scribe/if] ll build/lib/
total 36
drwxr-xr-x 2 user user 4096 Sep 16 12:55 .
drwxr-xr-x 4 user user 4096 Sep 16 12:55 ..
-rw-r--r-- 1 user user 27002 Sep 16 12:58 scribe.jar


8. Build scribelog4j.jar

First Get Scribe-log4j-Appender using git.

If git is not installed.
yum -y install git

Get the code.
git clone http://github.com/lenn0x/Scribe-log4j-Appender.git

The code will be in Scribe-log4j-Appender directory. Change to the directory.

[user@localhost:~/pkgs/Scribe-log4j-Appender] cd src/java/org/apache/log4j

Copy scribe's build.xml here.
[user@localhost:~/pkgs/Scribe-log4j-Appender/src/java/org/apache/log4j] cp ~/pkgs/scribe/if/build.xml .

Edit build.xml and it should look like the following. Adjust the magenta part if needed.

<project name="scribelog4j" default="dist" basedir="..">

<!-- project wide settings. All directories relative to basedir -->
<property name="src.dir" value="log4j/scribe"/>
<property name="if.dir" value="if"/>
<property name="thrift_home" value="/usr/local"/>
<property name="pkgs.dir" value="${user.home}/pkgs"/>

<!-- temp build directories -->
<property name="build.dir" value="${src.dir}/build"/>
<property name="build.classes.dir" value="${build.dir}/classes"/>
<property name="build.lib.dir" value="${build.dir}/lib"/>

<!-- final distribution directories -->
<property name="dist.dir" value="/usr/local"/>
<property name="dist.lib.dir" value="${dist.dir}/lib"/>

<!-- make temporary and distribution directories -->
<target name="prepare">
<mkdir dir="${build.dir}"/>
<mkdir dir="${build.classes.dir}"/>
<mkdir dir="${build.lib.dir}"/>
<mkdir dir="${dist.dir}"/>
<mkdir dir="${dist.lib.dir}"/>
</target>

<!-- compile the base and thrift generated code and jar them -->
<target name="dist" depends="prepare">
<echo>Building scribelog4j.jar .... </echo>
<javac srcdir="${src.dir}" destdir="${build.classes.dir}" debug="on">
<classpath>
<pathelement location="${pkgs.dir}/thrift-0.2.0/lib/java/libthrift.jar"/>
<pathelement location="${pkgs.dir}/slf4j-1.6.1/slf4j-api-1.6.1.jar"/>
<pathelement location="${pkgs.dir}/thrift-0.2.0/contrib/fb303/java/build/lib/libfb303.jar"/>
<pathelement location="${pkgs.dir}/scribe/if/build/lib/scribe.jar"/>
</classpath>
<include name="*.java"/>
<include name="${build.dir}/scribe"/>
</javac>
<jar jarfile="${build.lib.dir}/scribelog4j.jar" basedir="${build.classes.dir}">
</jar>
</target>

<!-- copy the build jar to the distribution library directory -->
<target name="install" depends="dist">
<copy todir="${dist.lib.dir}">
<fileset dir="${build.lib.dir}" includes="scribelog4j.jar"/>
</copy>
</target>

<target name="clean">
<echo>Cleaning old stuff .... </echo>
<delete dir="${build.dir}/classes/com"/>
<delete dir="${build.dir}/lib"/>
<delete dir="${build.dir}"/>
</target>
</project>


Run ant.
[user@localhost:~/pkgs/Scribe-log4j-Appender/src/java/org/apache/log4j] ant

You should get scribelog4j.jar in the current directory.

9. Write and run a test program

Here is the test program called scribelog4j_test.java

public class scribelog4j_test {
private static Logger logger = Logger.getLogger(scribelog4j_test.class);
public static void main(String[] args) {
long time = System.currentTimeMillis();
logger.info("main method called..");
logger.info("another informative message");
logger.warn("This one is a warning!");
logger.log(Level.TRACE,
"And a trace message using log() method.");
long logTime = System.currentTimeMillis() - time;

logger.debug("Time taken to log the previous messages: "
+ logTime + " msecs");

// Exception logging example:
try{
String subs = "hello".substring(6);
}catch (Exception e){
logger.error("Error in main() method:", e);
}
}
}


Create a file called log4j.properties in the current directory. It should look like this:

log4j.rootLogger=debug, stdout, R, scribe

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout

# Pattern to output the caller's file name and line number.
log4j.appender.stdout.layout.ConversionPattern=%5p [%t] (%F:%L) - %m%n

log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=example.log

log4j.appender.R.MaxFileSize=100KB
# Keep one backup file
log4j.appender.R.MaxBackupIndex=1

log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n

#
# Add this to your log4j.properties
#
log4j.appender.scribe=org.apache.log4j.scribe.ScribeAppender
log4j.appender.scribe.scribe_category=MyScribeCategoryName
log4j.appender.scribe.scribe_host=127.0.0.1
log4j.appender.scribe.layout=org.apache.log4j.PatternLayout
log4j.appender.scribe.layout.ConversionPattern=%5p [%t] %d{ISO8601} %F (line %L) %m%n


For convenience, copy all the necessary jars to a directory called lib.

[user@localhost:~/proj/scribe_log4j] ls lib
libfb303.jar log4j-1.2.16.jar scribelog4j.jar slf4j-log4j12-1.6.1.jar
libthrift.jar scribe.jar slf4j-api-1.6.1.jar

To build it:
/usr/bin/javac -classpath .:lib/scribelog4j.jar:lib/log4j-1.2.16.jar ./scribelog4j_test.java

To run it:

Start the Scribe server in a new terminal.
[user@localhost:~/scribe/src] scribed ../examples/example1.conf

In the original terminal, run scribelog4j_test program.
[user@localhost:~/proj/scribe_log4j] /usr/bin/java -classpath .:lib/log4j-1.2.16.jar:lib/scribelog4j.jar::lib/libthrift.jar:lib/slf4j-api-1.6.1.jar:lib/slf4j-log4j12-1.6.1.jar:lib/scribe.jar:lib/libfb303.jar scribelog4j_test

You should find the log file in /tmp/scribetest directory.