hammer / avro-rpc-quickstart

Apache Avro RPC Quick Start. Avro is a subproject of Apache Hadoop.

Avro RPC Quick Start

Author: Patrick Hunt (follow me on twitter)
Just Poking Around: Jeff Hammerbacher (follow me on facebook)

Summary

This is a starter kit, a project template if you will, for Apache Avro intended to bootstrap your Avro based project. You’ll learn how to declare a protocol, generate and compile your code, and run a working “Hello World” type example. This project supports building with either Maven or Ant/Ivy, choose one.

What’s Apache Avro?

From the official site: “Avro is a data serialization system”

Avro provides:

• Rich data structures.
• A compact, fast, binary data format.
• A container file, to store persistent data.
• Remote procedure call (RPC).
• Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.

Requirements

Java Requirements

If you decide to use Maven you will need the Avro Maven Plugin to compile the protocol specification. I’ve included the Avro Maven plugin jar file in this repository for simplicity and to get you started quickly. However you can examine the source here: avro-maven-plugin

Run the following if you are using maven:

mvn install:install-file -Dfile=./avro-maven-plugin-1.0-SNAPSHOT.jar -DpomFile=./avro-maven-plugin-1.0-SNAPSHOT.pom

Ant based build has no additional requirements (just ant).

Python Requirements

The toplevel “lib/python” directory contains the Avro python library as well as a module “odict” used by Avro.

Additionally Avro requires simplejson be available. Easiest way to do that is through python’s “easy_install”:

sudo easy_install simplejson

Introduction

The sample application included in this project simulates a remote service, Mail, where Avro RPC is used to send a message using the service. This document details how to build and run the sample using either Ant or Maven. In either case the Avro jar files (and jars they depend upon) will be downloaded automatically.

In this sample project you will find four sets of files:

1. This documentation
2. Sample Avro protocol declaration
3. Java quick start
   • Ant/Ivy build files
   • Maven build file
   • Sample application – i.e. Main program
4. Python quick start
   • Sample application – i.e. send_message program

Overview of the files

First I’ll go through the Avro protocol declaration, then the build files, Ant/Ivy or Maven, the main java source for the sample Mail application, and finally an implementation of the mail service/client in python. Notice that java and python share the same Avro protocol declaration. I’ve not demonstrated it here, but the python & java implmentations are interoperable – the java client can talk to the python server and vice-versa. (which I’ll leave as an exercise for the reader)

mail.avpr – Avro Protocol Declaration

src/main/avro should contain all of the Avro protocol & schema specifications. mail.avpr declares our simple “Mail” service. You will see:

1. the name & namespace of the protocol
2. any specialized types used in the messages, Message in this case
3. we are declaring a “send” message type which takes a Message as an argument and returns a result string

Read more about Avro’s protocol declaration

Java Ant/Ivy build files

• build.xml – the Ant build file
• ivy.xml – this file specifies the dependencies of the project, primarily the Avro jar file

Java Maven build file

• pom.xml – this file specifies the dependencies of the project, primarily the Avro jar file

The profile section adds the ibiblio repository, which holds the Avro dependencies.

    <profile>
      <id>default</id>
      <repositories>
        <repository>
          <id>ibiblio</id>
          <name>www.ibiblio.org</name>
          <url>http://people.apache.org/repo/m2-ibiblio-rsync-repository</url>
        </repository> 
      </repositories> 
    </profile>

You’ll also see the plugin section, which contains:

This plugin element causes the Avro Maven Plugin’s protocol goal defined in avro-maven-plugin to run during the “generate-sources” maven phase.

      <plugin>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <version>1.0-SNAPSHOT</version>
        <executions>
          <execution>
            <phase>generate-sources</phase>
            <goals>
              <goal>protocol</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

The Paranamer plugin is necessary for Avro to access method parameter metadata (i.e. names) during runtime.

      <plugin>
        <groupId>com.thoughtworks.paranamer</groupId>
        <artifactId>paranamer-maven-plugin</artifactId>
        <executions>
          <execution>
            <id>run</id>  <!-- id is optional -->
            <configuration>
              <sourceDirectory>${project.build.directory}/generated-sources/avro</sourceDirectory>
              <outputDirectory>${project.build.directory}/classes</outputDirectory>
            </configuration>        
            <goals>
              <goal>generate</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

This tells Maven to target 1.5 JVM:

      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
          <source>1.5</source>
          <target>1.5</target>
        </configuration>
      </plugin>

Main.java – the main() routine of the Java Mail sample

src/main/java/example/Main.java

1. the MailImpl class implements the Mail protocol defined in mail.avpr
2. the startServer() method starts the server which implements the Mail service (Mail/MailImpl)
3. the main function takes three arguments; to, from and body of the message. After the server is started a Mail client is created, attached to the service, and used to send a “Message”, the result of the RPC call is printed to the console.

Python – send_message.py

You’ll see that the structure of the python code is similar to the java source.

src/main/python/send_message.py

1. the MailResponder class implements the Mail protocol defined in mail.avpr
2. the start_server() method starts the server which implements the Mail service (Mail/MailResponder)
3. the main function takes three arguments; to, from and body of the message. After the server is started a Mail client is created, attached to the service, and used to send a “Message”, the result of the RPC call is printed to the console.

Note: version 1.1.0 of Avro is generating an error after closing the server. There is a JIRA pending for this.

Run the python

From the src/main/python directory run:

PYTHONPATH=../../../lib/python ./send_message.py avro_user pat Hello_World

Compiling the Java sample

With either build system, maven or ant, all generated files (source, class, etc…) are written to the “target” directory.

Maven:
mvn compile

Ant/Ivy:
ant compile

Note: integration with eclipse is very simple. If you are using Ant, create a project and add the Main.java and jars contained in the target/lib directory (created/downloaded by ivy as part of compiling the project using build.xml). If using Maven just type “mvn eclipse:eclipse” (see the maven-eclipse-plugin documentation for more details).

Running the Java sample

Maven:
mvn -e exec:java -Dexec.mainClass=example.Main -Dexec.args='avro_user pat Hello_World'

Ant/Ivy:
ant exec:java