The ecoadapters from dlyubimov

ecoadapters's People

Contributors

Stargazers

Watchers

ecoadapters's Issues

Degraded performance of ProtoSerDe when dealing with raw byte

Thanks a lot for making ProtoSerD, it rocks !

Some of my hive jobs (reading protos via ProtoSerDe) run much slower than expected, as they had raw bytes field in it. I tracked down performance issues and have fix, below. Is there way to incorporate it back to the master branch. I confirmed this resulted in 10x performance boost in case where raw byte fields are ~1kb. I tested compatibility of output of old string output and new, and they are same.

PS: Below is the fix diff

commit 3fcaca69b8fa897f57b9f1bf7277456d2d422a40
Author: Nemanja Spasojevic <[email protected]>
Date:   Tue Oct 30 13:08:28 2012 -0700

    Fix degraded performance of ProtoSerDe when dealing with BYTES fields.

diff --git a/ecoadapters/src/main/java/com/inadco/ecoadapters/hive/ProtoSerDe.java b/ecoadapters/src/main/java/com/inadco/ecoadapters/hive/ProtoSerDe.java
index 918e630..69740bc 100644
--- a/ecoadapters/src/main/java/com/inadco/ecoadapters/hive/ProtoSerDe.java
+++ b/ecoadapters/src/main/java/com/inadco/ecoadapters/hive/ProtoSerDe.java
@@ -230,8 +230,8 @@ public class ProtoSerDe implements SerDe {
             return protoMsg2Hive(msgField, fd.getMessageType());
         case BYTES:
             byte[] val = ((ByteString) src).toByteArray();
-            // we convert to string using %X formatting 
-            return String.format("%X", new BigInteger(1, val));
+            // we convert to string using %X formatting
+            return getHexString(val);
         case BOOL:
         case DOUBLE:

@@ -275,4 +275,12 @@ public class ProtoSerDe implements SerDe {

     }

+  private static String getHexString(byte[] b) {
+    StringBuilder result = new StringBuilder();
+    for (int i = 0; i < b.length; i++) {
+      result.append(Integer.toString((b[i] & 0xff) + 0x100, 16).substring(1));
+    }
+    return result.toString().toUpperCase();
+  }
+
 }

Recommend Projects