ruedigermoeller / fast-serialization Goto Github PK

I know you are receiving a lot of request for Android support.
I just want to add this report so you know where to look if you ever decide to go for it.

java.lang.NoClassDefFoundError: Failed resolution of: Ljava/awt/Color;
    at org.nustaq.serialization.FSTConfiguration.addDefaultClazzes(FSTConfiguration.java:466)
    at org.nustaq.serialization.FSTConfiguration.createDefaultConfiguration(FSTConfiguration.java:184)
    at org.nustaq.serialization.simpleapi.DefaultCoder.<init>(DefaultCoder.java:32)

I don't understand however why you are using AWT in a serialization library, could you enlighten me ?

Hi,

I work with Guillaume Smet who reported #36 . The proposed fix works fine for test case we submitted, but after further investigation, we notice that behavior is incorrect when a same Externalizable / readResolve instance is serialized several times (in different fields, or twice in a collection for example).

In this case, only the first deserialized instance is correct (both external and readResolve contract are handled). For subsequent instances, readResolve is not called and raw instance is used.

As multiple instances are serialized only once, then are serialized as reference, I suppose this bug is caused by an incorrect reference caching during deserialization (raw instance is cached during first deserialization, then this raw instance is reused without calling readResolve), but I do not dig further.

You can find here a simple test case : http://share.openwide.fr/share/zone/lalmeras/2196471291
There is 3 test cases which serialize twice the same instance :

• as two fields of a same bean
• as two elements of a list
• as a field and an element of a set

They all fails (fst v2.12).

Thanks

Using version 1.63.

I am in an scenario where I serialize using IBM JDK 1.6, and I deserialize using Oracle JDK 1.6. BigDeciaml in those implementations differs, not having the same attributes, and both overriding readObject and writeObject. IBM's implementation makes sure the necessary fields for Oracle's implementation are written in the serialized data, having the following body:

private synchronized void writeObject(ObjectOutputStream out)
    throws IOException
{
    ObjectOutputStream.PutField fields = out.putFields();

    fields.put("scale", scale());
    fields.put("intVal", unscaledValue());
    out.writeFields();
}

When deserializing, I got the following exception:

Exception in thread "main" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at de.ruedigermoeller.serialization.util.FSTUtil.rethrow(FSTUtil.java:119)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:474)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatible(FSTObjectInput.java:448)
    at de.ruedigermoeller.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:399)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:262)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:230)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:210)
    at com.agabar.laboratory.serialization.Deserializer.deserialization(Deserializer.java:40)
    at com.agabar.laboratory.serialization.Deserializer.main(Deserializer.java:27)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:471)
    ... 7 more
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at de.ruedigermoeller.serialization.util.FSTUtil.rethrow(FSTUtil.java:119)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:474)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatible(FSTObjectInput.java:448)
    at de.ruedigermoeller.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:399)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:262)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:544)
    at de.ruedigermoeller.serialization.FSTObjectInput$2.defaultReadObject(FSTObjectInput.java:1180)
    at de.ruedigermoeller.serialization.FSTObjectInput$MyObjectStream.defaultReadObject(FSTObjectInput.java:1481)
    at com.agabar.laboratory.serialization.Objekt.readObject(Objekt.java:27)
    ... 12 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:471)
    ... 19 more
Caused by: java.io.StreamCorruptedException: BigDecimal: null intVal in stream
    at java.math.BigDecimal.readObject(BigDecimal.java:3551)
    ... 24 more

This exception corresponds to Oracle's BigDecimal not receiving intVal, which was actually written when serializing. I suspect there is some inconsistency when written fields and actual fields differ.

I would appreciate any help regarding this issue.

I can provide sample code if you need so.

Thank you.

Dimas Cabré i Chacón

First of all, thanks for this great serializer! I'm using it for every project I find a reason for because it is one of the best and most convenient one.

Today, I updated to the newest version (2.19) and also switched to the latest BouncyCastle version (bcprov-jdk15on:1.51). Serializing and deserializing a keypair generated with bouncy castle provider seems to run into an error. For your convenience, I created a Github Gist here.

Do I need to use a custom serializer to prevent this error (i.e. serializing keypairs manually)? Thanks for your help in advance!

Entered on behalf L. Almeida from blog.

I am testing fast-serialization library, but i replace ObjectInputStream and ObjectOutputStream in a socket and my app stoped work.

client
out = conf.getObjectOutput(s.getOutputStream());
out.writeObject(msg);
out.flush();
s is a socket

In the other side:

Server
FSTObjectInput in = conf.getObjectInput(str.getInputStream());

Stop here....

JCL_message msg = (JCL_message) in.readObject();

André Luís

[rm] probably readAhead on inputstream blocks on socketread ..

de.ruedigermoeller.serialization.FSTClazzNameRegistry.classForName source code

    public Class classForName(String clName) throws ClassNotFoundException {
        if ( parent != null ) {
            return parent.classForName(clName);
        }
        while( ! classCacheLock.compareAndSet(false,true) );
        Class res = classCache.get(clName);
        if ( res == null ) {
            try {
                res = Class.forName(clName, false, conf.getClassLoader() );
            } catch ( Throwable th ) {
                if ( clName.endsWith("_Struct") ) // hack to define struct proxys on the fly if sent from another process
                {
                    try {
                        clName = clName.substring(0,clName.length()-"_Struct".length());
                        Class onHeapStructClz = classCache.get(clName);
                        if ( onHeapStructClz == null )
                            onHeapStructClz = Class.forName(clName);
                        res = FSTStructFactory.getInstance().getProxyClass(onHeapStructClz);
                    } catch (Throwable th1) {
                        throw FSTUtil.rethrow(th1);
                    }
                } else {
                    throw new RuntimeException("CLASSNAME:"+clName,th);
                }
            }
            if ( res != null ) {
                classCache.put(clName, res);
            }
        }
        classCacheLock.set(false);
        return res;
    }

Should use a try finally block

    public Class classForName(String clName) throws ClassNotFoundException {
        if ( parent != null ) {
            return parent.classForName(clName);
        }
        while( ! classCacheLock.compareAndSet(false,true) );
        try {
            Class res = classCache.get(clName);
            if ( res == null ) {
                try {
                    res = Class.forName(clName, false, conf.getClassLoader() );
                } catch ( Throwable th ) {
                    if ( clName.endsWith("_Struct") ) // hack to define struct proxys on the fly if sent from another process
                    {
                        try {
                            clName = clName.substring(0,clName.length()-"_Struct".length());
                            Class onHeapStructClz = classCache.get(clName);
                            if ( onHeapStructClz == null )
                                onHeapStructClz = Class.forName(clName);
                            res = FSTStructFactory.getInstance().getProxyClass(onHeapStructClz);
                        } catch (Throwable th1) {
                            throw FSTUtil.rethrow(th1);
                        }
                    } else {
                        throw new RuntimeException("CLASSNAME:"+clName,th);
                    }
                }
                if ( res != null ) {
                    classCache.put(clName, res);
                }
            }
        } finally {
            classCacheLock.set(false);
        }
        return res;
    }

Or the code described here will deadlock on the on the deserialization of Bar if Foo is modified after it has been saved to file

package fstDeadlock;

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.Serializable;

import org.junit.Test;

import de.ruedigermoeller.serialization.FSTObjectInput;
import de.ruedigermoeller.serialization.FSTObjectOutput;

public class DeadLockTest {

    /**
     * Uncommenting this field and changing the uid will cause FST to throw an
     * exception and not release a lock
     * 
     * @author ostreifel
     * 
     */
    private static class Foo implements Serializable {
        private static final long serialVersionUID = 3859819121221212412L;
//      private String extraField = new String();
        private int fooField = 124124;
    }
    private static class Bar implements Serializable {
        private static final long serialVersionUID = -7928124124152171886L;
        private long barField = 124512L;
    }


    /**
     * Run once then<br />
     *  change the UID on Foo and uncomment its extra field<br />
     *  and it will hang on the deserialization of bar on the next run
     */
    @Test
    public void testWriteRead() {
        Foo foo = null;
        Bar bar = null;

        //Some reads
        if(new File("foo.dat").exists()) {
            try (FSTObjectInput fis = new FSTObjectInput(new BufferedInputStream(new FileInputStream("foo.dat")))) {
                System.out.println("Reading foo...");
                foo = (Foo) fis.readObject();
                System.out.println("Read foo: " + foo);
            } catch (IOException | ClassNotFoundException e) {
                System.out.println("couldn't read foo");
                e.printStackTrace();
            }
        }
        if(new File("bar.dat").exists()) {
            try (FSTObjectInput fis = new FSTObjectInput(new BufferedInputStream(new FileInputStream("bar.dat")))) {
                System.out.println("Reading bar...");
                bar = (Bar) fis.readObject();
                System.out.println("Read bar: " + bar);
            } catch (IOException | ClassNotFoundException e) {
                System.out.println("couldn't read bar");
            }
        }

        //Some writes
        foo = new Foo();
        try (FSTObjectOutput fos = new FSTObjectOutput(new BufferedOutputStream(new FileOutputStream("foo.dat")))) {
            System.out.println("Writing foo...");
            fos.writeObject(foo);
            System.out.println("Wrote foo: " + foo);
        } catch (IOException e) {
            e.printStackTrace();
        }
        bar = new Bar();
        try (FSTObjectOutput fos = new FSTObjectOutput(new BufferedOutputStream(new FileOutputStream("bar.dat")))) {
            System.out.println("Writing bar...");
            fos.writeObject(bar);
            System.out.println("Wrote bar: " + bar);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

}

While running FST 1.55 on Java 1.6 the following error occurs

Exception in thread "main" java.lang.UnsupportedClassVersionError: JVMCFRE003 bad major version; class=de/ruedigermoeller/serialization/FSTObjectOutput, offset=6

According to this thread https://code.google.com/p/fast-serialization/issues/detail?id=22 FST was to be made backwards compatible. Is that still the expectation?

Sorry for spamming you with nasty exceptions of Android incompatibilities. Here's the next one:

java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:876) ~[fst-2.22.jar:na]
    at org.hive2hive.core.security.FSTSerializer.deserialize(FSTSerializer.java:103) ~[classes/:na]
    at org.hive2hive.core.network.messages.MessageReplyHandler.reply(MessageReplyHandler.java:92) ~[classes/:na]
    at net.tomp2p.rpc.DirectDataRPC.handleResponse(DirectDataRPC.java:164) ~[tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at net.tomp2p.rpc.DispatchHandler.forwardMessage(DispatchHandler.java:166) ~[tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at net.tomp2p.connection.Dispatcher.channelRead0(Dispatcher.java:178) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at net.tomp2p.connection.Dispatcher.channelRead0(Dispatcher.java:59) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at net.tomp2p.message.TomP2PCumulationTCP.decoding(TomP2PCumulationTCP.java:71) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at net.tomp2p.message.TomP2PCumulationTCP.channelRead(TomP2PCumulationTCP.java:47) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at net.tomp2p.connection.IdleStateHandlerTomP2P.channelRead(IdleStateHandlerTomP2P.java:112) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-common-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.25.Final.jar:4.0.25.Final]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.io.IOException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:240) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:874) ~[fst-2.22.jar:na]
    ... 31 common frames omitted
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:561) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatible(FSTObjectInput.java:523) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:508) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:657) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:515) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:323) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:304) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:238) ~[fst-2.22.jar:na]
    ... 32 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:558) ~[fst-2.22.jar:na]
    ... 41 common frames omitted
Caused by: java.io.IOException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.FSTObjectInput$2.defaultReadObject(FSTObjectInput.java:988) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput$MyObjectStream.defaultReadObject(FSTObjectInput.java:1268) ~[fst-2.22.jar:na]
    at org.bouncycastle.jcajce.provider.asymmetric.rsa.BCRSAPublicKey.readObject(Unknown Source) ~[bcprov-jdk15on-1.51.jar:1.51.0]
    ... 46 common frames omitted
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:561) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatible(FSTObjectInput.java:523) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:508) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:657) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput$2.defaultReadObject(FSTObjectInput.java:978) ~[fst-2.22.jar:na]
    ... 48 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:558) ~[fst-2.22.jar:na]
    ... 53 common frames omitted
Caused by: java.lang.RuntimeException: unknown object tag -128
    at org.nustaq.serialization.FSTObjectInput.instantiateSpecialTag(FSTObjectInput.java:402) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:349) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readCompatibleObjectFields(FSTObjectInput.java:782) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput$2.readFields(FSTObjectInput.java:1001) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput$MyObjectStream.readFields(FSTObjectInput.java:1273) ~[fst-2.22.jar:na]
    at java.math.BigInteger.readObject(BigInteger.java:3096) ~[na:1.7.0_71]
    ... 57 common frames omitted

As soon as I've more time, I'll look deeper into the FST code to probably be able to fix them myself.

all other type is good. only Float type has invalid results.

my serializer like this

class FstRedisSerializer[T] extends RedisSerializer[T] {

    private lazy val log = LoggerFactory.getLogger(getClass)



    override def serialize(graph: T): Array[Byte] = {
        if (graph == null || graph == None)
            return EMPTY_BYTES

        var bos = None: Option[ByteArrayOutputStream]

        try {
            bos = Some(new io.ByteArrayOutputStream())
            Try(FstRedisSerializer.conf.getObjectOutput(bos.get)) match {

                case Success(oos) =>
                    oos.writeObject(graph, Seq[Class[_]](): _*)
                    oos.flush()
                    bos.get.toByteArray

                case Failure(e) =>
                    log.error(s"Fail to serialize graph. $graph", e)
                    EMPTY_BYTES
            }
        } finally {
            if (bos.isDefined) bos.get.close()
        }
    }

    override def deserialize(bytes: Array[Byte]): T = {
        if (bytes == null || bytes.length == 0)
            return null.asInstanceOf[T]

        var bis = None: Option[ByteArrayInputStream]

        try {
            bis = Some(new ByteArrayInputStream(bytes))
            Try(FstRedisSerializer.conf.getObjectInput(bis.get)) match {

                case Success(ois) =>
                    val instance = ois.readObject.asInstanceOf[T]
                    instance

                case Failure(e) =>
                    log.error(s"Fail to deserialize data.", e)
                    null.asInstanceOf[T]
            }
        } finally {
            if (bis.isDefined) bis.get.close()
        }
    }
}

object FstRedisSerializer {
    lazy val conf = FSTConfiguration.createDefaultConfiguration()
}

I wrote same version in java, same results.

Model data is

@SerialVersionUID(-8245742950718661800L)
class Person extends Serializable {

    @Id
    @GeneratedValue
    var id: java.lang.Long = _

    var age: Integer = _
    var firstName: String = _
    var lastName: String = _

    var height: java.lang.Float = 180.8f
    var weight: java.lang.Double = 77.7
}

Double type return same value, but Float type return other value

180.8 => -4.28247136E8

my code is correct?

#2 is back, tested with 2.08. Test case available at http://pastebin.com/NYmJiu4s

It works if the setForceSerializable line is uncommented, and the custom serializer does get invoked in that case. But the documentation doesn't say you need setForceSerializable to use custom serializers, and that would result in unwanted behavior for other non-serializable classes that don't have custom serializers registered.

Correct me if I'm wrong but one must use a ByteArrayOutputStream if using compression?

e.g.

      ByteArrayOutputStream bos = new ByteArrayOutputStream();
      BufferedOutputStream buffered = new BufferedOutputStream(new GZIPOutputStream(bos));
      FSTObjectOutput oos = conf.getObjectOutput(buffered);
      oos.writeObject(chart);
      oos.flush();
      buffered.close();
      serialized_data.add(bos.toByteArray());

I only ask because of your comment to use conf.asByteArray

 byte barray[] = conf.asByteArray(mySerializableObject);

Which is simpler, cleaner, and more efficient(?) but at least for my data even with registering all the classes ~3x larger than after gzip.

Thanks!

I have some classes that I want to serialize. These classes happen to contain an NIO ByteBuffer. The first time I tried to serialize an object, FST correctly complained that ByteBuffer is neither Serializable nor Externalizable. So I implemented a custom serializer for ByteBuffer and registered it with alsoForAllSubclasses = true, but I'm still getting the same exception:

Exception in thread "main" java.lang.RuntimeException: Class java.nio.HeapByteBuffer does not implement Serializable or externalizable
    at de.ruedigermoeller.serialization.FSTClazzInfo.<init>(FSTClazzInfo.java:108)
    at de.ruedigermoeller.serialization.FSTClazzInfoRegistry.getCLInfo(FSTClazzInfoRegistry.java:129)
    at de.ruedigermoeller.serialization.FSTObjectOutput.getFstClazzInfo(FSTObjectOutput.java:387)
    at de.ruedigermoeller.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:321)
    at de.ruedigermoeller.serialization.FSTObjectOutput.writeObjectInternal(FSTObjectOutput.java:241)
    at de.ruedigermoeller.serialization.FSTObjectOutput.writeObject(FSTObjectOutput.java:210)
    at de.ruedigermoeller.serialization.FSTObjectOutput.writeObject(FSTObjectOutput.java:198)
    at dg.FSTByteBufferTest.main(FSTByteBufferTest.java:22)

Simple test that reproduces the problem:

package dg;

import java.io.IOException;
import java.nio.ByteBuffer;

import de.ruedigermoeller.serialization.FSTBasicObjectSerializer;
import de.ruedigermoeller.serialization.FSTClazzInfo;
import de.ruedigermoeller.serialization.FSTClazzInfo.FSTFieldInfo;
import de.ruedigermoeller.serialization.FSTConfiguration;
import de.ruedigermoeller.serialization.FSTObjectInput;
import de.ruedigermoeller.serialization.FSTObjectOutput;

public class FSTByteBufferTest {

    public static void main(String[] args) throws Exception {
        ByteBuffer buffer = ByteBuffer.allocate(100);

        FSTConfiguration fst = FSTConfiguration.createDefaultConfiguration();
        fst.registerSerializer(ByteBuffer.class, new ByteBufferSerializer(), true);

        FSTObjectOutput out = fst.getObjectOutput();
        out.writeObject(buffer);
        System.out.println(out.getWritten());
    }

    public static class ByteBufferSerializer extends FSTBasicObjectSerializer {

        @Override
        public void writeObject(FSTObjectOutput out, Object toWrite, FSTClazzInfo clzInfo,
                FSTFieldInfo referencedBy, int streamPosition) throws IOException {
            ByteBuffer buffer = (ByteBuffer) toWrite;
            byte[] array = new byte[buffer.limit()];
            buffer.position(0);
            buffer.get(array);
            out.writeFInt(array.length);
            out.write(array);
        }

        @Override
        public Object instantiate(@SuppressWarnings("rawtypes") Class objectClass, FSTObjectInput in,
                FSTClazzInfo serializationInfo, FSTFieldInfo referencee, int streamPositioin) throws IOException,
                ClassNotFoundException, InstantiationException, IllegalAccessException {
            byte[] array = new byte[in.readFInt()];
            in.read(array);
            return ByteBuffer.wrap(array);
        }

    }

}

Thanks for the quick fix of #12. Unfortunately version 1.56 introduced another problem.

Take the following test program:

import de.ruedigermoeller.serialization.FSTObjectInput;
import de.ruedigermoeller.serialization.FSTObjectOutput;
import javax.security.auth.Subject;
import java.security.Principal;
import java.io.ByteArrayOutputStream;
import java.io.ByteArrayInputStream;
import java.io.Serializable;

import java.net.InetAddress;

public class Test
{
    public static void main(String[] args) throws Exception
    {
        ByteArrayOutputStream buffer = new ByteArrayOutputStream();
        FSTObjectOutput out = new FSTObjectOutput(buffer);
        out.writeObject(InetAddress.getByName("::1"));
        out.close();
        FSTObjectInput stream = new FSTObjectInput(new
                                                   ByteArrayInputStream(buffer.toByteArray()));
        stream.readObject();
    }
}

This works fine with 1.55 but fails with 1.56:

$ java -cp ~/.m2/repository/de/ruedigermoeller/fst/1.55/fst-1.55.jar:. Test
$ java -cp ~/.m2/repository/de/ruedigermoeller/fst/1.56/fst-1.56.jar:. Test
    Exception in thread "main" java.io.IOException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:168)
        at Test.main(Test.java:21)
    Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at de.ruedigermoeller.serialization.util.FSTUtil.rethrow(FSTUtil.java:119)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:460)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatible(FSTObjectInput.java:434)
        at de.ruedigermoeller.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:395)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:261)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:230)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:210)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:165)
        ... 1 more
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:457)
        ... 7 more
    Caused by: java.lang.NullPointerException
        at java.net.Inet6Address.readObject(Inet6Address.java:622)
        ... 12 more

This is with Oracle JRE 1.7.0_51.

It appears that method writeObjectCompatibleRecursive is trying to serialize the super class of the Externalizable class first:

writeObjectCompatibleRecursive(referencee,toWrite,serializationInfo,cl.getSuperclass());

This makes no sense. See Externalizable spec http://docs.oracle.com/javase/7/docs/api/java/io/Externalizable.html

readExternal and writeExternal should do all the work

My test case is Externalizable sub-class of a Serializable base class.

What is the proper way of running the test suite?

I could not find instructions and the usual routes. I wanted to contribute something but would be more comfortable if I could TDD it to make sure nothing else is broken.

BigDecimal cannot processed. Try this little code(Java8):

Vector data = new Vector();
// data.add(new Double(22.65));
data.add(new BigDecimal(22.65));

    byte[] result = null;

    try (ByteArrayOutputStream out = new ByteArrayOutputStream(); FSTObjectOutput o = new FSTObjectOutput(out)) {
        o.writeObject(data);
        result = out.toByteArray();
    }

    try (ByteArrayInputStream in = new ByteArrayInputStream(result); FSTObjectInput o = new FSTObjectInput(in)) {
        try {
            Object data1 = o.readObject();
            data1 = data1;
        } catch (Exception ex) {
            Logger.getLogger(Serializer.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

I would like to reuse object handles based on the Object.equals method for a subset of classes in a graph as opposed to just using IdentityHashCode. Is there a way to plug-in this kind of object custom replacement code? For standard Java ObjectOutputStream we subclass it and override replaceObject method.

Hi Rüdiger,

Still exploring using FST for our Wicket apps and we think we found an issue: AFAICS, FST doesn't call the readResolve method in the case of an Externalizable object whereas the standard JDK serialization do so.

A coworker of mine worked on a test case to demonstrate the issue I posted here: http://share.openwide.fr/share/zone/gsmet/180169339 .

It's a simple Maven project with a unit test.

Our real case is obviously less dumb than this test case!

Thanks!

I have a big problem with the library which is not very transparent for me. Here the exception i get in diffrent situations:

Exception in thread "main" de.hetzge.sgame.common.exception.NetworkException: java.io.IOException: java.lang.RuntimeException: CLASSNAME:
at de.hetzge.sgame.network.Client.connect(Client.java:47)
at de.hetzge.sgame.network.NetworkModule.update(NetworkModule.java:36)
at de.hetzge.sgame.application.ModulePool.update(ModulePool.java:30)
at de.hetzge.sgame.application.Application.update(Application.java:27)
at de.hetzge.sgame.application.Application.start(Application.java:17)
at de.hetzge.sgame.game.Client.main(Client.java:32)
Caused by: java.io.IOException: java.lang.RuntimeException: CLASSNAME:
at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:242)
at de.hetzge.sgame.common.serializer.Serializer.deserialize(Serializer.java:35)
at de.hetzge.sgame.network.Connection.read(Connection.java:26)
at de.hetzge.sgame.network.Client.connect(Client.java:36)
... 5 more
Caused by: java.lang.RuntimeException: CLASSNAME:
at org.nustaq.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:212)
at org.nustaq.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:183)
at org.nustaq.serialization.FSTClazzNameRegistry.decodeClass(FSTClazzNameRegistry.java:166)
at org.nustaq.serialization.coders.FSTStreamDecoder.readClass(FSTStreamDecoder.java:422)
at org.nustaq.serialization.FSTObjectInput.readClass(FSTObjectInput.java:855)
at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:340)
at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:659)
at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:517)
at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:358)
at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:325)
at org.nustaq.serialization.serializers.FSTCollectionSerializer.instantiate(FSTCollectionSerializer.java:90)
at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:461)
at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356)
at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:659)
at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:517)
at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:358)
at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:325)
at org.nustaq.serialization.serializers.FSTCollectionSerializer.instantiate(FSTCollectionSerializer.java:90)
at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:461)
at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356)
at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:659)
at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:517)
at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:358)
at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:325)
at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:306)
at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:240)
... 8 more
Caused by: java.lang.ClassNotFoundException:
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at org.nustaq.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:190)
... 33 more

The classname of the class which is not found is empty. I think the reason for the exception ist something else. The application sends objects over the network and deserialize them with the library. The error happens if i send more then a special amount of objects. If i send 10 objects (packed in a collection) it works fine. But if i increase to say 50 objects the error happens. All changed is the amount of the objects which are created the same way.

Here are my serialize methods:

public static byte[] serialize(Object object) throws IOException {
        FSTObjectOutput objectOutput = CommonConfig.INSTANCE.fst.getObjectOutput();
        objectOutput.writeObject(object);

        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        byteArrayOutputStream.write(objectOutput.getBuffer(), 0, objectOutput.getWritten());

        byte[] byteArray = byteArrayOutputStream.toByteArray();
        byteArrayOutputStream.close();
        return byteArray;
    }

    public static Object deserialize(byte[] bytes) throws IOException, ClassNotFoundException {
        ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
        FSTObjectInput objectInput = CommonConfig.INSTANCE.fst.getObjectInput(byteArrayInputStream);
        Object readObject = objectInput.readObject();
        byteArrayInputStream.close();
        return readObject;
    }

Here the code i use for the network communication:

public class Connection {

    private final Socket socket;
    private final DataOutputStream dataOutputStream;
    private final DataInputStream dataInputStream;

    public Connection(Socket socket) throws IOException {
        this.socket = socket;
        this.dataOutputStream = new DataOutputStream(socket.getOutputStream());
        this.dataInputStream = new DataInputStream(socket.getInputStream());
    }

    public Object read() throws IOException, ClassNotFoundException {
        int length = this.dataInputStream.readInt();
        byte[] byteArray = new byte[length];
        this.dataInputStream.read(byteArray);
        return Serializer.deserialize(byteArray);
    }

    public void write(Object message) throws IOException {
        byte[] serialized = Serializer.serialize(message);
        this.dataOutputStream.writeInt(serialized.length);
        this.dataOutputStream.write(serialized);
        this.dataOutputStream.flush();
    }

}

It looks like registering the classes allows a higher amount of classes.

Any idea what i am doing wrong ?
... or are more informations needed ?

I'd like to be able to create a temporary (eventually garbage-collected) configuration with a custom ClassLoader.

I had tried clicking on

https://github.com/RuedigerMoeller/fast-serialization/blob/master/src/test/java/de/ruedigermoeller/serialization/testclasses/docusample/FSTTestApp.java

but that file (referred in the explanation page) is not avialable

what can I use as example source ( Need to serialize a sorted array of structs)

Allow to read serialized Java Objects from other languages such as .js or dart

Comment #2 on issue 24 by ludo.antin: no exception when deserializing object with list of objects (with changed class)
http://code.google.com/p/fast-serialization/issues/detail?id=24

I used FST since 6 months. For the next release of my software, I must add a new field on a serialized class. Like "mindwire..." I got a NullPointerException.

Is there a way to deserialize my previous serialized objects which don't have the new field ? If not, how can I update my previous serialized objects ?

Thanks for your help

Hi Mr. Ruediger,

I have question more than issue. I have object which has reference to other quite big other objects. I don't want to serialize this objects so I change this references to transient (eg.: private transient MyProcessor myProcessor;) I also annotate MyProcessor interface with org.nustaq.serialization.annotations.Transient annotation but it still ends up with stack overflow because of recurrent reference. Can you please give me some advice how to solve this issue?

Hello,

I am getting an error when I serialize a private nested class, because its parent (the one where the inner class is defined in) is not serializable. I was not expecting that the parent class would also be serialized, this seems a bit odd ... and possible not intended? Moving the particular class to its own type resolved the issue, but we might have more cases like this yet to be detected.

This is on version 1.55 (JRE 6 compatible version).

Please advise on the best way to proceed.

thanks!

Hi Rüdiger,

I finally pushed our Fast work to Wicketstuff (see wicketstuff/core#361) but we have an issue with this PR.

Fast 1.x was compiled with JDK 1.6 (at least until 1.38 which was the last version used in the serializer-fast module of Wicketstuff) but 1.61 is compiled with 1.7.

While I have no problem pushing the new fast2 module for Wicketstuff to 7, I think the 1.x line should still be compiled with 1.6.

WDYT?

FSTObjectInput#read() currently does this:

return getCodec().readFByte();

Java sign extends the value when performing a widening conversion, from byte to int in this case, which means that read() is returning a signed int.

The Javadoc for ObjectInput#read() implies that the return value should be unsigned. It says the method should return "the byte read, or -1 if the end of the stream is reached". If the value were supposed to be signed, then -1 would be a valid value and could not be used as a sentinel.

So read() should do this instead, to make the value unsigned:

return getCodec().readFByte() & 0xFF;

@Test
public void fstTest3() throws InterruptedException {
    DefaultCoder c = new DefaultCoder();
    Random r = new Random();
    CountDownLatch l = new CountDownLatch(1);
    for (int i = 0; i < 10; i++) {
        Thread t = new Thread() {
            @Override
            public void run() {
                try {
                    l.await();
                    long a = r.nextLong();
                    byte[] buf = c.toByteArray(a);
                    long b = (long) c.toObject(buf);
                    Assert.assertEquals(a, b);
                } catch (InterruptedException e) {
                }
            }
        };
        t.start();
    }

    l.countDown();
    Thread.sleep(10000);
}

env:

java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)

<dependency>
    <groupId>de.ruedigermoeller</groupId>
    <artifactId>fst</artifactId>
    <version>2.14</version>
</dependency>

and that's what I got:

Exception in thread "Thread-9" Exception in thread "Thread-7" Exception in thread "Thread-2" Exception in thread "Thread-1" Exception in thread "Thread-8" Exception in thread "Thread-0" java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.nustaq.serialization.FSTObjectOutput.getCopyOfWrittenBuffer(FSTObjectOutput.java:1069)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toByteArray(DefaultCoder.java:79)
    at mysh.benchmark.FstTest$1.run(FstTest.java:38)
Exception in thread "Thread-4" Exception in thread "Thread-5" Exception in thread "Thread-3" Exception in thread "Thread-6" java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.nustaq.serialization.FSTObjectOutput.getCopyOfWrittenBuffer(FSTObjectOutput.java:1069)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toByteArray(DefaultCoder.java:79)
    at mysh.benchmark.FstTest$1.run(FstTest.java:38)
java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:127)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:97)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:102)
    at mysh.benchmark.FstTest$1.run(FstTest.java:39)
Caused by: java.io.IOException: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:246)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:95)
    ... 2 more
Caused by: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.FSTObjectInput.instantiateSpecialTag(FSTObjectInput.java:408)
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:355)
    at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:329)
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:310)
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:244)
    ... 3 more
java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.nustaq.serialization.FSTObjectOutput.getCopyOfWrittenBuffer(FSTObjectOutput.java:1069)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toByteArray(DefaultCoder.java:79)
    at mysh.benchmark.FstTest$1.run(FstTest.java:38)
java.lang.AssertionError: expected:<-5094390369870799988> but was:<13>
    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.failNotEquals(Assert.java:743)
    at org.junit.Assert.assertEquals(Assert.java:118)
    at org.junit.Assert.assertEquals(Assert.java:555)
    at org.junit.Assert.assertEquals(Assert.java:542)
    at mysh.benchmark.FstTest$1.run(FstTest.java:40)
java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.nustaq.serialization.FSTObjectOutput.getCopyOfWrittenBuffer(FSTObjectOutput.java:1069)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toByteArray(DefaultCoder.java:79)
    at mysh.benchmark.FstTest$1.run(FstTest.java:38)
java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.nustaq.serialization.FSTObjectOutput.getCopyOfWrittenBuffer(FSTObjectOutput.java:1069)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toByteArray(DefaultCoder.java:79)
    at mysh.benchmark.FstTest$1.run(FstTest.java:38)
java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:127)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:97)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:102)
    at mysh.benchmark.FstTest$1.run(FstTest.java:39)
Caused by: java.io.IOException: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:246)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:95)
    ... 2 more
Caused by: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.FSTObjectInput.instantiateSpecialTag(FSTObjectInput.java:408)
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:355)
    at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:329)
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:310)
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:244)
    ... 3 more
java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.nustaq.serialization.FSTObjectOutput.getCopyOfWrittenBuffer(FSTObjectOutput.java:1069)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toByteArray(DefaultCoder.java:79)
    at mysh.benchmark.FstTest$1.run(FstTest.java:38)
java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:127)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:97)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:102)
    at mysh.benchmark.FstTest$1.run(FstTest.java:39)
Caused by: java.io.IOException: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:246)
    at org.nustaq.serialization.simpleapi.DefaultCoder.toObject(DefaultCoder.java:95)
    ... 2 more
Caused by: java.lang.RuntimeException: unknown object tag -126
    at org.nustaq.serialization.FSTObjectInput.instantiateSpecialTag(FSTObjectInput.java:408)
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:355)
    at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:329)
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:310)
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:244)
    ... 3 more

Process finished with exit code 0

Your lib does not seem to be working on Android as the sun.reflect.ReflectionFactory is missing:

https://gist.github.com/knyttl/dfcfefbc0fc58a268ecb

I would really like to use your library, however this makes it unusuable :/

Hi, I got a NullPointerException when I tried to serialize an object from a custom class which has two stack of string fields, but there wasn't any problem when serializing objects with only one of that field (so I suppose it's not only a matter of Stack class incompatibility). Both cases can be tested through the following code:

import java.io.IOException;
import java.io.Serializable;
import java.util.Stack;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.nustaq.serialization.FSTConfiguration;
import org.nustaq.serialization.FSTObjectInput;
import org.nustaq.serialization.FSTObjectOutput;

public class FSTEqualFieldsTest {

static class OneStackClass implements Serializable {

    Stack<String> a;

    public OneStackClass(Stack<String> a) {
        this.a = a;
    }
}

static class TwoStackClass implements Serializable {

    Stack<String> a;
    Stack<String> b;

    public TwoStackClass(Stack<String> a, Stack<String> b) {
        this.a = a;
        this.b = b;
    }
}

public static void main(String args[]) {

    OneStackClass osc = new OneStackClass(new Stack<String>());
    byte[] mywriteMethod1 = FSTSerialization.mywriteMethod(osc);
    OneStackClass myreadMethod1 = FSTSerialization.myreadMethod(mywriteMethod1, OneStackClass.class);
    System.out.println("object well serializated: " + myreadMethod1);
    //
    TwoStackClass tsc = new TwoStackClass(new Stack<String>(), new Stack<String>());
    byte[] mywriteMethod2 = FSTSerialization.mywriteMethod(tsc);
    TwoStackClass myreadMethod2 = FSTSerialization.myreadMethod(mywriteMethod2, TwoStackClass.class);
    System.out.println("object well serializated? " + myreadMethod2);

}

static public class FSTSerialization {

    final static FSTConfiguration configuration;

    public static FSTConfiguration getConfiguration() {
        return configuration;
    }

    static {
        configuration = FSTConfiguration.createDefaultConfiguration();
        configuration.setShareReferences(true);
        configuration.registerClass(
                Stack.class,
                String.class
        );
    }

    public static <T> T myreadMethod(byte[] stream, Class<T> c) {
        FSTObjectInput in = configuration.getObjectInput(stream);
        T result = null;
        try {
            result = (T) in.readObject(c);
        } catch (Exception ex) {
            Logger.getLogger(FSTSerialization.class.getName()).log(Level.SEVERE, null, ex);
        }
        return result;
    }

    @SuppressWarnings("null")
    public static byte[] mywriteMethod(Object toWrite) {
        return mywriteMethod(null, toWrite);
    }

    public static byte[] mywriteMethod(byte[] stream, Object toWrite) {
        FSTObjectOutput out = configuration.getObjectOutput(stream);
        try {
            out.writeObject(toWrite, toWrite.getClass());
        } catch (IOException ex) {
            Logger.getLogger(FSTSerialization.class.getName()).log(Level.SEVERE, null, ex);
        }
        try {
            out.flush();
        } catch (IOException ex) {
            Logger.getLogger(FSTSerialization.class.getName()).log(Level.SEVERE, null, ex);
        }
        return out.getBuffer();
    }

}

}

Hello,

First off, much appreciate your efforts on this awesome library.

We are currently using fast-serialization indirectly through https://github.com/debop/hibernate-redis. And we recently migrated our project to Java 8 to take advantage of all that lambda function goodness. Currently FST doesn't seem to support Java 8. I'm seeing the attached stack trace when trying to deserialize an enum that is part of a hibernate bean.

Any plans to support Java 8 in the near future?

Much thanks.

22:48:59.295 [application-akka.actor.default-dispatcher-9] WARN o.h.c.r.s.FstRedisSerializer.deserialize 53 - Fail to deserialize bytes.
java.io.IOException: java.lang.RuntimeException: CLASSNAME:com.bvg.models.enums.Gender
at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:168) ~[fst-1.53.jar:na]
at org.hibernate.cache.redis.serializer.FstRedisSerializer.deserialize(FstRedisSerializer.java:51) ~[hibernate-redis-1.5.10.jar:na]
at org.hibernate.cache.redis.serializer.SnappyRedisSerializer.deserialize(SnappyRedisSerializer.java:41) [hibernate-redis-1.5.10.jar:na]
at org.hibernate.cache.redis.jedis.JedisClient.deserializeValue(JedisClient.java:475) [hibernate-redis-1.5.10.jar:na]
at org.hibernate.cache.redis.jedis.JedisClient.get(JedisClient.java:165) [hibernate-redis-1.5.10.jar:na]
at org.hibernate.cache.redis.regions.RedisTransactionalDataRegion.get(RedisTransactionalDataRegion.java:75) [hibernate-redis-1.5.10.jar:na]
at org.hibernate.cache.redis.strategy.AbstractReadWriteRedisAccessStrategy.get(AbstractReadWriteRedisAccessStrategy.java:52) [hibernate-redis-1.5.10.jar:na]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:55) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:67) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:597) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:451) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:212) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:274) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:150) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1070) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.internal.SessionImpl.access$2000(SessionImpl.java:176) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.internal.SessionImpl$IdentifierLoadAccessImpl.load(SessionImpl.java:2551) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.internal.SessionImpl.get(SessionImpl.java:955) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at com.bvg.models.daos.DaoImpl.get(DaoImpl.java:78) [classes/:na]
at com.bvg.models.daos.DaoImpl.getWithRetry(DaoImpl.java:83) [classes/:na]
at com.bvg.actors.DbBasedActor.execute(DbBasedActor.java:52) [classes/:na]
at com.bvg.actors.DbBasedActor.execute(DbBasedActor.java:16) [classes/:na]
at com.bvg.models.session.HibernateSessionHelper.executeTransactionally(HibernateSessionHelper.java:82) [classes/:na]
at com.bvg.actors.DbBasedActor.onReceive(DbBasedActor.java:37) [classes/:na]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.10.jar:2.2.0]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) [akka-actor_2.10.jar:2.2.0]
at akka.actor.ActorCell.invoke(ActorCell.scala:456) [akka-actor_2.10.jar:2.2.0]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) [akka-actor_2.10.jar:2.2.0]
at akka.dispatch.Mailbox.run(Mailbox.scala:219) [akka-actor_2.10.jar:2.2.0]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [akka-actor_2.10.jar:2.2.0]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library.jar:na]
Caused by: java.lang.RuntimeException: CLASSNAME:com.bvg.models.enums.Gender
at de.ruedigermoeller.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:198) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:178) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTClazzNameRegistry.decodeClass(FSTClazzNameRegistry.java:161) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readClass(FSTObjectInput.java:879) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.instantiateEnum(FSTObjectInput.java:335) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.instantiateSpecialTag(FSTObjectInput.java:272) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:254) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readArray(FSTObjectInput.java:816) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.instantiateArray(FSTObjectInput.java:325) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.instantiateSpecialTag(FSTObjectInput.java:287) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:254) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:522) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:402) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:261) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:230) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:210) ~[fst-1.53.jar:na]
at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:165) ~[fst-1.53.jar:na]
... 33 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.bvg.models.enums.Gender
at java.net.URLClassLoader$1.run(URLClassLoader.java:372) ~[na:1.8.0_05]
at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ~[na:1.8.0_05]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_05]
at java.net.URLClassLoader.findClass(URLClassLoader.java:360) ~[na:1.8.0_05]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_05]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_05]
at java.lang.Class.forName0(Native Method) ~[na:1.8.0_05]
at java.lang.Class.forName(Class.java:340) ~[na:1.8.0_05]
at de.ruedigermoeller.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:184) ~[fst-1.53.jar:na]
... 49 common frames omitted

On this page: https://github.com/RuedigerMoeller/fast-serialization/wiki/Serialization it is shown how to make a custom serializor.

But how can a Custom serializor be made for a Date[] for instance, which uses a special method for construting this?

Tried upgrading to 2.00 but can't set the classloader now for FSTConfiguration. I'm developing a NetBeans Platform (NBP) project, in NBP an app consists of modules, each having it's own classloader. Libraries are also wrapped in modules, so FST sits in it's own module with it's own classloader, which knows nothing about surrounding classes, loaded by other modules' classloaders. In 1.58 I've just did:

FSTConfiguration conf = FSTConfiguration.createDefaultConfiguration();
conf.setClassLoader(classloaderThatKnowsEverything);

I see that there are some .registerXXX() methods in there, but undocumented, so I'm not sure if that's an omission or a deliberate change of API.

I am interested in using your library for replacing Java serialization, but I need support for replacing exported remote objects with stubs (Similar to MarshalOutputStream)?
Or is it possible to add this through some extension point?

Would you ever consider changing to a more permissive license? Something like BSD, MIT, or Apache License, Version 2.0?

LGPL can be a bit difficult for some companies to stomach, particularly the clause about allowing users to reverse engineer their application. Also on Android, all of your code & dependent libraries are combined into a DEX file, making it difficult comply with the part of LGPL mandating that the user be able to swap out the library with an updated version.

I'm working on an Android port of this library (and of course plan to submit PRs with the changes), but may not actually be able to use it with the current license.

When cloning and trying to build the project, it gives gradle resolution errors:
Caused by: java.net.UnknownHostException: terminator

It appears to take that from gradle.properties which specifies:
systemProp.http.proxyHost=terminator

This seems like it should be in an environment specific location instead.

Hi

I am trying to use FTS with Android. ( android.graphics.path ) But I get errors and the app crashes.
Is Android supported to use FTS ?
If generally Android should be supported:
I would have stacktraces and a full android project if you like to analyse it.

kind regards
Andreas

Although stated as Android compatible, I ran into an exception today:

java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: java.lang.NoSuchMethodError: sun.misc.Unsafe.putBoolean
            at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[na:0.0]
            at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:875) ~[na:0.0]
            at org.hive2hive.core.security.FSTSerializer.deserialize(FSTSerializer.java:59) ~[na:0.0]
            at org.hive2hive.core.security.H2HDefaultEncryption.decryptAES(H2HDefaultEncryption.java:43) ~[na:0.0]
            at org.hive2hive.core.network.data.vdht.EncryptedVersionManager.get(EncryptedVersionManager.java:178) ~[na:0.0]
            at org.hive2hive.core.network.data.UserProfileManager$QueueWorker.run(UserProfileManager.java:203) ~[na:0.0]
            at java.lang.Thread.run(Thread.java:841) ~[na:0.0]
     Caused by: java.io.IOException: java.lang.RuntimeException: java.lang.NoSuchMethodError: sun.misc.Unsafe.putBoolean
            at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:240) ~[na:0.0]
            at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:873) ~[na:0.0]
    ... 5 common frames omitted
    Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: sun.misc.Unsafe.putBoolean
            at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:327) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:304) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:238) ~[na:0.0]
    ... 6 common frames omitted
    Caused by: java.lang.NoSuchMethodError: sun.misc.Unsafe.putBoolean
            at org.nustaq.serialization.FSTClazzInfo$FSTFieldInfo.setBooleanValue(FSTClazzInfo.java:858) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:634) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:515) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:657) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:515) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:323) ~[na:0.0]
    ... 8 common frames omitted

Unfortunately, the Unsafe implementation of Android is incomplete and not all methods are available.

I tested with FST v.2.21 and Android 4.4.4. For the compilation, Android Studio with Java 7u71 is used.

The following issue was discovered while porting our application to Java 8. The core problem is however an API incompatibility between FST and stock Java Serialization that just happens to cause problems with the Java 8 version of javax.security.auth.Subject.

First a test case for reproducing the issue:

import de.ruedigermoeller.serialization.FSTObjectInput;
import de.ruedigermoeller.serialization.FSTObjectOutput;
import javax.security.auth.Subject;
import java.io.ByteArrayOutputStream;
import java.io.ByteArrayInputStream;

public class Test
{
    public static void main(String[] args) throws Exception
    {
        ByteArrayOutputStream buffer = new ByteArrayOutputStream();
        FSTObjectOutput out = new FSTObjectOutput(buffer);
        out.writeObject(new Subject());
        out.close();
        FSTObjectInput stream = new FSTObjectInput(new
        ByteArrayInputStream(buffer.toByteArray()));
        stream.readObject();
    }
}

This works just fine on Java 7, but fails on Java 8 with:

Exception in thread "main" java.io.IOException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:168)
        at Test.main(Test.java:17)
        Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at de.ruedigermoeller.serialization.util.FSTUtil.rethrow(FSTUtil.java:119)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:460)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatible(FSTObjectInput.java:434)
        at de.ruedigermoeller.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:395)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:261)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:230)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:210)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObject(FSTObjectInput.java:165)
        ... 1 more
        Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:457)
        ... 7 more
        Caused by: java.lang.RuntimeException: CLASSNAME:%
        at de.ruedigermoeller.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:198)
        at de.ruedigermoeller.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:178)
        at de.ruedigermoeller.serialization.FSTClazzNameRegistry.decodeClass(FSTClazzNameRegistry.java:161)
        at de.ruedigermoeller.serialization.FSTObjectInput.readClass(FSTObjectInput.java:879)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:245)
        at de.ruedigermoeller.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:230)
        at de.ruedigermoeller.serialization.FSTObjectInput$2.readFields(FSTObjectInput.java:1171)
        at de.ruedigermoeller.serialization.FSTObjectInput$MyObjectStream.readFields(FSTObjectInput.java:1452)
        at javax.security.auth.Subject.readObject(Subject.java:966)
        ... 12 more
        Caused by: java.lang.ClassNotFoundException: %
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:340)
        at de.ruedigermoeller.serialization.FSTClazzNameRegistry.classForName(FSTClazzNameRegistry.java:184)
        ... 20 more

The core of the issue is how Subject is serialized in Java 8. It implements writeObject and readObject, with writeObject calling defaultWriteObject and readObject calling readFields on the ObjectInputStream and reading back individual fields. In regular Java Serialization, the wire format used by defaultWriteObject/defaultReadObject and writeFields/readFields is apparently the same. In FST, the wire formats for these calls appear to be different, preventing the class to read with readFields what was written with defaultWriteObject.

The above behavior of Java Serialization doesn't appear to be defined anywhere, but since the JDK depends on in, I am afraid FST has to mimic this part if it wants to be a drop-in replacement.

Just to be clear: I am not talking about FST using the same wire protocol as Java Serialization. What I am trying to say is that the serialized form generated by defaultWriteObject must be readable with readFields if you want FST to be API compatible (which from my point of view is the killer feature of FST over competitors).

Hi, I have issue with your library. Unfortunately, I can not post here any source code to be more specific. It seems, when you deserialize my object which is not marked as Serializable, you read incorrect class name from stream which contains also data which are no more part of class name.

During my investigation I find out that in FSTStreamDecoder.readStringUTF() you are using readFInt() which returns incorrect value. In general, this can be also problem of serialization process, who knows.
For more information please contact me at: stehlik.michal(at)gmail.com.

Can't increase fc 3 history beyond 1.3 GB

I have a scenario where I have existing Java serialized streams stored in databases (BLOB fields) and files and these were serialized with the JDK's built in serialization from many years ago.

Can fast-serialization read in these old streams and realize that they are the original JDK and so deserialize using the JDKs deserializer instead of it's own?

I profiled serialization of several large graphs 8 -10 MB serialized binary.

It appears that this method is essentially synchronized by using a single AtomicBoolean. It becomes a synchronization hotspot very quickly. I think it would be more efficient to just use a ConcurrentHashMap to store the class info data.

Hi Ruediger,

As you might have noticed, we are currently testing alternatives for serialization: our apps are based on Wicket which relies heavily on serialization and we need speed which leads us to Fast.

We fixed a few things here and there but we are overall very happy with the results so far.

There is a nice project here: https://github.com/wicketstuff/core/tree/master/jdk-1.7-parent/serializer-fast using fast as a serializer. It uses a pretty old version of Fast and we are working on improving this: upgrade the version to the latest 1.x here and provide a fast2 serializer.

The people at Wicketstuff did a very interesting work which allows to inspect the serialization content using the FSTSerialisationListener: it gives very interesting insights about the stuff Wicket is serializing and generates very useful reports (text, D3js...) - the code for this part is here: https://github.com/wicketstuff/core/tree/master/jdk-1.7-parent/serializer-common.

It uses the FSTSerialisationListener machinery you removed in the latest versions of Fast (in 1.x and 2.x) and it would be really great if we could reintroduce this feature as it is used by external libraries.

The methods you introduced in FSTObjectOutput aren't very convenient for external debugging of serialization as it's a bit difficult to override them: our FSTObjectOutput is built in FSTConfiguration and I'm not very fond of overriding all this machinery.

We are willing to do the work in 1.x and 2.x if you confirm you would include such a patch in the official codebase (the patch will mostly be a revert of your previous commit removing this stuff). I don't think the overhead will be high and it's a very useful feature.

Thanks for your feedback.

Here is cooked up test, please not that in readExternal bytes2 should be all "2" and bytes1 should be all "1". However I get bytes2 as all "1".

public static class TestArray extends ArrayList<Integer> implements Externalizable {
    @Override
    public void writeExternal(ObjectOutput out) throws IOException {
        out.writeInt(2);
        out.writeInt(548);
        out.writeInt(348);
        byte[] bytes = new byte[1024];
        for (int i = 0; i < 548; i++) {
            bytes[i] = 1;
        }
        out.write(bytes, 0, 548);
        bytes = new byte[1024];
        for (int i = 0; i < 348; i++) {
            bytes[i] = 2;
        }
        out.write(bytes, 0, 348);
    }

    @Override
    public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
        int v1 = in.readInt();
        int v2 = in.readInt();
        int v3 = in.readInt();
        byte[] bytes1 = new byte[v2];
        in.readFully(bytes1);
        byte[] bytes2 = new byte[v3];
        in.readFully(bytes2);
        int v4 = 0;
    }
}

Calling method:
private Object fastRoundTrip(Object list)
throws IOException, ClassNotFoundException {
ByteArrayOutputStream os = new ByteArrayOutputStream();
FSTObjectOutput objOut = new FSTObjectOutput(os);
objOut.writeObject(list);
objOut.close();

    ByteArrayInputStream is = new ByteArrayInputStream(os.toByteArray());
    FSTObjectInput objIn = new FSTObjectInput(is);
    objIn.setReadExternalReadAHead(16000);
    return objIn.readObject();
}