Occasional freeze at "Sent tests to slave" about pitest

[hcoles]

From [email protected] on June 09, 2012 14:52:33

The command for running PIT in the above mentioned project is "mvn clean verify -Pcoverage-report"

[hcoles]

From [email protected] on June 13, 2012 04:34:50

I'm a little confused by this one. The logs show that the parents have successfully written data to the socket and flushed, however the slave process is sat there waiting on it without receiving.

I can add timeouts, which will substitute a hang for an error, but the root cause will remain.

How often are you seeing this? I've not been able to reproduce.

[hcoles]

From [email protected] on June 13, 2012 13:38:51

This happens to me maybe every couple of builds. I was able to find out the reason for the freeze, though not yet the original source of the problem.

In org.pitest.util.SafeDataInputStream#readString the readInt() method returns sometimes a huge value. I've attached a log (debug printing.txt) where it returned 3342336 instead of 69. When I did a heap dump on the slave during that freeze, I found out that in org.pitest.util.SafeDataInputStream#readString the byte[] data array contained the following bytes: 0 69 102 105 47 106 117 109 105 47 and so on. In other words, the bytes 0 and 69, followed by the string "fi/jumi/" and so on. Since readInt() reads 4 bytes, we can conclude that the stream had 2 additional bytes. 3342336 in hex is 0x00330000, so those two additional were 0x00 and 0x33 (51 in decimal).

I modified PIT 0.27 to print the data like this:

org.pitest.mutationtest.CoverageCommunicationThread.SendData#sendTests

private void sendTests(final SafeDataOutputStream dos) {
  // send individually to reduce memory overhead of deserializing large
  // suite
  System.out.println("CoverageCommunicationThread$SendData.sendTests");
  System.out.println("testClasses.size() = " + testClasses.size());
  dos.writeInt(this.testClasses.size());
  for (final String tc : this.testClasses) {
    System.out.println("tc = " + tc);
    dos.writeString(tc);
  }
  dos.flush();
  LOG.info("Sent tests to slave");
}

org.pitest.util.SafeDataInputStream#readString

public String readString() {
System.err.println("SafeDataInputStream.readString");
try {
final int length = this.dis.readInt();
System.err.println("length = " + length);
final byte[] data = new byte[length];
this.dis.readFully(data);
String result = new String(data, "UTF-8");
System.err.println("result = " + result);
return result;
} catch (final IOException e) {
throw Unchecked.translateCheckedException(e);
}
}

org.pitest.coverage.execute.CoverageSlave#receiveTestClassesFromParent

private static List receiveTestClassesFromParent(
final SafeDataInputStream dis) {
System.err.println("CoverageSlave.receiveTestClassesFromParent");
final int count = dis.readInt();
System.err.println("count = " + count);
final List classes = new ArrayList(count);
for (int i = 0; i != count; i++) {
String className = dis.readString();
System.err.println("className = " + className);
classes.add(new ClassName(className));
}
LOG.fine("Receiving " + count + " tests classes from parent");
return classes;
}

Attachment: debug printing.txt

[hcoles]

From [email protected] on June 13, 2012 13:57:35

Actually I found another occurrence of faulty data in the byte array (since it read all the data which followed in the stream). In one occasion instead of encoding the string length using 4 bytes, it only uses 3 bytes. The byte sequence is like this: 99 107 101 114 84 101 115 116 0 0 57 102 105 47 106. In ASCII: "ckerTest" 0 0 57 "fi/j", which matches the end of "fi/jumi/threadsafetyagent/ThreadSafetyCheckerTest" and the beginning of "fi/jumi/threadsafetyagent/util/ClassFileTransformerTest$1".

There is definitely something wrong in the data streams. Do you have ideas?

[hcoles]

From [email protected] on June 13, 2012 14:28:14

Another occurrence, this time with the first test class that CoverageCommunicationThread$SendData.sendTests sent:

SafeDataOutputStream.writeString wrote a string of length 32 (0x00000020) but SafeDataInputStream.readString received a length 536870912 (0x20000000). The data that it read after that starts 32 102 105 47 which is 32 "fi/". So instead of getting 0 0 0 32 102 105 47, it got 32 0 0 0 32 102 105 47. Now this is weird.

[hcoles]

From [email protected] on June 13, 2012 14:32:00

I'm starting to suspect JVM bugs. It seems that the slaves used Java 1.6.0_24, so I'll upgrade everything to the latest Java.

[hcoles]

From [email protected] on June 13, 2012 15:17:05

Upgraded to Java 1.7.0_05 - still happens: the slave got for length the bytes 0 0 instead of 0 0 0 66.

[hcoles]

From [email protected] on June 13, 2012 16:12:11

I'm starting to suspect that this might be caused by a web protection software which I'm using. I'll do some more experiments with and without it...

[hcoles]

From [email protected] on June 14, 2012 04:18:21

As the behavior seems to be non deterministic, this suggests the cause is either environmental or a thread safety/timing issue.

There ought to be only a single reader and writer in play, so threading issue doesn't seem likely, but can you try adding a synchronized blocks the the input and output streams and see if you are still able to reproduce?

[hcoles]

From [email protected] on June 16, 2012 09:41:13

I was able to locate the cause of the problem: K9 Web Protection. I uninstalled it and switched to using OpenDNS. The problem hasn't appeared since then - I've been running the build in a loop for over 10 hours without problems, whereas earlier the problem would have happened within 30 minutes.

This issue may now be closed.

[hcoles]

From [email protected] on June 17, 2012 13:48:24

closing as per esko's comment

Status: Invalid