facebookarchive / jcommon

concurrency, collections, stats/analytics, config, testing, etc

jcommon

jcommon: a set of libraries used inside Facebook java projects, internal and open source.

Requirements

jcommon builds and is tested on both Java 8 and Java 11.

Building jcommon

Maven pom.xml files are provided as a convenience. However, we do not regularly test them, and do not consider them supported.

Additional information

We do not guarantee API or ABI stability. We do not tag specific versions, and no longer publish artifacts to Maven Central.

Pull requests are permitted, but not regularly reviewed.

License

jcommon is provided under the Apache License version 2.0, as found in the LICENSE file.

jcommon's Issues

add "levels" to stats code

Working on, diff almost complete

add reset method to counters

add reset method for counters

A Bug in HyperLogLog implemetation

Hey, Guys
I find a serialization bug in HyperLogLog, I can not deserialize correctly from the byte array that I had serialized in before, after I checkout the HyperLogLog code, I find you forgot to calculate the nonZeroBucket. The code below is the test code. I am trying fix this bug, if you think I am right, please merge my PR, Thanks
`package me.line.hiro.filterAndAggregator;

import me.line.hiro.common.algorithm.HyperLogLog;

import org.junit.Test;

public class HyperLogLogTest {

@Test
public void testSeriaizer() {
    HyperLogLog hll = new HyperLogLog(16);
    for (int i = 0; i < 10000; i++) {
        hll.add(i);
    }
    System.out.println("original eatiminate: " + hll.estimate());
    System.out.println("original data:");
    opBucket(hll);
    System.out.println("currentSum: " + hll.getCurrentSum() + " nonZeroBcuket: " + hll.getNonZeroBuckets());

    System.out.println();
    HyperLogLog anothHll = new HyperLogLog(hll.buckets());
    System.out.println("deSerializer estiminate: " + anothHll.estimate());
    System.out.println("DeSerializer data : ");
    opBucket(anothHll);
    System.out.println("currentSum: " + anothHll.getCurrentSum() + " nonZeroBcuket: "
            + anothHll.getNonZeroBuckets());
}

private void opBucket(HyperLogLog hll) {
    int[] bucket = hll.buckets();
    int len = Math.min(20, bucket.length);
    for (int i = 0; i < len; i++) {
        System.out.print(bucket[i] + " ");
    }
    System.out.println();
}

}
`

`original eatiminate: 8975
original data:
9 10 9 13 9 12 10 9 10 9 9 12 15 10 9 10
currentSum: 0.019195556640625 nonZeroBcuket: 16

deSerializer estiminate: 8975
DeSerializer data :
9 10 9 13 9 12 10 9 10 9 9 12 15 10 9 10
currentSum: 0.019195556640625 nonZeroBcuket: 0`

fix race condition in MultiWindow*

There is a case we could undercount and no doubt happens; but the race condition period is very small; when we roll the bucket, each counter may choose to merge the previous "current" by value rather than keeping a ref. We don't guard updates to it during this time.

solutions with a RW lock solve it trivially, but reduce throughput from 20M/s to 13M/s as measured by one core on an older dev box.

Notes.

1. I believe I have some optimistic concurrency/locking methods that may work with volatiles. While still slower, the RW-lock read-lock is expensive
2. timw has noted a new way to store the bucketed counters, as one time-series, with breakpoints at the 60, 600, 3600, alltime. meaning, we'd compose a 60 + 540 + 3000 + rest and use composite counters for each of these ranges

int this, the counter is merged into only one and optimistic techniques become easier

getValue() in OptionImpl class should be synchronised to match the synchronisation on its set counterpart

As per squid:S2886:

"When one part of a getter/setter pair is synchronized the other part should be too. Failure to synchronize both sides of a pair may result in inconsistent behavior at runtime as callers access an inconsistent method state."

The setValue() method in OptionImpl is synchronised, but the getValue() method is not.

getValue() should be synchronised.

pull topk approximation into stats

we should get the non-aggregation version in here.

pull in json utils to serialize to json

Any ETA on opensourcing Calligraphus?

This would be great since scribe is EOL'd :) :)

ConcurrentLinkedHashMap --> Caffeine when Java-8 based

There is an unused dependency on an older version of ConcurrentLinkedHashMap. If it is used internally, please consider upgrading to Caffeine when Java 8 based. CLHM is now end-of-life and maintained only for Java 6 users. Guava's cache is in a similar state as there is no one left familiar with the implementation and Guava must remain Android compatible (~JDK6). Since I am no longer involved with Guava I can't speak to that cache's future, other than say there is no current owner.

Caffeine is a Java 8 rewrite of Guava's cache with additional features and higher performance.

Exception throwing flavors of java.util.function.* classes are missing.

The ExtSupplier, ExtRunnable, and ExtCallable classes are extremely useful, but all of the other interfaces in java.util.function are missing.

Along with adding these exception throwing interfaces, it would be nice if these interfaces had static factory methods that could convert an exception throwing version to a standard java.util version.
The method would take an exception throwing version and return a "quiet" version that wraps any checked exceptions in a runtime exceptions (UncheckedCheckedException).

Link error for Media:Java--CodingStandards--IntelliJIdea11-settings.jar

The link in wiki under https://github.com/facebook/jcommon/wiki/Coding-Standards#settings

Media:Java--CodingStandards--IntelliJIdea11-settings.jar

is pointing to https://github.com/facebook/jcommon/wiki which is not a file. Looking forward to trying it out.

fix the gh-pages build

quote, timw:
"Just an FYI: the jenkins build for jcommon was unhappy. With Aaron's
help, we traced it down to changing the github URL ("@gh" -> "@github")
and something to do with the "gh-pages" branch. Rather than mess with
figuring out the "gh-pages" issue, I changed it to only build "master"
(i.e., changed "Git -> Branches to build" from "**" to "master").

Please let me know if that's not a good thing to do. :)
"

Dependency convergence error

During building with Maven:

[WARNING]
Dependency convergence error for joda-time:joda-time:2.9.4 paths to dependency are:
+-com.facebook.jcommon:util:0.1.29
+-joda-time:joda-time:2.9.4
and
+-com.facebook.jcommon:util:0.1.29
+-com.fasterxml.jackson.datatype:jackson-datatype-joda:2.7.4
+-joda-time:joda-time:2.4