There is something wrong with the AntiEntropy. The setBit works fine from the pql path(SetBit via the query endpoint), but sometimes fails via the syncIndex route.
Sometimes....once every hour or so, the container array doesn't have any elements and this line will throw an out of bounds panic.
func (c *container) arrayWriteTo(w io.Writer) (n int64, err error) {
nn, err := w.Write((*[0xFFFFFFF]byte)(unsafe.Pointer(&c.array[0]))[:4*c.n])
}
I've put in some bounds checking and it prevents the panic but the problem still remains. Here is a stack trace of the bad path, i suspect something in syncBlock or MergeBlock
github.com/umbel/pilosa/roaring.(*container).arrayWriteTo(0xc441afbac0, 0x9f07c0, 0xc5d60254c0, 0x8, 0x0, 0x0)
/Users/tgruben/review/src/github.com/umbel/pilosa/roaring/roaring.go:1031 +0xc0
github.com/umbel/pilosa/roaring.(*container).WriteTo(0xc441afbac0, 0x9f07c0, 0xc5d60254c0, 0x8, 0x0, 0x0)
/Users/tgruben/review/src/github.com/umbel/pilosa/roaring/roaring.go:1006 +0x91
github.com/umbel/pilosa/roaring.(*Bitmap).WriteTo(0xc440c52d70, 0x9f07c0, 0xc5d60254c0, 0xc886cdb000, 0x1000, 0x1000)
/Users/tgruben/review/src/github.com/umbel/pilosa/roaring/roaring.go:461 +0x2c0
github.com/umbel/pilosa.(*Fragment).snapshot(0xc440c986e0, 0x0, 0x0)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:965 +0x3d8
github.com/umbel/pilosa.(*Fragment).incrementOpN(0xc440c986e0, 0xc442720b40, 0xc444cf7608)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:938 +0x4a
github.com/umbel/pilosa.(*Fragment).setBit(0xc440c986e0, 0x53f, 0x2c0669, 0x1, 0x21, 0xc5d1f5b200)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:363 +0x13f
github.com/umbel/pilosa.(*Fragment).MergeBlock(0xc440c986e0, 0xd, 0xc6dccd2330, 0x1, 0x1, 0xc9008c11a0, 0x2, 0x2, 0xc9008c1200, 0x2, ...)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:841 +0xf1e
github.com/umbel/pilosa.(*FragmentSyncer).syncBlock(0xc444cf7c40, 0xd, 0x2, 0xc6dca2f400)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:1371 +0x688
github.com/umbel/pilosa.(*FragmentSyncer).SyncFragment(0xc444cf7c40, 0xc42015ed33, 0x3)
/Users/tgruben/review/src/github.com/umbel/pilosa/fragment.go:1322 +0x714
github.com/umbel/pilosa.(*IndexSyncer).syncFragment(0xc444cf7eb8, 0xc42015ed33, 0x3, 0xc440c4d637, 0x1, 0x2, 0x1, 0x0)
/Users/tgruben/review/src/github.com/umbel/pilosa/index.go:469 +0xfa
github.com/umbel/pilosa.(*IndexSyncer).SyncIndex(0xc444cf7eb8, 0x83782a, 0x14)
/Users/tgruben/review/src/github.com/umbel/pilosa/index.go:354 +0x32a
github.com/umbel/pilosa.(*Server).monitorAntiEntropy(0xc42015a070)
/Users/tgruben/review/src/github.com/umbel/pilosa/server.go:173 +0x410
github.com/umbel/pilosa.(*Server).Open.func2(0xc42015a070)
/Users/tgruben/review/src/github.com/umbel/pilosa/server.go:117 +0x57