Optimize notice framework cluster communication about zeek

[rsmmr]

Not a blocker, but would be a substantial improvement.

[0xxon]

I might take a stab at this.

After looking at the code for a bit - I think there are two potential ways to solve this.

The first one was already noted by Jon - moving notice processing to workers. I am currently tempted to do this one - however this, in theory, could make it more difficult for users to write notice policies (they will only have local state available for the policies).

The second solution would be to change the datastructures that are being sent over the network - so that they are smaller and do not by default contain the full connection record, etc. This obviously comes with the disadvantage that users won't have as much state available in their policies).

Both approaches break compatibility a bit - at the moment I think that the first one (do the work mostly on the workers) is the more appealing one.

[rsmmr]

I agree that the 1st one seems more appealing; it sounds conceptually right, as it's in line with how other stuff works in the cluster (like log processing). It does raise a major backwards compatibility concern, though; it's just sufficiently convenient to not have to consider cluster issues for notice processing that I'm sure it's been relied on out there somewhere. Not quite sure what to do about it other than warn in NEWS.

[0xxon]

I tried this and it actually turned out to not rally be very invasive to just move this over to the workers.

Code in branch topic/johanna/gh-214-notice-on-workers or in pull-request #440

One difference is that there now is a chance that notices will not be suppressed immediately - they only will be suppressed once the suppression notice has been distributed over the cluster. I don't really think that is a huge problem though.