flightstats / hub Goto Github PK

4 items are missing from dev:
http://ops-jenkins01.cloud-east.dev/job/hub-v2-dev-replication-verifier/1170/console

'http://hub/channel/syslog_web_legacy_pdx/2015/05/07/11/32/03/288/NECZjE',
'http://hub/channel/syslog_web_legacy_pdx/2015/05/07/11/32/05/387/HtPn3U',
'http://hub/channel/syslog_web_legacy_pdx/2015/05/07/11/32/07/657/sTm5nV',
'http://hub/channel/syslog_web_legacy_pdx/2015/05/07/11/32/09/911/SybIsp',

The second one has the same data as the first

http://hub-v2.svc.prod/channel/escalationAlerts/2015/05/14/14/53/40/610/V8uaYx
http://hub-v2.svc.prod/channel/escalationAlerts/2015/05/14/15/38/56/066/2N4FbT

Channel is replicated from prod into staging, and the group callback is from staging.

Aashish says they aren't using them, and aren't required now.

When sending a POST request to a channel that has a replication source, the response "cannot modified while replicating" was sent as plain text when JSON was requested and expected.

This issue seems nitpicky but our hub client's HTTP library was very unhappy.

In a replicated channel, @chriskessel has found the following
1 - call /next/10 near the head of the channel
2 - record the last item
3 - call /latest
4 - latest returns an item previous to step 2

This happens infrequently, ~once in 50 attempts.

https://flightstats.pagerduty.com/incidents/PRVG1R5

"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/28?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/26?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/27?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/29?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/28?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/30?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/29?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/31?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/30?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/32?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/31?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/33?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/34?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/32?stable=true",
"http://hub-v2.svc.prod/channel/aav2_obfuscatedItineraries/2015/05/14/13/33?stable=true",

For two different groups in the encrypted hub, the same patten:

INFO 2015-06-23 19:10:31,730 [LeaderRotatorService RUNNING] com.flightstats.hub.cluster.CuratorLeader [line 88] - abdicating leadership for /GroupLeader/aav2_tdmItineraryToLegacyBridge

did not follow the expected pattern:

INFO 2015-06-23 19:10:31,730 [LeaderRotatorService RUNNING] com.flightstats.hub.cluster.CuratorLeader [line 88] - abdicating leadership for /GroupLeader/zomboEscalationAlerts
INFO 2015-06-23 19:10:35,242 [Curator-LeaderSelector-8] com.flightstats.hub.group.GroupCaller [line 117] - stopping last completed at 2015/05/13/23/56/55/254/EbKKmf zomboEscalationAlerts
INFO 2015-06-23 19:10:35,243 [Curator-LeaderSelector-8] com.flightstats.hub.cluster.CuratorLeader [line 70] - lost leadership /GroupLeader/zomboEscalationAlerts

It is possible to incorrectly return an empty lastCompleted

Aashish wants to modify parallel calls to tune performance w/out having to recreate the group callback.

For example, if you pass in 99 for the month you'll get an HTTP 500 with a JodaTime error message.

Request:

curl http://hub-v2.svc.dev/channel/zanderSandbox/2015/99/01/12/30/00/000/a

Response:

HTTP ERROR: 500

Problem accessing /channel/zanderSandbox/2015/99/01/12/30/00/000/a. Reason:

    org.joda.time.IllegalFieldValueException: Value 99 for monthOfYear must be in the range [1,12]
Powered by Jetty://

I would expect a 400 for this kind of error.

Lucas was having an issue with this:

http://hub-v2.svc.staging/channel/kratzen_lgw_last/2015/04/16/23/49/13/164/TrNHM6/previous/7
would hang
but
http://hub-v2.svc.staging/channel/kratzen_lgw_last/2015/04/16/23/49/13/164/TrNHM6/previous/6
would return quickly

This appears possible when leadership changes before alert status is changed.
http://ops-jenkins01.cloud-east.dev/job/hub-v2-dev-alert-verifier/25/console

/status & /time both return results for missing channels

This only seems applicable to legacyScraperRaw

When a group callback has many parallel processes, a spike in data can cause a large spike in CPU usage. This can adversely impact other traffic to the hub.

Mitigation:
(non - coding solutions)

• Increase hub node size to accommodate the spikes - more AWS spend for normal traffic.
• Reduce the number of parallel callbacks manually (conversations with callback owners)

(dev solutions)
Callback only Hub node - in addition to the traditional 3 - so callbacks won't affect other traffic
Spin up node as needed (might be SciFi)
Improve performance of hub processes

This will rely on the NAS for redundancy and compression.
Create a new ChannelService implementation which uses a single volume, and does not use S3 or Dynamo for items, channel metadata or group metadata.
This should be able to reuse most of FileSpokeStore for actual storage on disk.

Trey would like to have a self contained version of the hub which does not use S3 or Dynamo to run inside a Docker container for testing.

Request from @aashishpatel
Allow creation of group callbacks which go backwards from a starting point to an optional end point
After a complete traversal, the group callback should delete itself

Advantages over client implementing:
Load can be more easily distributed
Client does not need to track cursor location

This should allow users to easily support longer cache times and arbitrary values

WARN 2015-03-09 17:59:40,054 [pool-6-thread-1] com.flightstats.hub.spoke.RemoteSpokeStore [line 111] - unable to get content aav2_pfTdmItineraries/2015/03/09/17/58/59/991/a2sLK8
com.sun.jersey.api.client.ClientHandlerException: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching encrypted-hub-v2-03.cloud-east.staging found.
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:155) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.api.client.WebResource.get(WebResource.java:193) ~[jersey-client-1.19.jar:1.19]
at com.flightstats.hub.spoke.RemoteSpokeStore.read(RemoteSpokeStore.java:102) ~[hub-2015-02-17.29.jar:na]
at com.flightstats.hub.spoke.SpokeContentDao.read(SpokeContentDao.java:68) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue.writeContent(S3WriteQueue.java:75) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue.access$100(S3WriteQueue.java:23) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue$2.call(S3WriteQueue.java:62) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue$2.call(S3WriteQueue.java:59) [hub-2015-02-17.29.jar:na]
at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) [guava-retrying-1.0.6.jar:na]
at com.github.rholder.retry.Retryer.call(Retryer.java:110) [guava-retrying-1.0.6.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue.write(S3WriteQueue.java:59) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue.access$000(S3WriteQueue.java:23) [hub-2015-02-17.29.jar:na]
at com.flightstats.hub.dao.s3.S3WriteQueue$1.call(S3WriteQueue.java:50) [hub-2015-02-17.29.jar:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_31]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching encrypted-hub-v2-03.cloud-east.staging found.
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1937) ~[na:1.8.0_31]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[na:1.8.0_31]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_31]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1478) ~[na:1.8.0_31]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:212) ~[na:1.8.0_31]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:957) ~[na:1.8.0_31]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:892) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1050) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1391) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1375) ~[na:1.8.0_31]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563) ~[na:1.8.0_31]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[na:1.8.0_31]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512) ~[na:1.8.0_31]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) ~[na:1.8.0_31]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_31]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_31]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:253) ~[jersey-client-1.19.jar:1.19]
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153) ~[jersey-client-1.19.jar:1.19]
... 18 common frames omitted
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching encrypted-hub-v2-03.cloud-east.staging found.

@aashishpatel I recreated my group callback starting a the first item and noticed that the errors are still associated with the new group(i used the same name).
https://enc-v2-prod.hub.flightstats.com/group/aav2_tdmObserverProfileToLegacyBridge

This appears to have a race condition between when the underlying group callback is created, and when the first item is posted.

From Lucas:

$ nc hub-v2.svc.staging 80
GET /channel/kratzen_ams/2015/03/19/03/47/55/655/ujs8Nb HTTP/1.1
User-Agent: Java/1.7.0_71
Host: hub-v2.svc.staging
Accept: text/html, image/gif, image/jpeg, ; q=.2, */; q=.2

Stack trace from logs:
java.lang.IllegalArgumentException: java.text.ParseException: Expected separator '/' instead of ';'
at com.sun.jersey.core.header.MediaTypes.createMediaTypes(MediaTypes.java:234) ~[jersey-bundle-1.19.jar:1.19]
at com.flightstats.hub.channel.ChannelContentResource.contentTypeIsNotCompatible(ChannelContentResource.java:324) ~[hub-2015-03-18.7.jar:na]

at com.flightstats.hub.channel.ChannelContentResource.getValue(ChannelContentResource.java:186) ~[hub-2015-03-18.7.jar:na]

Caused by: java.text.ParseException: Expected separator '/' instead of ';'
at com.sun.jersey.core.header.reader.HttpHeaderReader.nextSeparator(HttpHeaderReader.java:121) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.impl.provider.header.MediaTypeProvider.valueOf(MediaTypeProvider.java:90) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.reader.HttpHeaderReader$2.create(HttpHeaderReader.java:320) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.reader.HttpHeaderReader$2.create(HttpHeaderReader.java:318) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.reader.HttpHeaderReader.readList(HttpHeaderReader.java:481) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.reader.HttpHeaderReader.readMediaTypes(HttpHeaderReader.java:325) ~[jersey-bundle-1.19.jar:1.19]
at com.sun.jersey.core.header.MediaTypes.createMediaTypes(MediaTypes.java:228) ~[jersey-bundle-1.19.jar:1.19]

requested by @crozierm

The hub currently support channel alerting when data stop being written for a given period.

It would be awesome if we could do something similar for group callbacks to make sure they are keeping up with the latest item on a channel.

As /previous goes backwards in time, if the TTL is greater than 21 years, links are not returned.

Create a channel with Body works as expected but:

curl -i -X PUT http://hub/channel/testBen2

returns

HTTP/1.1 500 Server Error
Date: Tue, 24 Feb 2015 20:28:35 GMT
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html; charset=ISO-8859-1
Content-Length: 315
Server: Jetty(9.2.6.v20141205)

HTTP ERROR: 500

Problem accessing /channel/testBen2. Reason:

    java.lang.NullPointerException

This is only for query apis, not for group callback or websocket

Start with a simple daily period when an alert should not be sent.

dev integration tests are showing that earliest/3 sometimes returns only two items

Mostly, would love some docs on how to set up the back end, etc.

Thanks!

/spoke/latest/FlexFlightHistoryReplicatedHubV1Staging/2015/03/17/17/36/49/000/ZZZZZ
returns
FlexFlightHistoryReplicatedHubV1Staging/2015/03/17/17/31/43/691/2905200

/spoke/latest/FlexFlightHistoryReplicatedHubV1Staging/2015/03/17/17/37/15/000/ZZZZZ
returns
FlexFlightHistoryReplicatedHubV1Staging/2015/03/17/17/31/13/686/2905180

I would love to use the hub to store data permanently. One missing feature is the ability to delete content from the hub.

When querying the forward end of a channel:
hub/channel/{name}/{key path}/next/10
If the hub returns zero items, it should still have a previous link.

The reverse is true for the back end of the channel.

http://hub-v2.svc.staging/channel/ProcessedFlightHistory/2015/04/13/16/45/49/118/Lb1a8U/next/10?trace=true

If a CallbackQueue has an error, it does not cause the GroupCaller to abdicate leadership.

currently this returns a 500 with an NPE
INFO 2015-03-10 21:00:04,082 [qtp1969238242-80136] com.flightstats.hub.spoke.SpokeResource [line 132] - NPE - unable to get latest Station 2015/03/10/20/59/59/000/ZZZZZ

should return a 404

when resizing an instance, the cluster's delta time can swing dramatically.
add a 50% margin to the adjustment, which will have minimal impact during normal operations, and a wider buffer during time synchronization.