tilezen / tapalcatl Goto Github PK

Currently it looks like we when we deploy we use the master branches of all our dependencies. We should look into pegging the versions.

Additionally, we can consider moving the binary generation to a continuous integration step, and put the resulting binary, or an rpm/deb file onto s3 and use that for deployments.

Related to #3, we'll want to be able to track whether tapalcatl was able to satisfy the request from s3 or not. If tapalcatl will continue to act as proxy for tileserver, it will need to expose this information. If not, then the fastly config should get updated accordingly.

Additionally, we should think about logging anything that might be useful to capture for metrics/analytics/alerting purposes. If it's something that we want to capture in redshift for longer term storage, it's easiest to stick the relevant information in a response header and have the fastly log parsing processes handle it.

We went back and forth on expvars, and I don't think we'll end up using it any longer. @zerebubuth can we remove it from the code base?

The idiomatic way to do commands in Go is $PROJECT_ROOT/cmd/$COMMAND_NAME. Instead of tapalcatl_server, we should move it to cmd/tapalcatl. The change in this repo is fairly straightforward, but would also need corresponding changes in the Chef recipes.

The currently fashionable way to do this is go dep. This means vendoring in the dependencies so that the tagged repo is self-contained (other than the Go compiler and runtime). This does annoyingly bloat the repo, but there doesn't seem to be much we can do about that.

For multiple storage configurations that would share the same health check, redundant health checks should be omitted. Health checks should only be performed for unique storage type / configuration combinations.

Related to #26

Thanks to @iandees for pointing this out yesterday.

At the moment, the healthcheck just checks that the service is up. Unfortunately, this means that when the service is working fine, but S3 isn't, the service reports an OK healthcheck while returning 5xx errors to all requests.

We could solve this in several ways, and a couple that we briefly discussed were:

1. Check a static file on S3 when healthcheck is called. This would mean we'd need to remember to put the static file in place, but assuming it's one per bucket then that would be a infrequent. Alternatively, we could use a different API operation, such as LIST or HEAD bucket which doesn't require a static file.
2. Passively check the rate of 5xx errors coming from S3 over the past minute(s) and if >80% of requests resulted in a 5xx, then fail the healthcheck. This seems like a good metric for failure, but passive monitoring means it'll never come back afterwards (unless we treat 0/0 as success, in which case it might ping-pong up and down when it should stay failed).

I was thinking about the second last night, but now it looks like a bad idea. What do you think, @iandees & @rmarianski?

The storage configuration "type" and "name" are conflated. These should be separated out so multiple storages can be configured with the same type.

At the moment, tapalcatl is responsible for reverse proxying to tileserver when s3 doesn't have the tile. Can we move this responsibility back to fastly? This should both simplify tapalcatl and leave the burden of keeping more connections open at fastly.

The error code string seems to have changed to NoSuchKey (no longer No Such Key).

The HTTP spec says:

All HTTP date/time stamps MUST be represented in Greenwich Mean Time (GMT), without exception. For the purposes of HTTP, GMT is exactly equal to UTC (Coordinated Universal Time). This is indicated in the first two formats by the inclusion of "GMT" as the three-letter abbreviation for time zone...

Tapalcatl is currently sending timestamps in RFC1123Z format, which does not exactly comply with the spec. Most servers and clients appear to be fine parsing it, but it causes a problem with Varnish, as the latter is very strict. This, in turn, causes Fastly to be unable to take advantage of "304 Not Modified" responses to If-Modified-Since conditional requests.

The Go http package provides a TimeFormat constant and says:

TimeFormat is the time format to use when generating times in HTTP headers. It is like time.RFC1123 but hard-codes GMT as the time zone. The time being formatted must be in UTC for Format to generate the correct format.

We should use the http.TimeFormat for generating times, and http.ParseTime for parsing them, to make sure we're standards-compliant and Varnish is happy.

A buffer pool should be used as space for reading the storage zip data. To me it looks like the sized buffer pool implementation is the one we should be using. Best practice for the buffer pool implementation suggests using the 95%. We should also be logging this size.

The value of alloc should seek to provide a buffer that is representative of
most data written to the the buffer (i.e. 95th percentile) without being
overly large (which will increase static memory consumption). You may wish to
track the capacity of your last N buffers (i.e. using an []int) prior to
returning them to the pool as input into calculating a suitable alloc value.

Set up continuous integration to automatically run the tests.

Json messages with newlines aren't being formatted properly. We're currently formatting after dumping the json, we should interpolate the message first and then encode to json to ensure everything is escaped properly.

Can we remove the need to send requests to tapalcatl for pre 1.0 urls? Those won't have zip files on s3 anyway. Is it possible to just have the fastly talk to s3 directly for those requests, and tapalcatl for v1+ urls?

Related to #3

There are some bits of configuration that are more global in nature, but are now duplicated across each request pattern, eg the region. Let's consider decoupling this a little bit.

this, as opposed to tapalcatl-py doesn't seem to have documentation on how to provide configurations

I see from the code that that I can provide a config file, but what should its structure be?

The tilejson requests were originally handled by dispatching from fastly, but once the prefix / bucket logic moved to tapalcatl nothing was handling it any more. Since tapalcatl is aware of the prefix / bucket mapping, it makes sense to have it be responsible for serving the tilejson files.

Looks like CloudFront supports serving compressed MVTs now.

Which means we can set the MIME content-type to 'application/vnd.mapbox-vector-tile' to place nice with wider range of tooling.