This repository contains the source code for the Cloud Foundry router. GoRouter is deployed by default with Cloud Foundry (cf-release) which includes routing-release as submodule.
Note: This repository should be imported as code.cloudfoundry.org/gorouter
.
You can find the old router here
The following instructions may help you get started with gorouter.
- Go should be installed and in the PATH
- GOPATH should be set as described in http://golang.org/doc/code.html
- gnatsd installed and in the PATH
- Install direnv
GoRouter is part of routing-release.
git clone https://github.com/cloudfoundry-incubator/routing-release
cd routing-release
./scripts/update
cd src/code.cloudfoundry.org/gorouter
Note: direnv will automatically set your GOPATH when you cd into the routing-release directory. You will need to run direnv allow
the first time.
We are using Ginkgo, to run tests.
Running scripts/test
will:
- Checks for Go
- Checks that GOPATH is set
- Installs gnatsd and ginkgo (or use the one already downloaded into the GOPATH)
- Runs all the tests with ginkgo (in random order, without benchmarks)
Any flags passed into scripts/test
will be passed into ginkgo.
# run all the tests
scripts/test
# run only tests whose names match Registry
scripts/test -focus=Registry
# run only the tests in the registry package
scripts/test registry
Building creates an executable in the gorouter/ dir:
go build
Installing creates an executable in the $GOPATH/bin dir:
go install
# Start NATS server in daemon mode
go get github.com/nats-io/gnatsd
gnatsd &
# Start gorouter
gorouter
When the gorouter starts, it sends a router.start
message. This message contains an interval that other components should then send router.register
on, minimumRegisterIntervalInSeconds
. It is recommended that clients should send router.register
messages on this interval. This minimumRegisterIntervalInSeconds
value is configured through the start_response_delay_interval
configuration property. GoRouter will prune routes that it considers to be stale based upon a seperate "staleness" value, droplet_stale_threshold
, which defaults to 120 seconds. GoRouter will check if routes have become stale on an interval defined by prune_stale_droplets_interval
, which defaults to 30 seconds. All of these values are represented in seconds and will always be integers.
The format of the router.start
message is as follows:
{
"id": "some-router-id",
"hosts": ["1.2.3.4"],
"minimumRegisterIntervalInSeconds": 20,
"prunteThresholdInSeconds": 120,
}
After a router.start
message is received by a client, the client should send router.register
messages. This ensures that the new router can update its routing table and synchronize with existing routers.
If a component comes online after the router, it must make a NATS request called router.greet
in order to determine the interval. The response to this message will be the same format as router.start
.
The format of the router.register
message is as follows:
{
"host": "127.0.0.1",
"port": 4567,
"uris": [
"my_first_url.vcap.me",
"my_second_url.vcap.me"
],
"tags": {
"another_key": "another_value",
"some_key": "some_value"
},
"app": "some_app_guid",
"stale_threshold_in_seconds": 120,
"private_instance_id": "some_app_instance_id"
}
stale_threshold_in_seconds
is the custom staleness threshold for the route being registered. If this value is not sent, it will default to the router's default staleness threshold.
app
is a unique identifier for an application that the endpoint is registered for. This value will be included in router access logs with the label app_id
, as well as being sent with requests to the endpoint in an HTTP header X-CF-ApplicationId
.
private_instance_id
is a unique identifier for an instance associated with the app identified by the app
field. Gorouter includes an HTTP header X-CF-InstanceId
set to this value with requests to the registered endpoint.
Such a message can be sent to both the router.register
subject to register
URIs, and to the router.unregister
subject to unregister URIs, respectively.
Create a simple app
$ nohup ruby -rsinatra -e 'get("/") { "Hello!" }' &
Send a register message
$ nats-pub 'router.register' '{"host":"127.0.0.1","port":4567,"uris":["my_first_url.vcap.me","my_second_url.vcap.me"],"tags":{"another_key":"another_value","some_key":"some_value"}}'
Published [router.register] : '{"host":"127.0.0.1","port":4567,"uris":["my_first_url.vcap.me","my_second_url.vcap.me"],"tags":{"another_key":"another_value","some_key":"some_value"}}'
See that it works!
$ curl my_first_url.vcap.me:8081
Hello!
To scale GoRouter horizontally for high-availability or throughput capacity, you must deploy it behind a highly-available load balancer (F5, AWS ELB, etc).
GoRouter has a health endpoint /health
on port 8080 that returns a 200 OK which indicates
the GoRouter instance is healthy; any other response indicates unhealthy.
This port can be configured via the router.status.port
property in the BOSH
deployment manifest or via the status.port
property under
/var/vcap/jobs/gorouter/config/gorouter.yml
$ curl -v http://10.0.32.15:8080/health
* Trying 10.0.32.15..
* Connected to 10.0.32.15 (10.0.32.15) port 8080 (#0)
> GET /health HTTP/1.1
> Host: 10.0.32.15:8080
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: private, max-age=0
< Expires: 0
< Date: Thu, 22 Sep 2016 00:13:54 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
<
ok
* Connection #0 to host 10.0.32.15 left intact
DEPRECATED:
Your load balancer can be configured to send an HTTP healthcheck on
port 80 with the User-Agent
HTTP header set to HTTP-Monitor/1.1
. A 200
response indicates the GoRouter instance is healthy; any other response
indicates unhealthy. GoRouter can be configured to accept alternate values for
the User Agent header using the healthcheck_user_agent
configuration
property; as an example, AWS ELBS send User-Agent: ELB-HealthChecker/1.0
.
$ curl -v -A "HTTP-Monitor/1.1" "http://10.0.32.15"
* Rebuilt URL to: http://10.0.32.15/
* Hostname was NOT found in DNS cache
* Trying 10.0.32.15...
* Connected to 10.0.32.15 (10.0.32.15) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: HTTP-Monitor/1.1
> Host: 10.0.32.15
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: private, max-age=0
< Expires: 0
< X-Vcap-Request-Id: 04ad84c6-43dd-4d20-7818-7c47595d9442
< Date: Thu, 07 Jan 2016 22:30:02 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
<
ok
* Connection #0 to host 10.0.32.15 left intact
DEPRECATED:
The /healthz
endpoint provides a similar response, but it always returns a 200
response regardless of whether or not the GoRouter instance is healthy.
The /routes
endpoint returns the entire routing table as JSON. This endpoint requires basic authentication and is served on port 8080. Each route has an associated array of host:port entries.
$ curl "http://someuser:[email protected]:8080/routes"
{"0295dd314aaf582f201e655cbd74ade5.cloudfoundry.me":["127.0.0.1:34567"],"03e316d6aa375d1dc1153700da5f1798.cloudfoundry.me":["127.0.0.1:34568"]}
Because of the nature of the data present in /varz
and /routes
, they require http basic authentication credentials. These credentials can be found the BOSH manifest under the router
job:
properties:
router:
status:
password: zed292_bevesselled
port:
user: paronymy61-polaric
Or on the Gorouter VM under /var/vcap/jobs/gorouter/config/gorouter.yml
:
status:
port: 8080
user: some_user
pass: some_password
The /varz
endpoint provides status and metrics. This endpoint requires basic authentication.
$ curl "http://someuser:[email protected]:8080/varz"
{"bad_gateways":0,"bad_requests":20,"cpu":0,"credentials":["user","pass"],"droplets":26,"host":"10.0.32.15:8080","index":0,"latency":{"50":0.001418144,"75":0.00180639025,"90":0.0070607187,"95":0.009561058849999996,"99":0.01523927838000001,"samples":1,"value":5e-07},"log_counts":{"info":9,"warn":40},"mem":19672,"ms_since_last_registry_update":1547,"num_cores":2,"rate":[1.1361328993362565,1.1344545494448148,1.1365784133171992],"requests":13832,"requests_per_sec":1.1361328993362565,"responses_2xx":13814,"responses_3xx":0,"responses_4xx":9,"responses_5xx":0,"responses_xxx":0,"start":"2016-01-07 19:04:40 +0000","tags":{"component":{"CloudController":{"latency":{"50":0.009015199,"75":0.0107408015,"90":0.015104917100000005,"95":0.01916497394999999,"99":0.034486261410000024,"samples":1,"value":5e-07},"rate":[0.13613289933245148,0.13433569936308343,0.13565885617276216],"requests":1686,"responses_2xx":1684,"responses_3xx":0,"responses_4xx":2,"responses_5xx":0,"responses_xxx":0},"HM9K":{"latency":{"50":0.0033354,"75":0.00751815875,"90":0.011916812100000005,"95":0.013760064,"99":0.013760064,"samples":1,"value":5e-07},"rate":[1.6850238803894876e-12,5.816129919395257e-05,0.00045864309255845694],"requests":12,"responses_2xx":6,"responses_3xx":0,"responses_4xx":6,"responses_5xx":0,"responses_xxx":0},"dea-0":{"latency":{"50":0.001354994,"75":0.001642107,"90":0.0020699939000000003,"95":0.0025553900499999996,"99":0.003677146940000006,"samples":1,"value":5e-07},"rate":[1.0000000000000013,1.0000000002571303,0.9999994853579043],"requests":12103,"responses_2xx":12103,"responses_3xx":0,"responses_4xx":0,"responses_5xx":0,"responses_xxx":0},"uaa":{"latency":{"50":0.038288465,"75":0.245610809,"90":0.2877324668,"95":0.311816554,"99":0.311816554,"samples":1,"value":5e-07},"rate":[8.425119401947438e-13,2.9080649596976205e-05,0.00022931374141467497],"requests":17,"responses_2xx":17,"responses_3xx":0,"responses_4xx":0,"responses_5xx":0,"responses_xxx":0}}},"top10_app_requests":[{"application_id":"063f95f9-492c-456f-b569-737f69c04899","rpm":60,"rps":1}],"type":"Router","uptime":"0d:3h:22m:31s","urls":21,"uuid":"0-c7fd7d76-f8d8-46b7-7a1c-7a59bcf7e286"}
The GoRouter runs the debugserver, which is a wrapper around the go pprof tool. In order to generate this profile, do the following:
# Establish a SSH tunnel to your server (not necessary if you can connect directly)
ssh -L localhost:8080:[INTERNAL_SERVER_IP]:17001 vcap@[BOSH_DIRECTOR]
# Run the profile tool.
go tool pprof http://localhost:8080/debug/pprof/profile
The GoRouter is, in simple terms, a reverse proxy that load balances between many backend instances. The default load balancing algorithm that GoRouter will use is a simple round-robin strategy. GoRouter will retry a request if the chosen backend does not accept the TCP connection.
Default load balancing algorithm that gorouter will use or may be explicity set in gorouter.yml
default_balancing_algorithm: round-robin
The GoRouter also supports least connection based routing and this can be enabled in gorouter.yml
default_balancing_algorithm: least-connection
Least connection based load balancing will select the endpoint with the least number of connections. If multiple endpoints match with the same number of least connections, it will select a random one within those least connections.
NOTE: GoRouter currently only supports changing the load balancing strategy at the gorouter level and does not yet support a finer-grained level such as route-level. Therefore changing the load balancing algorithm from the default (round-robin) should be proceeded with caution.
When terminating TLS in front of Gorouter with a component that does not support sending HTTP headers
If you terminate TLS in front of Gorouter, your component should send the X-Forwarded-Proto
HTTP header in order for applications and Cloud Foundry system components to correctly detect when the original request was encrypted. As an example, UAA will reject requests that do not include X-Forwarded-Proto: https
.
If your TLS-terminating component does not support sending HTTP headers, we recommend also terminating TLS at Gorouter. In this scenario you should only disable TLS at Gorouter if your TLS-terminating component rejects unencrypted requests and your private network is completely trusted. In this case, use the following property to inform applications and CF system components that requests are secure.
properties:
router:
force_forwarded_proto_https: true
If you terminate TLS in front of Gorouter, your component should also send the X-Forwarded-Proto
HTTP header in order for X-Forwarded-For
header to applications can detect the requestor's IP address.
If your TLS-terminating component does not support sending HTTP headers, you can use the PROXY protocol to send Gorouter the requestor's IP address.
If your TLS-terminating component supports the PROXY protocol, enable the PROXY protocol on Gorouter using the following cf-release manifest property:
properties:
router:
enable_proxy: true
You can test this feature manually:
echo -e "PROXY TCP4 1.2.3.4 [GOROUTER IP] 12345 [GOROUTER PORT]\r\nGET / HTTP/1.1\r\nHost: [APP URL]\r\n" | nc [GOROUTER IP] [GOROUTER PORT]
You should see in the access logs on the GoRouter that the X-Forwarded-For
header is 1.2.3.4
. You can read more about the PROXY Protocol here.
The GoRouter does not currently support proxying HTTP/2 connections, even over TLS. Connections made using HTTP/1.1, either by TLS or cleartext, will be proxied to backends over cleartext.
The router's logging is specified in its YAML configuration file. It supports the following log levels:
fatal
- A fatal error has occurred that makes gorouter unable to handle any requests. Examples: the router can't bind to its TCP port, a CF component has published invalid data to the router.error
- An unexpected error has occurred. Examples: the router failed to fetch token from UAA service.info
,debug
- An expected event has occurred. Examples: a new CF component was registered with the router, the router has begun to prune routes for stale droplets.
Access logs provide information for the following fields when recieving a request:
<Request Host> - [<Start Date>] "<Request Method> <Request URL> <Request Protocol>" <Status Code> <Bytes Received> <Bytes Sent> "<Referer>" "<User-Agent>" <Remote Address> x_forwarded_for:"<X-Forwarded-For>" x_forwarded_proto:"<X-Forwarded-Proto>" vcap_request_id:<X-Vcap-Request-ID> response_time:<Response Time> app_id:<Application ID> <Extra Headers>
- Status Code, Response Time, Application ID, and Extra Headers are all optional fields
- The absence of Status Code, Response Time or Application ID will result in a "-" in the corresponding field
Access logs are also redirected to syslog.
If an user wants to send requests to a specific app instance, the header X-CF-APP-INSTANCE
can be added to indicate the specific instance to be targeted. The format of the header value should be X-Cf-App-Instance: APP_GUID:APP_INDEX
. If the instance cannot be found or the format is wrong, a 404 status code is returned.
There is a separate docs folder which contains more advanced topics.
Refer doc to learn more troubleshooting slow requests.
Please read the contributors' guide