Begin logging true origin as well as internal IPs for requests about mybinder.org-deploy

[minrk]

Now that we are having traffic issues, discarding this information is becoming a real impediment to diagnostics.

[yuvipanda]

@minrk do we do this now?

[betatim]

If we are (or not) we should add to the SRE guide what a good way of looking at this is. We do have the real IP in some of the logs on stackdriver. I think mostly in the CHP logs. The single user notebook servers only log internal IPs.

Right now I am asking myself "why is https://github.com/QuantStack/xeus-cling so popular since about 12h ago?" and struggling to find an answer that satisfies me. Seeing the referrer site would help as well as having the real IP in the pod logs.

[minrk]

I believe this is fixed everywhere but the singleuser logs, where we need to set trust_xheaders = True. This would be true by default for jupyterhub, but since we override it back to the vanilla notebook, it doesn't trust these by default.

[minrk]

I'd like to try addressing this with #760

[minrk]

Actually, no, we are already doing this. All our logs have the right ips, looks like.

[betatim]

When I look at the logs of a user pod I see lines like:

[I 10:25:12.753 NotebookApp] 302 GET /user/my-repo-here-6sl70jup/ (10.12.0.69) 0.84ms

from your comment above it sounds like the IP here should be the external one or is this one more thing we need to enable?

[minrk]

That's a request that is indeed coming from inside the cluster. It's JupyterHub's internal check that the server is up. Check the subsequent logs, or in general 302 messages which show real ips.

[betatim]

Aha! Thank you :)