Web Server Log Delivery Setup IP: missing IP blocks (& user agent) about edgio-docs

[jsernaque]

What are the IP blocks that should be whitelisted?

Thanks for pointing this issue out. We will update the docs shortly to indicate that RTLD requests originate from the following IP block: 198.7.21.0/24

Also, is there a specific user agent used by the service that performs the post? If so, could it be identified? (NOTE: The user-agent appears to currently be rtld-6.5.)

RTLD's user agent uses the following syntax:
rtld-{VERSION}

Although it is currently rtld-6.5, this version may be incremented at any time.

There's no "test" feature so it's difficult to determine if it's configured correctly through Edgio or not. (We use SendGrid and they have a button that initiates a test webhook post for test/debug purposes.) If I check out "Edgio Insights", there are many "400" errors logged, but there's no record of these attempts and/or errors in our logs.

We have passed this feedback to the devs.

Edge Insights

Edge Insights allows you to view near real-time traffic. By default, RTLD log delivery requests are not considered CDN traffic and therefore are not reported through Edge Insights.

Is it critical that the endpoint support HTTP/2? This isn't mentioned in the documentation. (I don't believe I'm able to test HTTP/2.0 using VSCode REST Client.)

No. It is not a requirement. It can deliver logs over HTTP/1.1 as well.

[jsernaque]

The fix for this issue has been merged.

If you are still experiencing log delivery issues, I recommend reaching out to our support.

https://edg.io/contact-support/

[JamoCA]

There's no "test" feature so it's difficult to determine if it's configured correctly through Edgio or not. (We use SendGrid and they have a button that initiates a test webhook post for test/debug purposes.) If I check out "Edgio Insights", there are many "400" errors logged, but there's no record of these attempts and/or errors in our logs.

I'm able to post the following to our endpoint using VSCode Rest Client and it works.:

NOTE: The json payload (below) is copied from the posted sample log data.

POST https://my-edgio-protected-host.com/rtldWAFendpoint HTTP/1.1
Content-Type: application/json
Authorization: Token ThisIsATestToken
User-Agent: rtld-6.5

{"agent_id":"1234500008619D55A","seq_num":0,"service":"waf","account_number":"0001","profile_id":0,"datestamp":"20201008","logs":[{"timestamp":1602200337.177535713,"user_agent":"curl/7.64.1","url":"https://cdn.example.com/","client_ip":"190.220.230.2","referer":"","host":"cdn.example.com","uuid":"38046679731278771327748811544613832704","client_country_code":"US","waf_profile_name":"Site 1","waf_profile_type":"PRODUCTION","waf_instance_name":"Site 1 Instance","sub_events_count":1,"sub_events":[{"total_anomaly_score":0,"matched_on":"REQUEST_METHOD","matched_value":"POST","rule_id":80009,"rule_message":"Method is not allowed by policy"}],"rule_tags":[],"rule_message":"Method is not allowed by policy","action_type":"BLOCK_REQUEST","server_port":443,"client_country":"United States","client_city":"Los Angeles"},{"timestamp":1602200338.598465258,"user_agent":"curl/7.64.1","url":"https://cdn.example.com/","client_ip":"230.180.240.23","referer":"","host":"cdn.example.com","uuid":"38046679731278771327748811544613832998","client_country_code":"US","waf_profile_name":"Site 1","waf_profile_type":"PRODUCTION","waf_instance_name":"Site 1 Instance","sub_events_count":1,"sub_events":[{"total_anomaly_score":0,"matched_on":"REQUEST_METHOD","matched_value":"POST","rule_id":80009,"rule_message":"Method is not allowed by policy"}],"rule_tags":[],"rule_message":"Method is not allowed by policy","action_type":"BLOCK_REQUEST","server_port":443,"client_country":"United States","client_city":"Los Angeles"}]}

[JamoCA]

I noticed that the HTTP post (according to "Edgio Insights" is logged as virt_request_protocol: HTTP_2_0.

Is it critical that the endpoint support HTTP/2? This isn't mentioned in the documentation. (I don't believe I'm able to test HTTP/2.0 using VSCode REST Client.)

[JamoCA]

I discovered the error after creating a VSCode REST Code sample after GZIPping the content. The error message returned from our server is HTTP/1.1 400 Input JSON is not well formed and I can now troubleshoot it further.

POST ttps://my-edgio-protected-host.com/rtldWAFendpoint HTTP/2.0
Content-Type: application/json;charset=UTF-8
Content-Encoding: gzip
Authorization: Token ThisIsATestToken
User-Agent: rtld-6.5

< c:\rtldWAF-payload.json

[jsernaque]

Also, you can check log delivery performance for a specific RTLD profile from within the Edgio Console.

Learn more at:
https://docs.edg.io/guides/v7/logs/rtld/log_data_verification#log-performance-statistics

[JamoCA]

After further review, my sample payload (now gzipped) works when using VSCode REST Client to post from internal (VPN) or external IP routed through Edgio property.

Yet the RTLD donut graph is a "100% red 400 fail" and RTLD posts are still not posting. (Confirmed by monitoring Edgio Insights.) I've disabled the logging for now and will re-review when I'm fresh.