Pagination is not working properly (not fetching every pages) about fastly-exporter

[peterbourgon]

Can you please provide more complete debug logs of version 7?

[peterbourgon]

@phamann Do you have an example of a multi-page response set that we could test against?

[AWKIF]

Thanks for quick reply, there's nothing really useful in logs just the same lines:

level=info filter=services type="name allowlist" expr=.*p22655.*
level=debug component=api.fastly.com service_id=XXXXXX service_name="XXXXXXXX" service_version=1 result=rejected reason="service name rejected by name filter"

then:

level=debug component=api.fastly.com refresh_took=731.848444ms total_service_count=106 accepted_service_count=0
level=info component=server listen=0.0.0.0:8080

i'm using allowlist so it doesn't subscribe, if i'm removing allowlist it proceed with the ones found (the 106 instead of 7,5K):

level=info component=api.fastly.com service=found service_id=XXX name=XXXX version=1
level=info component=rt.fastly.com service_id=XXXXX subscriber=create

[phamann]

Thanks for reporting this @AWKIF and sorry you're having issues with this recent change.

@peterbourgon here's a multi-page response, the main thing to note is the Link header value and structure.

Request:

curl -sv -X GET 'https://api.fastly.com/service?page=1&per_page=1' -H "Fastly-Key: <redacted>"

Response:

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 25953
Status: 200 OK
Content-Type: application/json
Cache-Control: no-store
Link: <https://api.fastly.com/service?page=3&per_page=1>; rel="last", <https://api.fastly.com/service?page=2&per_page=1>; rel="next"
Via: 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Date: Fri, 04 Mar 2022 12:14:16 GMT
Age: 55
X-Served-By: cache-lcy19267-LCY
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1646396057.822165,VS0,VE1
Vary: Accept-Encoding
Strict-Transport-Security: max-age=31536000


[
  {
    "name": "My test service",
    "deleted_at": null,
    "versions": [
      {
        "staging": false,
        "number": 1,
        "updated_at": "2021-09-20T16:06:04Z",
        "deployed": false,
        "locked": true,
        "testing": false,
        "active": false,
        "comment": "",
        "created_at": "2021-09-18T01:51:16Z",
        "service_id": "<redacted>",
        "deleted_at": null
      },
    ],
    "id": "<redacted>",
    "customer_id": "<redacted>",
    "version": 110,
    "created_at": "2021-09-18T01:51:16Z",
    "paused": false,
    "type": "wasm",
    "comment": "",
    "updated_at": "2021-09-18T01:51:16Z"
  }
]

Let me know if you need any further help to diagnose. Given @AWKIF's latest response I'm slightly suspicious if this is an issue relating to the combination of pagination and using an allowlist? But haven't dug into the code yet myself to confirm this.

[phamann]

Ok so it seems this is related to the method which we extract the next page URI value from the link header, in uriFromLink

Here is a playground reduced test case showing the wrong URI being returned for the next value from my response example above. I.e. its incorrectly returning the last page and not the next:
https://go.dev/play/p/x-KGLnAssWL