RDAP server connection error handling (for discussion) about whoisit

[meeb]

I've pushed the overrides=True feature into a new 2.4.0 release which you can install or update to via pip now. I've also removed the .build override as you note that's no longer required. Feel free to run the list_public_tlds_with_rdap_endpoints.py tool and make a PR to add more override manual RDAP endpoints if you notice any into overrides.py. Cheers.

[meeb]

Thanks! Glad you find the library useful! And thanks also for the very detailed issue.

As you note this is more of a library design issue rather than a straight bug. The issue is with the bootstrapping information provided by the IANA so I suppose, technically, the current behavior is exactly correct (just not very useful).

There are a litany of HTTP errors that can occur so it's probably OK to just bubble up whatever requests raises, assuming developers who implement whoisit are almost certainly experienced with handing requests-style exceptions already. A bunch of the RDAP endpoints for smaller TLDs and RIRs have esoteric or out of date TLS requirements, for example, which can raise some interesting errors.

Most likely I don't want to manually maintain a specific list of work-arounds in the library as that sounds like a headache. You've probably already done the best thing by contacting people to get the IANA data updated which is the actual fix for this.

I could add an endpoint override if that would be useful? You could then implement your own work-arounds without them getting messy in the base library? Something like:

whoisit.domain('nic.build', rdap='https://rdap.centralnic.com/build/domain/')

or similar? This would just override the bootstrap-derived RDAP endpoint URL so you could manually select the correct upstream RDAP server when you want to query certain currently broken TLDs until the IANA get their data updated.

[case]

Thank you again for indulging my rambling odyssey down the domains rabbit hole here.

I agree that this feels more like a library functionality issue, and it sounds like there is a case to be made for adding the ability to supplement or override the IANA bootstrap data somehow.

Another example came to mind since you mentioned this: DENIC apparently runs an RDAP server, but doesn't list it with IANA for some reason:

https://rdap.denic.de/domain/apple.de

(This sort of highlights the subtle divide between the ccTLD world vs the ICANN + gTLD world.)

Thinking out loud: like you mentioned, this is similar to the allow_insecure_ssl issue. It'd be nice to configure these things centrally, prior to running the whoisit.domain() checks, rather than per-check.

(I'm a novice developer, so I probably lean on libraries like yours more than I should! E.g. the library is parsing out the TLD already). For example:

• For domains under de, always try https://rdap.denic.de/
• For domains under build, always use https://rdap.centralnic.com/build/
• For domains under work, always use allow_insecure_ssl=True

Speaking of .work, it's migrating to the GoDaddy Registry backend on 2022-01-25, so the allow_insecure_ssl workaround for it might not be needed post-migration. (though it will probably still be needed for other zones someday)

[meeb]

The above commits add a new feature which permits overriding the IANA bootstrap data specified RDAP endpoints. Usage is pretty easy, just add a override flag when bootstrapping such as bootstrap(override=True) or load_bootstrap_data(some_data, override=True). This overrides the specified RDAP endpoints for a TLD (or IP/ASN/entity if needed in the future) from a static set list specified here:

https://github.com/meeb/whoisit/blob/main/whoisit/overrides.py

Details on usage are also in the README.

The format should be pretty obvious. You're welcome to submit a PR to expand that file if you keep the date and comments up to date as well. The feature has tests and should be stable. I'll bundle it into the next whoisit release.

As for interface, this feature will be opt-in (so overrides=False is the default).

With handling SSL downgrades I don't want to specify this as a default even when using overrides. It may not be clear that enabling overrides will downgrade their SSL security at the same time. While it's read-only and data that's generally considered public it's likely not going to be a great idea to weaken security for some endpoints as a default without opting in or a big warning.

If you prefer total reliability on querying over security you can probably just set allow_insecure_ssl=True on every query. All it does is allow known slightly weaker TLS cipher suites, it doesn't permit self-signed certs or anything terrible.

One thing to note is that the non-IANA listed RDAP endpoints, in my brief testing, return data that is not in the standard format returned by endpoints listed by the IANA. Such as, the handle may be missing or some dates aren't set etc. By enabling overrides the quality of the data returned may be reduced or some elements entirely missing.

Thanks for your feedback on using this for domains. I created whoisit for querying IPs, CIDRs, ASNs and entities mostly myself so I didn't particularly test it for domain usage beyond the default IANA bootstrap data.

[meeb]

Oh and I've added this helper tool as well:

https://github.com/meeb/whoisit/blob/main/tools/list_public_tlds_with_rdap_endpoints.py

Which will run on the command line against the current repo if you want an update of what TLDs are supported, not supported and which are overrides.

[case]

Thank you again! This is great.

And coincidentally, .build is already fixed: