Comments (10)
Hrm. I'm guessing there's a limit to the request/payload size that's being hit here. It'll likely require some sort of batching of creates similar to some of the other providers.
from octodns-powerdns.
Hi @XooR. Finally got a few minutes to sit down and try and reproduce this (took me a bit to get a local pdns server going 😁) I wasn't able to hit this problem: I tried with 500, 1500, and 5000 records all of which worked.
Any chance you have a proxy server in front of PowerDNS? Perhaps it has a max request limit or something, thought not sure how that'd be a 500. You might try turning pdns's loglevel
to get more verbosity there to see what it has to say. Otherwise maybe try turning on debug logging for the http backend requests is using on your system, some info on that. There may be useful info in the response content and/or headers.
from octodns-powerdns.
I don't have any proxy for pdns API.
In PowerDNS log I get (loglevel: 15
):
Oct 23 10:53:49 test-dns-01 pdns_server: Oct 23 10:53:49 HTTP ISE for "/api/v1/servers/localhost/zones": STL Exception: Attempt to print an unset dnsname
Oct 23 10:53:49 test-dns-01 pdns_server: Oct 23 10:53:49 HTTP: Error result for "/api/v1/servers/localhost/zones": 500
requests debug log:
2017-10-23T10:58:05 [140568917448448] DEBUG Manager sync: checking safety
2017-10-23T10:58:05 [140568917448448] DEBUG Manager sync: applying
2017-10-23T10:58:05 [140568917448448] INFO PowerDnsProvider[powerdns] apply: making changes
2017-10-23T10:58:05 [140568917448448] DEBUG PowerDnsProvider[powerdns] _apply: zone=foo.lan., len(changes)=471
2017-10-23T10:58:05 [140568917448448] DEBUG PowerDnsProvider[powerdns] _apply: sending change request
2017-10-23T10:58:05 [140568917448448] DEBUG PowerDnsProvider[powerdns] _request: method=PATCH, path=zones/foo.lan.
2017-10-23T10:58:05 [140568917448448] DEBUG urllib3.connectionpool Resetting dropped connection: 10.10.10.10
2017-10-23T10:58:05 [140568917448448] DEBUG urllib3.connectionpool http://10.10.10.10:8081 "PATCH /api/v1/servers/localhost/zones/foo.lan. HTTP/1.1" 422 52
2017-10-23T10:58:05 [140568917448448] DEBUG PowerDnsProvider[powerdns] _request: status=422
2017-10-23T10:58:05 [140568917448448] INFO PowerDnsProvider[powerdns] _apply: creating zone=foo.lan.
2017-10-23T10:58:05 [140568917448448] DEBUG PowerDnsProvider[powerdns] _request: method=POST, path=zones
2017-10-23T10:58:05 [140568917448448] DEBUG urllib3.connectionpool Resetting dropped connection: 10.10.10.10
2017-10-23T10:58:06 [140568917448448] DEBUG urllib3.connectionpool http://10.10.10.10:8081 "POST /api/v1/servers/localhost/zones HTTP/1.1" 500 21
2017-10-23T10:58:06 [140568917448448] DEBUG PowerDnsProvider[powerdns] _request: status=500
2017-10-23T10:58:06 [140568917448448] ERROR PowerDnsProvider[powerdns] _apply: status=500, text=Internal Server Error
Traceback (most recent call last):
File "./bin/octodns-sync", line 11, in <module>
sys.exit(main())
File "/home/foo/env/local/lib/python2.7/site-packages/octodns/cmds/sync.py", line 39, in main
dry_run=not args.doit, force=args.force)
File "/home/foo/env/local/lib/python2.7/site-packages/octodns/manager.py", line 313, in sync
total_changes += target.apply(plan)
File "/home/foo/env/local/lib/python2.7/site-packages/octodns/provider/base.py", line 143, in apply
self._apply(plan)
File "/home/foo/env/local/lib/python2.7/site-packages/octodns/provider/powerdns.py", line 376, in _apply
self._post('zones', data)
File "/home/foo/env/local/lib/python2.7/site-packages/octodns/provider/powerdns.py", line 67, in _post
return self._request('POST', path, data=data)
File "/home/foo/env/local/lib/python2.7/site-packages/octodns/provider/powerdns.py", line 47, in _request
resp.raise_for_status()
File "/home/foo/env/local/lib/python2.7/site-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://10.10.10.10:8081/api/v1/servers/localhost/zones
Also pip list from my virtualenv:
certifi (2017.7.27.1)
chardet (3.0.4)
dnspython (1.15.0)
futures (3.1.1)
idna (2.6)
incf.countryutils (1.0)
ipaddress (1.0.18)
natsort (5.1.0)
octodns (0.8.7)
pip (9.0.1)
pkg-resources (0.0.0)
python-dateutil (2.6.1)
PyYAML (3.12)
requests (2.18.4)
setuptools (36.6.0)
six (1.11.0)
urllib3 (1.22)
wheel (0.30.0)
UPDATE: I also have installed pdns 4.1.0 version on CentOS 7 with Postgres backend. I can make Ansible playbooks for test setup if that would help.
from octodns-powerdns.
Fwiw, the version of powerdns i'm using is 4.0.4 backed by mysql, it was just the version that the docker container I found relied on. This is starting to 👃 more like a pdns bug than hitting some sort of max request size. I'll poke around and see if I can get my container to use 4.1.0 to see if that makes a difference. If the difference is mysql vs postgres then it'll almost certainly be a pdns issue.
from octodns-powerdns.
I'll try it out and will let you know what happens. I have already automated setup, so I can test easily.
from octodns-powerdns.
It looks that 4.1.0 version put limit in webapi requests because when I installed PowerDNS Auth with mysql backend it still failed. With version 4.0.4 it doesn't fail.
I'll check with PowerDNS what we should do in this case and will update this Issue.
from octodns-powerdns.
I'll check with PowerDNS what we should do in this case and will update this Issue.
👍. Cool. I kind of expect it to be a bug more than a limit as if it was an intentional limit I'd imagine they'd use a different status code and print a specific error about the problem. Either way pdns is the place to look at this point. Depending on what we find we can look at a setting a batch size-limit if necessary. Feel free to CC me if you end up filling issues there and we can link this up to them if so.
from octodns-powerdns.
FWIW it seems like powerdns updated it's underlying http library. The update included changes like support for multipart which likely has an impact on large requests like this. I don't see any substantive changes in the webserver that might be applicable to this issue.
from octodns-powerdns.
Just ran into this myself; trying to repopulate a new pdns installation results in failure. The workaround that seems to work for me is to create the new zone using only one or two records, and then load the rest of the zone in bulk.
from octodns-powerdns.
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.
from octodns-powerdns.
Related Issues (9)
- Add support for 'LUA' records in PowerDNS HOT 19
- Escaped chars mismatch for PowerDNS to CloudFlare HOT 8
- configuraton option to manage zone mode of operation and soa_edit_api
- Incorrect type error in debug mode
- version 0.0.2 ? HOT 2
- PowerDnsProvider does call to PowerDNS server for version information HOT 1
- unexpected keyword argument 'ssl_verify' error HOT 3
- Wierd log display when using ssl_verify: False HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from octodns-powerdns.