Comments (10)
Yes, libuv uses a thread pool to do getaddrinfo
and getnameinfo
, and so does uvloop. AFAIK nodejs doesn't use c-ares anymore.
I'm still deciding how to approach this problem -- should we use c-ares? or getdns? Or keep things as is, since DNS in a thread pool is fine for most use cases?
from uvloop.
somewhat relevant to the conversation: https://gist.github.com/deltaindiatango/7bf157202c17681d62667a006eb15abd
found some existing pure-Python resolver, did basic work to make async.
It's not noticably slower than getaddrinfo/getnameinfo.
from uvloop.
Hi, Node guy here - Node.js does use c-ares. Here's discussion about replacing it nodejs/node#1013
DNS thread pool is a compromise. Node documents .lookup
:
Though the call to dns.lookup() will be asynchronous from JavaScript's perspective, it is implemented as a synchronous call to getaddrinfo(3) that runs on libuv's threadpool. Because libuv's threadpool has a fixed size, it means that if for whatever reason the call to getaddrinfo(3) takes a long time, other operations that could run on libuv's threadpool (such as filesystem operations) will experience degraded performance. In order to mitigate this issue, one potential solution is to increase the size of libuv's threadpool by setting the 'UV_THREADPOOL_SIZE' environment variable to a value greater than 4 (its current default value). For more information on libuv's threadpool, see the official libuv documentation.
And .resolve
:
These functions are implemented quite differently than dns.lookup(). They do not use getaddrinfo(3) and they always perform a DNS query on the network. This network communication is always done asynchronously, and does not use libuv's threadpool.
from uvloop.
@benjamingr Benjamin, thanks for chiming in! My current understanding is that for asyncio we'll have to approach this problem similarly to Node: add a completely new set of DNS APIs that doesn't use getaddrinfo
and uses c-ares
or getdns
instead. Maybe we want to make it possible to plug custom DNS resolvers so that create_connection
can use them.
from uvloop.
@1st1 I'm in the opinion you're doing an excellent job using libuv and that you shouldn't worry about this until it becomes a real problem.
I'd argue that the vast majority of programs don't perform a huge amount of uncacheable DNS queries and that it's an edge case. I'd invest more in getting more library adoption and feedback.
In practice the wrapper will behave like users expect for the vast majority of users. I don't see how you'd solve this without using something other than libuv either.
from uvloop.
aiohttp 1.0
uses c-ares
by default if aiodns
is installed.
from uvloop.
But won't c-ares
break expectations in case /etc/nsswitch.conf
has somethings more complex than hosts: files dns
, like mdns
or resolved
(systemd-resolved
-based plugin, which supports LLMNR, per-interface domain names, etc.)?
from uvloop.
@WGH-
Honestly I don't know how these tools are widespread.
Let's look on users feedback.
from uvloop.
we encountered a dns problem using uvloop
with tornado
socket.gaierror: [Errno -3] Temporary failure in name resolution
this hangs our program
maybe there's a way to set a timeout on the lookup or may uvloop dns lookup async?
I found UV_THREADPOOL_SIZE may help, so I'm increasing it to maximum 128, still I think getting a way to timeout dns lookup or throw error will help the program, seems hanging is really bad...
from uvloop.
we encountered a dns problem using
uvloop
withtornado
socket.gaierror: [Errno -3] Temporary failure in name resolution
this hangs our program
maybe there's a way to set a timeout on the lookup or may uvloop dns lookup async?
I found UV_THREADPOOL_SIZE may help, so I'm increasing it to maximum 128, still I think getting a way to timeout dns lookup or throw error will help the program, seems hanging is really bad...
same issue here. +1
from uvloop.
Related Issues (20)
- Python 3.12 Support for uvloop HOT 22
- uvloop installs generated loop.c file
- Can we update cython and make uvloop compatible with python 3.12 please? HOT 1
- test_cython_coro_is_coroutine: python: Objects/codeobject.c:771: PyCode_Addr2Line: Assertion `addrq >= 0 && addrq < _PyCode_NBYTES(co)' failed. HOT 1
- `__init__.pyi` added on #570 breaks mypy HOT 6
- [ALTLinux] [ppc64le] [0.18.0] `AssertionError: Didn't reach a full buffer` HOT 2
- [0.18.0] Running black causes core dump: src/unix/core.c:633: uv__close: Assertion `fd > 2' failed. HOT 5
- Idea: changing libuv not to install a signal handler
- Incorrect typing for set_task_factory HOT 1
- Please upgrade cython (3.0.x) HOT 8
- Possible to add `_start_tls_compatible` attribute to `TCPTransport`?
- Current version of libuv HOT 1
- Embedded Python interpreter in a C++ application making use of libuv
- UDPTransport returning when data is empty
- pipes from subprocess_exec do not have extra info 'pipe' (incompatible with vanilla asyncio) HOT 1
- libuv 1.48.0: getaddrinfo('', 0): OSError: [Errno 22] Invalid argument HOT 5
- Type annotations
- Server.serve_forever() isn't cancelled when Server.close() is called
- uvloop fails to build with Python 3.13.0a5: error: implicit declaration of function ‘_Py_RestoreSignals’ HOT 2
- Qualify claim of being drop-in replacement to asyncio
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from uvloop.