aio-libs / aioelasticsearch Goto Github PK
View Code? Open in Web Editor NEWaioelasticsearch-py wrapper for asyncio
Home Page: https://pypi.python.org/pypi/aioelasticsearch
License: MIT License
aioelasticsearch-py wrapper for asyncio
Home Page: https://pypi.python.org/pypi/aioelasticsearch
License: MIT License
Since it is not supported since aiohttp 3.0 and the minimum supported version is going to be 3.5.0, unnecessary python <= 3.5.3 compatibility code should be removed
from elasticsearch.exceptions import *
place correct __ALL__
use in __init__.py
and all internal modules import from .exceptions
elasticsearch-py requests based realization
http://elasticsearch-py.readthedocs.io/en/master/index.html?#running-on-aws-with-iam
Amazon docs:
https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/
aiohttp 2.3.0 introduced ClientConnectorSSLError and ClientConnectorCertificateError
they should be used instead of str(exc)
hack
If 'max_retries=0' is set and the request fails, the client goes into an infinite loop, resulting in a ddos elasticsearch.
In the synchronous version of the client, this case is handled correctly.
elastic/elasticsearch-py#924
"Since scan search-type is deprecated since 2.1, there's simpler implementation for scan logic."
We can create a more understandable README. Take a look at this beautiful template from Billie Thompson https://gist.github.com/PurpleBooth/109311bb0361f32d87a2
Do like this:
REPO_NAME={{ YOUR_REPO_NAME }}
travis encrypt -r "aio-libs/${REPO_NAME}" --api-endpoint 'https://api.travis-ci.com/'
Ref: https://github.com/orgs/aio-libs/teams/admins/discussions/9
I am using this library in a concurrent environment but I frequently get timeout errors when there shouldn't be any.
I am instantiating the Elasticsearch
object with maxsize=200
since it could be called by around 200 coroutines at any given time. The default value for the timeout is 10 seconds and should be plenty, but I increased it to 30 seconds and even if the timeouts diminish in frequency I still get a lot of them.
Information about what internal IO operation was interrupted by timeout expiring is important in some cases, it allows to figure out what exact IO takes too long time.
raise TimeoutError from None
should be replaced with raise TimeoutError from exc
In my app which uses aioelasticsearch, I sometimes get these errors but I can’t figure out if there is something wrong with my app or the lib. Specifically, how do I interpret what is causing it / what it means?
python: File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/transport.py", line 338, in perform_request
python: ignore=ignore, timeout=timeout, headers=headers,
python: File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/transport.py", line 252, in _perform_request
python: connection = await self.get_connection()
python: File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/transport.py", line 236, in get_connection
python: return self.connection_pool.get_connection()
python: File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/pool.py", line 96, in get_connection
python: self.resurrect()
python: File "/home/sml/virtualenvs/cr-api-web/lib/python3.6/site-packages/aioelasticsearch/pool.py", line 82, in resurrect
python: self.dead.put_nowait((timestamp, connection))
python: File "/usr/lib/python3.6/asyncio/queues.py", line 150, in put_nowait
python: self._put(item)
python: File "/usr/lib/python3.6/asyncio/queues.py", line 231, in _put
python: heappush(self._queue, item)
python: TypeError: '<' not supported between instances of 'AIOHttpConnection' and 'AIOHttpConnection'
doc_type
was removed in elasticsearch 7.0.0.
tests requires related changes
Currently there is no way to use custom separate session for each host.
We can solve this by allowing users to pass their own session factory class which will be called on each AIOHttpConnection
initialisation if there is no explicit session provided.
aioelasticsearch.AIOHttpConnection
shouldn't call aiohttp.ClientSession.close
in case of custom supplied session, otherwise it breaks all connections, cuz session is shared for all connections.
Does this library support the ES Bulk helpers referenced here: http://elasticsearch-py.readthedocs.io/en/5.5.1/helpers.html#elasticsearch.helpers.bulk — if so, how would I go about importing it?
0.5.2 is the only release that require elasticsearch>=6
File "...../lib/python3.9/site-packages/aioelasticsearch/connection.py", line 61, in init
if StrictVersion(aiohttp.version).version < (3, 0):
File "/usr/lib/python3.9/distutils/version.py", line 40, in init
self.parse(vstring)
File "/usr/lib/python3.9/distutils/version.py", line 137, in parse
raise ValueError("invalid version number '%s'" % vstring)
ValueError: invalid version number '3.7.4.post0'
Hi,
from what i've testet this async es client is way slower than the sync version. So, did i make a mistake or is this behaviour expected?
Kind regrads
Now AIOHttpConnectionPool
accepts and silently ignores all unknown parameters.
It leads to mysterious bugs when type spelling in argument name doesn't raise an exception.
aioelasticsearch==0.5.2
elasticsearch==2.4.1
AIOHttpConnection
takes use_ssl
argument and passes one to the parent's __init__
.
Then self.use_ssl
gets used instead of use_ssl
here
https://github.com/aio-libs/aioelasticsearch/blob/master/aioelasticsearch/connection.py#L52
It works for the latest elasticsearch-py
versions, however, it doesn't work with >2.0.0
, b/c there's no self.use_ssl = use_ssl
there.
The fix is straightforward - use use_ssl
directly instead of self.use_ssl
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.