aio-libs-abandoned / aioes Goto Github PK

yarl (0.8.1) aio-libs/yarl#9

File "/usr/local/lib/python3.5/dist-packages/aioes/connection.py", line 41, in perform_request
    url = self._base_url.with_path(url)
AttributeError: 'URL' object has no attribute 'with_path'

I update aiohttp to 2.0.3 slove the "with_path" problem

And
I am not able to add http headers when I perform http request in es operation
I want to pass header as argument when instantiate the es client

headers = {"Host": u"***", "key": "***"}
es = Elasticsearch(['localhost:8888'], headers=headers)

TypeError: __init__() got an unexpected keyword argument 'headers'

with elasticsearch==5.1.0, I can do that
Now
I need to reserve header argument in connection.py->Connection->perform_request, and reserve for every method/class using it

I'd like to make new release, version 0.2 should support Python 3.5

Is anything left to be done before the release publishing?

You can clearly see here conversion of parameter to boolean type.
https://github.com/aio-libs/aioes/blob/master/aioes/client/__init__.py#L133
Then it fails here
https://github.com/aio-libs/yarl/blob/master/yarl/__init__.py#L673

Considering this, there's no way to use refresh parameter at all.

aioes is not able to json encode datetime objects.

Hi,

It seem the query need to be a POST on _delete_by_query:
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

But aioes use DELETE on _query
https://github.com/aio-libs/aioes/blob/a739c156704159fd2729188d2d192baa2c31ee97/aioes/client/__init__.py#L971

I'm not sure if the parameters are the same and in which version they have changed.

It looks like there is no chance to use HTTP Basic Auth for now?

Just instantiate Elasticsearch

es = Elasticsearch(["localhost:8000"])

When terminate:

Unclosed client session client_session: <aiohttp.client.ClientSession object at 0x7f64807de208>

Should aioes->connection.py->class Connection has a destructor ?

`

   def __del__(self):
          self._session.close()

`

aiohttp (2.0.4)

when usingELasticSearch.create(refresh=True) gives an error ('Invalid variable type: mapping value should be str or int, got True') because in latest version of aiohttp the params can only be str or int.

so it failing in all requests where bool arguments are accepted

Hello,

In the current state - the transport handles the dumping and loading of documents internally - using json.dumps.

Id like to use ujson for example, but without editing the sources of aioes.

the official ES client implements a Serializer interface - which conviniently supports changing the serializer backend (for example - mentioned here - elastic/elasticsearch-py#365).

Would you consider implementing such an interface?

As aiohttp changes ClientSession.close() to coroutine, we need to change Elasticsearch.close to one too or deprecate it in favor of a new finallize

I cant find it anywhere in the docs or the source code. Does this support the 1.X and 2.X series of ES?

It appears the fix is trivial and a few pull requests have already been submitted. I prefer the approach in:
#48

Can one use aioes with ca_certs passed in as a parameter like elasticsearch-py:

es = Elasticsearch(
    ['localhost', 'otherhost'],
    http_auth=('user', 'secret'),
    port=443,
    use_ssl=True,
    ca_certs='/path/to/cacert.pem',
    client_cert='/path/to/client_cert.pem',
    client_key='/path/to/client_key.pem',
)

The search with scan, scroll and clear_scroll primitives are implemented in the client, but a scan helper is still lacking. It would work like http://elasticsearch-py.readthedocs.io/en/master/helpers.html#scan

For Python 3.6+, it could be an asynchronous generator, which would be closest to the official ES client.
For Python 3.5+, it could be an async for:

async for hit in Scan(
        client=client, index=..., doc_type=...,
        query={"query": {"bool": {"filter": ...}}}):
    ...

The only way to support Python 3.4 would be to add a callback, I guess, but that's not a good fit with asyncio's model, so maybe 3.4 can be dropped for this feature.

Thoughts?

File "/home/dev/.local/lib/python3.6/site-packages/aioes/connection.py", line 52, in perform_request
raise exc_class(resp.status, resp_body, extra)
aioes.exception.TransportError: TransportError(406, '{"error":"Content-Type header [application/octet-stream] is not supported","status":406}')

With test this code:

import asyncio
 from aioes import Elasticsearch
 async def go():
     es = Elasticsearch(['localhost:9200'])
     ret = await es.create(index="users",
                                doc_type="id",
                                id=42,
                                body={"str": "data", "int": 1})
     assert (ret == {'_id': '32',
                     '_index': 'users',
                     '_type': 'id',
                     '_version': 1})
     answer = await es.get(index="users",
                                doc_type="id",
                                id=42)
     assert answer['_source'] == {'str': 'data', 'int': 1}
 loop = asyncio.get_event_loop()
 loop.run_until_complete(go())

I think should contains
connection.py

 43         headers = {"Content-type":"application/json"}
 44         resp = yield from self._session.request(
 45             method, url, params=params, data=body, headers=headers)

This will probably not belong here since it's more an implementation detail, so apologies in advance! I'm slowly porting my code to asyncio as it's mostly I/O based but am hitting an issue with bridging it with synchronous modules, in this case I'm looking at how aioes could be coupled with the excellent elasticsearch_dsl module. Wrapping aioes as a connection is no problem, but at one point one needs to bridge the regular connection call made in elasticsearch_dsl to the asynchronous aioes equivalent, somewhat like this -

import asyncio

@asyncio.coroutine
def target(value):
    return "{} world".format(value)

def intermediary(value):
    """Return result from coroutine target()
    """

@asyncio.coroutine
def call():
    result = intermediary('hello')
    print(result)

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(call())

I've been pulling my hair out about this one, but whatever I can think of I end up with something blocking. Would there be any pattern applicable to these type of situations?

It would be nice to have support rivers feature. As far as I see it currently missed.

If Elasticsearch created outside coroutine it produces aiohttp warning

ERROR [asyncio] Creating a client session outside of coroutine

due

https://github.com/KeepSafe/aiohttp/blob/c4539a7f3851483c985fa885f7e4a151e3bc3d43/aiohttp/client.py#L57

calling es.create without await seems to not work. Testing for py>=3.5.2.