cceh / kosh Goto Github PK

I think you could relax the python version requirement from 3.11 down to 3.8. Is there a reason to not support older python versions due to new syntax usage? Locally, I could run kosh with 3.8 but I have not yet tested all possible code paths just some random api requests.

Currently, when using the ids endpoint for some non-existent ID, the API returns an error instead of a sensible "not found" message, e.g.:

2019-07-04 12:49:39 [ERROR] <graphql.execution.executor> An error occurred while resolving field query.ids
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/graphql/execution/executor.py", line 450, in resolve_or_error
    return executor.execute(resolve_fn, source, info, **args)
  File "/usr/lib/python3.7/site-packages/graphql/execution/executors/sync.py", line 16, in execute
    return fn(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/kosh-0.0.1-py3.7.egg/kosh/api/graphql.py", line 50, in resolve_ids
    return search.ids(elex, ids)
  File "/usr/lib/python3.7/site-packages/kosh-0.0.1-py3.7.egg/kosh/elastic/search.py", line 29, in ids
    }) for item in find.mget(ids)]
  File "/usr/lib/python3.7/site-packages/kosh-0.0.1-py3.7.egg/kosh/elastic/search.py", line 29, in <listcomp>
    }) for item in find.mget(ids)]
AttributeError: 'NoneType' object has no attribute 'to_dict'

Which results in an graphql error:

{
  "errors": [
    {
      "message": "RequestError(400, 'Required routing not provided for documents .')",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "path": [
        "ids"
      ]
    }
  ],
  "data": {
    "ids": null
  }
}

See: https://www.elastic.co/guide/en/elasticsearch/reference/7.2/search-request-sort.html

You get an empty response if one of the requested IDs is invalid:

request

{
  ids(ids: ["lemma_agni_79", "invalidIDwhatever"]) {
    id
    headwordIso
  }
}

response

{
  "data": {
    "ids": []
  }
}

... should be either a response containing only the matched documents:

{
  "data": {
    "ids": [
      {
        "id": "lemma_agni_79",
        "headwordIso": "agni"
      }
    ]
  }
}

... or even a response with placeholders for the invalid IDs requested:

{
  "data": {
    "ids": [
      {
        "id": "lemma_agni_79",
        "headwordIso": "agni"
      },
      {
        "id": "invalidIDwhatever"
      }
    ]
  }
}

Currently, when the parsing process of any XML resource or .kosh definition fails, kosh breaks down completely. We should catch parsing errors, print a nice log message and ignore the respective (broken) data module.

An info/about ednpoint (REST) or query (GraphQL) should provide metadata about indexed datasets

It should be possible to use aggreation queries via either REST or GraphQL (and any other, future API modules).

Hi Phil @schlusslicht :)
While testing each CLI parameter again for #127, I noticed something odd with the data_sync setting. Using something like kosh --data_sync 20 raises a ValueError.

From the doc string and the default value set in the kosh.ini, data_sync is supposed to be an integer with a value of 0 meaning that the feature is disabled. This is also reflected in the watch function in kosh.py.

However, in the parse function for this parameter, the value is handled as a truth literal instead, using strtobool.

You introduced this parameter a long while ago but only changed the default value to an integer in your overhaul commit 4 months back, when you also added the watcher, I believe.
Did you mean to change the parser as well at some point or am I reading this wrong? 😅

We should implement a xpath property or a custom field within the JSON mappings to assign a human understandable name to any imported/served dictionary. Also information about its authors.

E.g.:

{
  "mappings": {
    "_meta": {

    "title": "My Nicename", // <--
    "authors": ["Peter Pan", "Wendy"],   // <--

      "_xpaths": {

        "title": "//title", // <--

        "id": "./@xml:id",
        "root": "//tei:cit[@type='item']",
        "fields": {
          ...
        }
      }
    },
    "properties": {
      ...
    }
  }
}

When trying to make kosh within a python venv, the pip and python binaries are not aliased by their version number, e.g. pip2/pip3 and pytho2/python3. Therefore the python3 call in the Makefile fails.

Kosh currently only supports indexing the text content of XML nodes, but not their attributes. This needs to be implemented as many datasets provide crutial information not within the XML node attributes.

When running kosh instances long-term, their memory.usage grows significantly, which indicates a memory leak. Most probable this leak originates in the XML-parsing components of kosh, possibly occurring because of the use of threads.

This needs to be traced by, e.g., using valagrind in combination with PYTHONMALLOC=malloc.

It seems essential, that one may query multiple fields. To implement this we need to:

• implement search logic to query multiple fields (e.g. encapsulate in boolean query)
• enable SwaggerUI/RESTful API to add key/value pairs with predefined keys
• enable Graphqli/Graphql API to add key/value pairs with predefined keys

We should add an endpoint that should list the indexed datasets per instance. We should also add basic information such as number of items per dataset.

On page Deploying Kosh's Backend > On Unix-like Systems step 3 mentions a make command (Build Kosh using the make command). This should be something old since I saw no Makefile in the repo. Or is this dependent on something else?

When using the entries query, searching within the id field is broken, e.g. never returns any results. Either the id field should not be searchable from the entries query or the search logic must be fixed.

It should be possible to (1) index nested objects via specifying a object structure in the respective JSON definitions file. Further, it then should be possible to (2) query the nested objects via either REST or GraphQL (and any other, future API modules).

Add fields to the output for each dictionary