mozilla / esfrontline

DEPRECATED - Limit restful requests to backend ElasticSearch cluster: Queries only.

esFrontLine

Limit restful requests to backend ElasticSearch cluster: Queries only.

Requirements

• Python 2.7
• An ElasticSearch cluster to forward queries to

Install

I will assume you have Python installed (if not, here are Windows7 instructions)

pip install esFrontLine

Setup

You must write your own setting.json file with the following properties set:

• elasticsearch - (Array of) ElasticSearch nodes

• elasticsearch.host - URL of the ElasticSearch node that will accept query requests

• elasticsearch.port - port for ES (default = 9200)

• flask - flask.run() parameters (default port = 5000)

• debug - turn on debugging

• whitelist - list of indexes that are allowed

Here is an example of my settings.json file

{
    "elasticsearch":[{
        "host":"http://elasticsearch4.metrics.scl3.mozilla.com",
        "port":9200
    },{
        "host":"http://elasticsearch5.metrics.scl3.mozilla.com",
        "port":9200
    },{
        "host":"http://elasticsearch7.metrics.scl3.mozilla.com",
        "port":9200
    },{
        "host":"http://elasticsearch8.metrics.scl3.mozilla.com",
        "port":9200
    }],
    "flask":{
        "host":"0.0.0.0",
        "port":9292,
        "debug":false,
        "threaded":true,
        "processes":1
    },
    "whitelist":["bugs", "org_chart", "bug_summary", "reviews"],
    "debug":{
        "log":[{
            "filename": "./tests/results/logs/app.log",
            "maxBytes": 10000000,
            "backupCount": 200,
            "encoding": "utf8"
        },{
            "stream":"sys.stdout"
        }]
    }

}

Execution

python app.py --settings-file <path_to_file_with_JSON_settings>

Code Source

https://github.com/klahnakoski/esFrontLine

esfrontline's Issues

Proxy always returns gzipped content

Even though I am not requesting compressed responses (by setting the Accept-Encoding header), the proxy still returns gzipped content:

curl -v -XGET -d '{"query": { "term": { "priority": "xxp4" } }}' "http://127.0.0.1:9292/bugs20121030/_search"
* About to connect() to 127.0.0.1 port 9292 (#0)
*   Trying 127.0.0.1...
* Adding handle: conn: 0x17f7ad0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x17f7ad0) send_pipe: 1, recv_pipe: 0
* Connected to 127.0.0.1 (127.0.0.1) port 9292 (#0)
> GET /bugs20121030/_search HTTP/1.1
> User-Agent: curl/7.32.0
> Host: 127.0.0.1:9292
> Accept: */*
> Content-Length: 45
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 45 out of 45 bytes
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< content-length: 122
< content-type: application/json; charset=UTF-8
< access-control-allow-origin: *
< content-encoding: gzip
< Server: Werkzeug/0.9.4 Python/2.7.5+
< Date: Wed, 13 Nov 2013 20:50:10 GMT
<
�L�A ... Compressed content follows

From a HTTP point of view this is not correct. This will also break clients and libraries. (Like curl for example)

Fix prefix url calculation

By allowing the url_prefix to represent the unmodified url path prefix (starting with not / and ending with /) then we can replace

        # Remove prefix from url to reach ES
        url_prefix = settings.get('url_prefix')
        if url_prefix:
            assert url_prefix.startswith('/'), '"url_prefix" must start with a /'
            assert not url_prefix.endswith('/'), '"url_prefix" must not end with a /'
            es_path = re.sub(r'^{}/'.format(url_prefix[1:]), '', path)
        else:
            es_path = path

with

        es_path = path.ltrim(settings.url_prefix)

Report ES response time

So we can log it in our backend at https://github.com/mozilla/release-services.

Proxy does not filter on collection name

This might not be an issue if the only index in ES is the bugzilla one. But right now it is possible to query any available index.

For example:

curl -v -XGET -d '{"query": { "term": { "email": "[email protected]" } }}' \
    "http://127.0.0.1:9292/org_chart131113_105007/_search"

I think it is best to add a list of allowed indexes to the settings.json so that the proxy can deny requests for anything not whitelisted.

Wiki changes

FYI: The following changes were made to this repository's wiki:

• defacing spam has been removed

• the wiki has been disabled, as it was not used

These were made as the result of a recent automated defacement of publically writeable wikis.

Add Hawk Credential Verification

Use mohawk for http authentication. MoDataSubmission has both client and server implementations

Add config file for testing

The test(s) should setup a server and test a client against it. This requires the /tests directory have a config file with some authentication token (just for testing of course).

merge the READMEs

There are two READMEs, and they should be merged to a single document

add travis testing

Travis testing, with a real Elasticsearch instance would be nice. Then we can be sure the project is in a good state many years from now.

Testing with ES will enable us to test integration issues, like #2

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

Missing import of sys

In the remove_util branch there is a missing import sys in app.py. This is triggered by having the following in your settings.py (taken from the README):

    "debug":{
        "log":[{
            "class":"esFrontLine.util.logs.Log_usingStream",
            "stream":"sys.stdout"
        }]
    }

The app fails with a cryptic message:

__init__() got an unexpected keyword argument 'class'

But what really happens is:

Traceback (most recent call last):
  File "app.py", line 184, in <module>
    ch = logging.StreamHandler(stream=eval(d["stream"]))
  File "<string>", line 1, in <module>
NameError: name 'sys' is not defined