oelachqar / system800 Goto Github PK

We shouldn't rely on twimlets.com

Example stacktrace:

Traceback (most recent call last):
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/http/client.py", line 1321, in getresponse
response.begin()
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/http/client.py", line 296, in begin
version, status, reason = self._read_status()
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/http/client.py", line 265, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 2, in raise_from
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/http/client.py", line 1321, in getresponse
response.begin()
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/http/client.py", line 296, in begin
version, status, reason = self._read_status()
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/http/client.py", line 265, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/celery/app/trace.py", line 382, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/celery/app/trace.py", line 641, in protected_call
return self.run(*args, **kwargs)
File "/Users/dger/projets/system800/api/tasks.py", line 110, in run
status = twilio.fetch_status(call_sid)
File "/Users/dger/projets/system800/workflow/call/twilio_call_wrapper.py", line 126, in fetch_status
return self.fetch_call(call_sid).status
File "/Users/dger/projets/system800/workflow/call/twilio_call_wrapper.py", line 107, in fetch_call
return self._client.calls.get(call_sid).fetch()
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/twilio/rest/api/v2010/account/call/init.py", line 422, in fetch
params=params,
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/twilio/base/version.py", line 78, in fetch
allow_redirects=allow_redirects,
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/twilio/base/version.py", line 47, in request
allow_redirects=allow_redirects
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/twilio/base/domain.py", line 46, in request
allow_redirects=allow_redirects
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/twilio/rest/init.py", line 128, in request
allow_redirects=allow_redirects
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/twilio/http/http_client.py", line 68, in request
timeout=timeout,
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/miniconda3/envs/asapEnv37/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

http://docs.celeryproject.org/en/latest/userguide/security.html#id2

We should make sure that our internal logging captures whatever exception has been thrown, as well as the input case number, and the task number we generated.

We should try to report something that's humanly understandable to the user.

Confirmed by manually raising exceptions

https://stackoverflow.com/questions/9731435/retry-celery-tasks-with-exponential-back-off#comment90534054_46467851

http://docs.celeryproject.org/en/latest/userguide/configuration.html#redis-backend-settings

It seems like we are using machine default timezone, which can cause problem if running on different workers.

For example If running a worker locally (PST timezone) vs one Azure (UTC)

Currently we are calling to endpoint to make sure it is correct.

This is problematic because the callback should only be called to return results, as it may interfere with any internal processing the consumers of the API might be doing. Also, if the callback is momentarily down we will fail too early.

Suggestion:
only check if the callback_url is not empty or null, and that is a well formatted url.

https://pyjwt.readthedocs.io/en/latest/usage.html

this was caused I believe by this change I made.

For easier monitoring and debugging, we should have a continuously running flower instance. ]

Before deploying it, we should setup some kind of authentication (basic auth is probably enough).

We currently use the default values for both celery and gunicorn, (i.e. prefork synchronous workers).

This is not optimal as all our tasks are I/O bound and not CPU bound. We should try using some async workers

We need to check that the config variables are not None before booting the API, especially the ones related to security

The env variable could be None because of various misconfiguration issues, in which case access to the API could be unrestricted

Currently using azure defaults, which might not be the most secure / optimized

We are currently calling Google Cloud transcription via the speech recognition package.

This may break if the audio exceeds 1 minute:
https://cloud.google.com/speech-to-text/docs/async-recognize#speech-async-rec

We should either truncate the audio that we send, or switch to using the above async call. (The latter may make date extraction more difficult -- for example, we may end up with two dates.)

We are currently having a 500 Error if the ain is null, we should return a 400 error.

@leandraking

https://github.com/GoogleCloudPlatform/flask-talisman or https://github.com/TypeError/secure.py

We make requests to

• twilio (many times)
• google
• the callback number

Redis is a memory key value store, which means if the memory is full or if the machine reboots we loose our data.

To mitigate this we need to:

1. Configure disk backups
2. Set a retention policy for records (Eg. auto delete after 1 week)
3. Choose a policy for automatic record deletion when memory is full (Eg. least recently used first)

Currently we authorize all IPs to ping Redis. Redis is password protected but for additional security we should blacklist everything and allow only a small subset of trusted IPs