GithubHelp home page GithubHelp logo

machinezone / cobra Goto Github PK

View Code? Open in Web Editor NEW
32.0 32.0 3.0 3.23 MB

Cobra is a realtime messaging server using Python3, WebSockets and Redis

License: BSD 3-Clause "New" or "Revised" License

Dockerfile 0.49% Makefile 1.15% Python 55.24% JavaScript 4.49% Shell 3.02% HTML 0.03% C# 2.46% CMake 1.61% C++ 31.46% C 0.04%
python3 redis websockets

cobra's People

Contributors

agtiwari avatar bsergean avatar dependabot[bot] avatar johnfoconnor avatar vravinarayanan avatar zoolander-applovin avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

agtiwari tiwariashish86 cperbren

cobra's Issues

Handling data not utf-8 encoded

Cobra can receive data in a weird encoding, we can use chardet to try to detect it and 'recover'.

Problem could also be linked to the fact that TEXT is sent while historically cobra has received BINARY data (websocket type).

Exceptions in some server code (redis subscribers) lead to hangs 

2019-08-21 11:58:08 ERROR Unhandled exception in async future
future: <Task finished coro=<redisSubscriber() done, defined at /Users/bsergeant/src/foss/cobra/cobras/server/redis_subscriber.py:44> exception=TypeError('respond() takes 3 positional arguments but 4 were given')>
Traceback (most recent call last):
  File "/Users/bsergeant/src/foss/cobra/cobras/server/redis_subscriber.py", line 60, in redisSubscriber
    await messageHandler.on_init(connection)
  File "/Users/bsergeant/src/foss/cobra/cobras/server/handlers/pubsub.py", line 204, in on_init
    await self.state.respond(self.ws, self.app, response)
TypeError: respond() takes 3 positional arguments but 4 were given
^C

Which kinds of errors should be send to sentry is also up in the air.

Ping timeout errors 

[2020-08-26 21:23:31.243] [ws] [info] messages received 0 4562752 sent 0 4562752
[2020-08-26 21:23:32.243] [ws] [info] messages received 0 4562752 sent 0 4562752
[2020-08-26 21:23:32.908] [ws] [info] Subscriber closed: {}Close code 1011 reason Ping timeout
[2020-08-26 21:23:33.007] [ws] [info] Subscriber connected
[2020-08-26 21:23:33.007] [ws] [info] Connection: upgrade
[2020-08-26 21:23:33.007] [ws] [info] Date: Wed, 26 Aug 2020 21:23:33 GMT
[2020-08-26 21:23:33.007] [ws] [info] Sec-WebSocket-Accept: Kwn3MLpeFbF3b78mGf/Ld3ew8P0=
[2020-08-26 21:23:33.007] [ws] [info] Sec-WebSocket-Extensions: permessage-deflate; server_max_window_bits=15; client_max_window_bits=15
[2020-08-26 21:23:33.007] [ws] [info] Sec-WebSocket-Protocol: json
[2020-08-26 21:23:33.007] [ws] [info] Server: nginx/1.15.10
[2020-08-26 21:23:33.007] [ws] [info] Strict-Transport-Security: max-age=15724800; includeSubDomains
[2020-08-26 21:23:33.007] [ws] [info] Upgrade: websocket
[2020-08-26 21:23:33.007] [ws] [info] X-Cobra-Node: cobra-subscribers-250-d9b5w
[2020-08-26 21:23:33.007] [ws] [info] X-Cobra-Version: 2.9.97
[2020-08-26 21:23:33.010] [ws] [info] Subscriber closed: {}Close code 1011 reason Ping timeout
[2020-08-26 21:23:33.103] [ws] [info] Subscriber connected
[2020-08-26 21:23:33.103] [ws] [info] Connection: upgrade
[2020-08-26 21:23:33.103] [ws] [info] Date: Wed, 26 Aug 2020 21:23:33 GMT
[2020-08-26 21:23:33.103] [ws] [info] Sec-WebSocket-Accept: SNt31vb4U809r1RWthd/MB78PEc=
[2020-08-26 21:23:33.103] [ws] [info] Sec-WebSocket-Extensions: permessage-deflate; server_max_window_bits=15; client_max_window_bits=15
[2020-08-26 21:23:33.103] [ws] [info] Sec-WebSocket-Protocol: json
[2020-08-26 21:23:33.103] [ws] [info] Server: nginx/1.15.10
[2020-08-26 21:23:33.103] [ws] [info] Strict-Transport-Security: max-age=15724800; includeSubDomains
[2020-08-26 21:23:33.103] [ws] [info] Upgrade: websocket
[2020-08-26 21:23:33.103] [ws] [info] X-Cobra-Node: cobra-subscribers-250-d9b5w
[2020-08-26 21:23:33.103] [ws] [info] X-Cobra-Version: 2.9.97
[2020-08-26 21:23:33.105] [ws] [info] Received websocket pong: ixwebsocket::heartbeat::30s::374
[2020-08-26 21:23:33.148] [ws] [info] Subscriber authenticated
[2020-08-26 21:23:33.148] [ws] [info] Subscribing to blah_id
[2020-08-26 21:23:33.148] [ws] [info] Subscribing at position 1598476980499-0
[2020-08-26 21:23:33.148] [ws] [info] Subscribing with filter 
[2020-08-26 21:23:33.207] [ws] [info] Subscriber: subscribed to channel blah_id
[2020-08-26 21:23:33.244] [ws] [info] messages received 0 4562752 sent 0 4562752
[2020-08-26 21:23:34.244] [ws] [info] messages received 34 4562786 sent 34 4562786
[2020-08-26 21:23:35.244] [ws] [info] messages received 876 4563662 sent 875 4563661

Dummy position supplied by server doesn't pass validation

Dummy position: 1519190184:559034812775

cobra/cobras/server/handlers/pubsub.py

Line 205 in d9cb898

"position": "1519190184:559034812775",

Position validation:

https://github.com/machinezone/rcc/blob/bbe0f517f30ca075cd021aac0252f489f730a444/rcc/subscriber.py#L19-L26

POSITION_PATTERN = re.compile('^(?P<id1>[0-9]+)-(?P<id2>[0-9]+)')

def validatePosition(position):
    if position is None or position == '$':
        return True
    return POSITION_PATTERN.match(position)

Reporting a vulnerability

Hello!

I hope you are doing well!

We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.

Can you enable it, so that we can report it?

Thanks in advance!

PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository

Fix unittest warnings from websockets-7.0

For

/Users/bsergeant/src/foss/cobra/venv/lib/python3.7/site-packages/asyncio_redis/protocol.py:549: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly
    signature = formatargspec(* self.specs[:6])

This could be used: sqlalchemy/alembic@e3c5fc3

Cobra server gets in a weird state if the redis-server it connects too gets restarted

Exception 1

[20ffff89::pubsub] < {"action": "rtm/subscribe", "body": {"subscription_id": "football", "channel": "football", "fast_forward": true, "filter": null, "position": "0-0"}, "id": 3}
2019-08-09 00:55:49 cobra-79cd6f74b9-jhp7n asyncio[1] ERROR Future exception was never retrieved
future: <Future finished exception=ConnectionClosedError('Reader at end of file')>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aioredis/commands/transaction.py", line 208, in _send_pipeline
    result_fut = conn.execute(cmd, *args, **kw)
  File "/usr/local/lib/python3.7/site-packages/aioredis/connection.py", line 319, in execute
    raise ConnectionClosedError(msg)
aioredis.errors.ConnectionClosedError: Reader at end of file
2019-08-09 00:55:49 cobra-79cd6f74b9-jhp7n asyncio[1] ERROR Future exception was never retrieved

Exception 2

[c4805c5d::pubsub] > {"action": "auth/authenticate/ok", "id": 1, "body": {}}
[c4805c5d::pubsub] < {"action": "rtm/subscribe", "body": {"subscription_id": "football", "channel": "football", "fast_forward": true, "filter": null, "position": "0-0"}, "id": 3}
[c4805c5d::pubsub] > {"action": "rtm/subscribe/ok", "id": 3, "body": {"position": "1519190184:559034812775", "subscription_id": "football", "redis_node": "redis-2-service"}}
[c4805c5d::pubsub] < {"action": "rtm/publish", "body": {"channel": "football", "message": {"data": {"user": "john", "text": "cadsc"}}}}
'NoneType' object has no attribute 'write'
Generic Exception caught in Traceback (most recent call last):
  File "/cobras/server/app.py", line 49, in cobraHandler
    await processCobraMessage(state, websocket, app, message)
  File "/cobras/server/protocol.py", line 588, in processCobraMessage
    await handler(state, ws, app, pdu, msg)
  File "/cobras/server/protocol.py", line 201, in handlePublish
    batchPublish)
  File "/cobras/server/pipelined_publisher.py", line 51, in push
    await self.publishNow(job)
  File "/cobras/server/pipelined_publisher.py", line 47, in publishNow
    await pipe.execute()
  File "/usr/local/lib/python3.7/site-packages/aioredis/commands/transaction.py", line 180, in execute
    return_exceptions=return_exceptions)
  File "/usr/local/lib/python3.7/site-packages/aioredis/commands/transaction.py", line 187, in _do_execute
    return_exceptions=True)
  File "/usr/local/lib/python3.7/site-packages/aioredis/commands/transaction.py", line 214, in _send_pipeline
    yield result_fut
  File "/usr/local/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/aioredis/connection.py", line 300, in _buffered
    self._writer.write(buf)
AttributeError: 'NoneType' object has no attribute 'write'

[c4805c5d::pubsub] cancelling #1 subscriptions
[c4805c5d::pubsub] (close) uptime 0:00:02 msgcount 4 connections 1
[c4805c5d::pubsub] Cancelling redis subscription
[c4805c5d::pubsub] Closing redis subscription

Handle / research UNBLOCKED aioredis error 

[4b166ffa::pubsub] UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)
[4b166ffa::pubsub] Generic Exception caught in Traceback (most recent call last):
  File "/cobras/server/redis_subscriber.py", line 52, in redisSubscriber
    latest_ids=[lastId])
  File "/usr/local/lib/python3.7/site-packages/aioredis/util.py", line 55, in wait_convert
    result = await fut
aioredis.errors.ReplyError: UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)

[4b166ffa::pubsub] Closing redis subscription

bavarde client / users should not be able to send messages if the connection cannot be established. 

> 2019-08-19 12:14:57 INFO Recovering and creating a new connection
> 2019-08-19 12:15:03 ERROR Task exception was never retrieved
future: <Task finished coro=<runClient() done, defined at /Users/bsergeant/src/foss/cobra/cobras/bavarde/client/client.py:103> exception=InvalidStatusCode('server rejected WebSocket connection: HTTP 403')>
Traceback (most recent call last):
  File "/Users/bsergeant/src/foss/cobra/cobras/bavarde/client/client.py", line 166, in runClient
    await args['connection'].publish(channel, message)
KeyError: 'connection'

Maybe we can just try / catch that error.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.