GithubHelp home page GithubHelp logo

pytroll / posttroll Goto Github PK

View Code? Open in Web Editor NEW
4.0 5.0 11.0 532 KB

High-level messaging system for pytroll.

Home Page: http://posttroll.readthedocs.org/

License: GNU General Public License v3.0

Python 96.05% Makefile 0.29% C++ 3.65%
messaging python zeromq hacktoberfest

posttroll's Issues

Decouple api from zmq backend

At the moment, the zmq calls are tightly woven into the postroll code, making it difficult to add new backends for postroll.

Abstracting the current code and isolating zmq calls behind a simple backend api would allow us to add new backends. New backends could be interesting for several reasons, among which at least a couple which I think will help us develop cleaner postroll based application:

  • a pytest or testing backend, which would not actually make any network calls but instead would allow devs to know what has sent and recieved without the need to patch or mock,
  • an asyncio backend for applications that just need simple concurrency without threads.

Other backends are of course possible, like rabbitmq or other more specific libraries (for kubernetes maybe?)

starting nameserver fails with "ValueError: can't have unbuffered text I/O"

Starting the nameserver in dæmon mode fails with Python 3.8.6, pytroll-collectors-0.10.0, and python-dæmon 2.2.4:

$ nameserver -d start -l /tmp/test.log
Traceback (most recent call last):
  File "/data/gholl/miniconda3/envs/py38b/bin/nameserver", line 129, in <module>
    angel = daemon.runner.DaemonRunner(APP)
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/daemon/runner.py", line 114, in __init__
    self._open_streams_from_app_stream_paths(app)
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/daemon/runner.py", line 134, in _open_streams_from_app_stream_paths
    self.daemon_context.stderr = open(
ValueError: can't have unbuffered text I/O

This is true while logging to a file or not.

Could this be a Python 2 / Python 3 issue?

Add curve authentication to posttroll connections

For more secure operations, it is necessary to add authentication to posttroll communication. This should be added as a complement to the non-authenticated way we have today, so that the user can choose which to use.

SyntaxErrors in documentation

The examples in the documentation contain SyntaxError as they are still written for Python 2. This may erroneously make people believe posttroll is a Python 2 package, whereas in reality it is already Python 3 only (see #28). There are some other formatting problems in the documentation as well, for example:

from posttroll.subscriber import Subscribe

    with Subscribe("a_service", "counter",) as sub:
        for msg in sub.recv():
            print msg

Allow to use NoisyPublisher as a context manager

Publisher can be used as a context manager, but NoisyPublisher cannot. With #35 we have an API that may return an instance of either. Therefore, it would be desirable if the NoisyPublisher would gain the same context manager capability that Publisher already has.

Subscriber does not correctly filter messages by the name of the service

If I understand it correctly, the class Subscribe should return only messages produced by the services that were specified in the init parameter services. However, if I run a new publisher after the Subscriber was created, the subscriber starts listening to it regardless of its service name.

The first publisher:

from posttroll.publisher import Publish
from posttroll.message import Message

with Publish("service_1") as pub:
    while True:
        message = Message("/some_topic", "info", "service_1")
        pub.send(message.encode())
        time.sleep(3)

Subscriber:

from posttroll.subscriber import Subscribe
with Subscribe("service_1", "/some_topic", addr_listener=True) as sub:
    for msg in sub.recv(1):
        if msg is not None:
            print(msg)

It starts getting messages from 'service_1', topic '/some_topic'. However, if I run a second publisher 'service_2' publishing messages with the same topic,

from posttroll.publisher import Publish
from posttroll.message import Message

with Publish("service_2") as pub:
    while True:
        message = Message("/some_topic", "info", "service_2")
        pub.send(message.encode())
        time.sleep(3)

the subscriber returns them as well.

Debug info shown after creating 'service_2'

DEBUG:posttroll.subscriber:Adding address tcp://my_ip:56679 service_1

displays a wrong service name. This indicates that this behaviour may be related to the following part of the code where the variable 'service' is overwritten by the for loop:

service = msg.data.get('service')
for service in self.services:
if not service or service in service:
LOGGER.debug("Adding address %s %s", str(addr_),
str(service))
self.subscriber.add(addr_)
break

Missing subscription due to race condition

When starting publishers and subscribers at the same time, occasionally a processes is not able to subscibe to topics.

Example (Trollduction):

  • 1x trollstalker, 3x l2processors started at the same time via supervisor.
  • only 2 l2processors receive messages of trollstalker

A delay before subscribing should avoid this error. Unfortunatly, supervisor does not support this out-of-the-box: supervisord Starts All Processes at the Same Time

Log files:
log-21102016-l2proc01-refuses-to-work.zip
Config: master_config.zip

Timezone aware datetimes

With Python 3.12 we get these warnings:

DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).

When running the tests, we get 107135 of them (the actual number changes depending on the run time). There are zero warnings with Python 3.11.

We need to make the datetimes timezone aware. I made a quick try, but ran into some snags that I don't have the time to tackle today:

  • the code needs flake8 work
  • we need to stop doing from datetime import datetime as it makes it impossible to do other imports from datetime package. Instead do import datetime as dt
  • making the datetimes tz-aware I got these assertion errors: AssertionError: '2010-12-03T16:28:39' != '2010-12-03T16:28:39+00:00', so I sense we might get backwards incompatibilites within the messages

Nameserver is jammed when publishing using space in the name topic

I accidentally used spaces in the name to the NoisyPublisher, and nameserver was jammed - the nameserver didn't handle this error gently and was not functioning after that - I think a thread was dead....

Instead of NoisyPublisher("active_fires_postprocessing") I used NoisyPublisher("Active Fires Postprocessing") and that caused nameserver to not respond properly after that

Nameserver is still running even if publisher fails to start

IF namserver is started when network is not available, the publisher seems to start but is not. Even if the publisher fails to start namserver is still running causing confusion since module can not connect to each other using info from nameserver.

[INFO: 2023-02-14 11:16:28 : posttroll.publisher] publisher started on port 16543
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/software/miniconda/envs/pytroll/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/software/miniconda/envs/pytroll/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/posttroll/address_receiver.py", line 176, in _run
    with Publish("address_receiver", self._port, ["addresses"],
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/posttroll/publisher.py", line 278, in __enter__
    return self.start()
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/posttroll/publisher.py", line 221, in start
    addr = ("tcp://" + str(get_own_ip()) + ":" +
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/posttroll/publisher.py", line 47, in get_own_ip
    sock.connect(("8.8.8.8", 80))
OSError: [Errno 101] Network is unreachable

A solution is to wrap https://github.com/pytroll/posttroll/blob/main/posttroll/address_receiver.py#L176-L215 in a try/except OSError and setting self._is_running() to False

That way after https://github.com/pytroll/posttroll/blob/main/posttroll/ns.py#L131-L134 can be added a check if not arec.is_running fail.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.