pytroll / posttroll Goto Github PK

For servers restricted by firewalls it is necessary to know which protocols (TCP, UDP, ?) and which ports or port ranges are required for posttroll to work. The requirements should be listed in documentation.

IF namserver is started when network is not available, the publisher seems to start but is not. Even if the publisher fails to start namserver is still running causing confusion since module can not connect to each other using info from nameserver.

[INFO: 2023-02-14 11:16:28 : posttroll.publisher] publisher started on port 16543
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/software/miniconda/envs/pytroll/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/software/miniconda/envs/pytroll/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/posttroll/address_receiver.py", line 176, in _run
    with Publish("address_receiver", self._port, ["addresses"],
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/posttroll/publisher.py", line 278, in __enter__
    return self.start()
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/posttroll/publisher.py", line 221, in start
    addr = ("tcp://" + str(get_own_ip()) + ":" +
  File "/software/miniconda/envs/pytroll/lib/python3.9/site-packages/posttroll/publisher.py", line 47, in get_own_ip
    sock.connect(("8.8.8.8", 80))
OSError: [Errno 101] Network is unreachable

A solution is to wrap https://github.com/pytroll/posttroll/blob/main/posttroll/address_receiver.py#L176-L215 in a try/except OSError and setting self._is_running() to False

That way after https://github.com/pytroll/posttroll/blob/main/posttroll/ns.py#L131-L134 can be added a check if not arec.is_running fail.

At the moment, the zmq calls are tightly woven into the postroll code, making it difficult to add new backends for postroll.

Abstracting the current code and isolating zmq calls behind a simple backend api would allow us to add new backends. New backends could be interesting for several reasons, among which at least a couple which I think will help us develop cleaner postroll based application:

• a pytest or testing backend, which would not actually make any network calls but instead would allow devs to know what has sent and recieved without the need to patch or mock,
• an asyncio backend for applications that just need simple concurrency without threads.

Other backends are of course possible, like rabbitmq or other more specific libraries (for kubernetes maybe?)

I accidentally used spaces in the name to the NoisyPublisher, and nameserver was jammed - the nameserver didn't handle this error gently and was not functioning after that - I think a thread was dead....

Instead of NoisyPublisher("active_fires_postprocessing") I used NoisyPublisher("Active Fires Postprocessing") and that caused nameserver to not respond properly after that

If I understand it correctly, the class Subscribe should return only messages produced by the services that were specified in the init parameter services. However, if I run a new publisher after the Subscriber was created, the subscriber starts listening to it regardless of its service name.

The first publisher:

from posttroll.publisher import Publish
from posttroll.message import Message

with Publish("service_1") as pub:
    while True:
        message = Message("/some_topic", "info", "service_1")
        pub.send(message.encode())
        time.sleep(3)

Subscriber:

from posttroll.subscriber import Subscribe
with Subscribe("service_1", "/some_topic", addr_listener=True) as sub:
    for msg in sub.recv(1):
        if msg is not None:
            print(msg)

It starts getting messages from 'service_1', topic '/some_topic'. However, if I run a second publisher 'service_2' publishing messages with the same topic,

from posttroll.publisher import Publish
from posttroll.message import Message

with Publish("service_2") as pub:
    while True:
        message = Message("/some_topic", "info", "service_2")
        pub.send(message.encode())
        time.sleep(3)

the subscriber returns them as well.

Debug info shown after creating 'service_2'

DEBUG:posttroll.subscriber:Adding address tcp://my_ip:56679 service_1

displays a wrong service name. This indicates that this behaviour may be related to the following part of the code where the variable 'service' is overwritten by the for loop:

    Could we use donfig to delegate the environment variable reading? https://github.com/pytroll/donfig

Originally posted by @mraspaud in #43 (comment)

The examples in the documentation contain SyntaxError as they are still written for Python 2. This may erroneously make people believe posttroll is a Python 2 package, whereas in reality it is already Python 3 only (see #28). There are some other formatting problems in the documentation as well, for example:

from posttroll.subscriber import Subscribe

    with Subscribe("a_service", "counter",) as sub:
        for msg in sub.recv():
            print msg

When starting publishers and subscribers at the same time, occasionally a processes is not able to subscibe to topics.

Example (Trollduction):

• 1x trollstalker, 3x l2processors started at the same time via supervisor.
• only 2 l2processors receive messages of trollstalker

A delay before subscribing should avoid this error. Unfortunatly, supervisor does not support this out-of-the-box: supervisord Starts All Processes at the Same Time

Log files:
log-21102016-l2proc01-refuses-to-work.zip
Config: master_config.zip

Publisher can be used as a context manager, but NoisyPublisher cannot. With #35 we have an API that may return an instance of either. Therefore, it would be desirable if the NoisyPublisher would gain the same context manager capability that Publisher already has.

For more secure operations, it is necessary to add authentication to posttroll communication. This should be added as a complement to the non-authenticated way we have today, so that the user can choose which to use.

Because the polling happens both on subscriptions but also address listener hooks, the timeout is reset when new publishers pop up.

This needs to be fixed in my opinion.

Starting the nameserver in dæmon mode fails with Python 3.8.6, pytroll-collectors-0.10.0, and python-dæmon 2.2.4:

$ nameserver -d start -l /tmp/test.log
Traceback (most recent call last):
  File "/data/gholl/miniconda3/envs/py38b/bin/nameserver", line 129, in <module>
    angel = daemon.runner.DaemonRunner(APP)
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/daemon/runner.py", line 114, in __init__
    self._open_streams_from_app_stream_paths(app)
  File "/data/gholl/miniconda3/envs/py38b/lib/python3.8/site-packages/daemon/runner.py", line 134, in _open_streams_from_app_stream_paths
    self.daemon_context.stderr = open(
ValueError: can't have unbuffered text I/O

This is true while logging to a file or not.

Could this be a Python 2 / Python 3 issue?