Stream Calling Endpoint output {"active": True, "cameras": {}} about deep-license-plate-recognition

[danleyb2]

@lukenuyen02 You need to specify the container name when you run stream_monitor.py like so:

 python3 stream_monitor.py -c focused_dubinsky 

Get your container name by running docker ps command in terminal

[lukenuyen02]

@lukenuyen02 You need to specify the container name when you run stream_monitor.py like so:

 python3 stream_monitor.py -c focused_dubinsky 

Get your container name by running docker ps command in terminal

You can not just close the ticket like this. please reopen the Ticket. Even with container the issue is still persist

[lukenuyen02]

@marcbelmont @danleyb2 I would like to reopen the issue ticket please.

[danleyb2]

@lukenuyen02 What is your stream container name? I tested the script and it works as expected for me with 1 camera

[lukenuyen02]

@danleyb2 with 1 camera, yes its working, but not with Multiple

my container name is stream as the script is already define it as the default value.

[danleyb2]

@lukenuyen02 I have been unable to reproduce this with multiple cameras.
Do the cameras startup okay and there is newer log from Stream container showing the health score?
Try enabling debug logging from the monitor script by setting the env variable like so export LOGGING=DEBUG

[lukenuyen02]

@danleyb2 everything looks good

support@ia-88aedd0bb29d:~$ sudo docker logs --tail 10  stream
INFO:3cc21d76-3a34-4794-8851-7c1c50bb29b5:2023-05-24T17:44:14+00:00: Health Score: 100%
INFO:3d92981d-5344-4e0c-8734-181b7a8cd7c0:2023-05-24T17:44:17+00:00: Health Score: 95%
INFO:3cc21d76-3a34-4794-8851-7c1c50bb29b5:2023-05-24T17:44:19+00:00: Health Score: 100%
INFO:3d92981d-5344-4e0c-8734-181b7a8cd7c0:2023-05-24T17:44:22+00:00: Health Score: 95%
INFO:3d92981d-5344-4e0c-8734-181b7a8cd7c0:2023-05-24T17:44:23+00:00: New vehicle: jys2511
INFO:3cc21d76-3a34-4794-8851-7c1c50bb29b5:2023-05-24T17:44:24+00:00: Health Score: 100%
INFO:3d92981d-5344-4e0c-8734-181b7a8cd7c0:2023-05-24T17:44:27+00:00: Health Score: 95%
INFO:3cc21d76-3a34-4794-8851-7c1c50bb29b5:2023-05-24T17:44:29+00:00: Health Score: 100%
INFO:3d92981d-5344-4e0c-8734-181b7a8cd7c0:2023-05-24T17:44:33+00:00: Health Score: 95%
INFO:3cc21d76-3a34-4794-8851-7c1c50bb29b5:2023-05-24T17:44:35+00:00: Health Score: 100%

would you like to my version of the stream_monitor.py so you can compare and test? i

[danleyb2]

@lukenuyen02 yes please share

[lukenuyen02]

# python stream_monitor.py --help
# python stream_monitor.py -c stream -i 30 -d 120
import re
import os
import time
import logging
import sys
import threading
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
import subprocess
from functools import partial
from datetime import datetime, timedelta
from django.core.management.base import BaseCommand


LOG_LEVEL = os.environ.get('LOGGING', 'INFO').upper()

logging.basicConfig(
    stream=sys.stdout,
    level=LOG_LEVEL,
    format='%(asctime)s %(levelname)s %(name)s %(threadName)s : %(message)s')

_state = {'container_active': False, 'last_log_times': {}}

log_line_regex = r'(\w+):([^:]*):(.*):\b'
compiled = re.compile(log_line_regex)


class GetHandler(BaseHTTPRequestHandler):

    def __init__(self, offline_diff_duration, *args, **kwargs):
        self.offline_diff_duration = offline_diff_duration
        super().__init__(*args, **kwargs)

    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type', 'application/json')
        self.end_headers()
        logging.debug('state: ')
        logging.debug(_state)

        online_time = datetime.now() - timedelta(
            seconds=self.offline_diff_duration)
        response = {'active': _state['container_running']}

        cameras = {}

        for k, v in _state['last_log_times'].items():
            logging.debug(f'Comparing times: [{v}] and [{online_time}]')
            if online_time < v:
                cameras[k] = {"status": "running"}
            else:
                cameras[k] = {"status": "offline"}

        response['cameras'] = cameras

        logging.debug('response')
        logging.debug(response)
        self.wfile.write(json.dumps(response).encode())
        self.wfile.write(b'\n')


def parse_log_line(line):
    """
    :param line: Stream log line
    :return: Log split into [level, camera, time]
    """
    m = re.match(log_line_regex, line)
    if m:
        groups = m.groups()
        return [groups[0], groups[1], groups[2]]


def monitor_worker(container_name, check_interval):
    captures = 0
    previous_log_time = None

    while True:
        result = subprocess.run(
            ['docker', 'logs', '--tail', '1', container_name],
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT)

        docker_log = result.stdout.decode('utf-8')
        logging.debug(f'docker_log: {docker_log}')

        if 'No such container' in docker_log:
            _state['container_running'] = False
            time.sleep(check_interval)
            continue
        else:
            _state['container_running'] = True

        if len(docker_log) > 0:
            log_line = parse_log_line(docker_log)
            logging.debug(f'log_line: {log_line}')
            if log_line:
                log_time = log_line[2]
                if previous_log_time and log_time == previous_log_time:
                    logging.debug('No new logs detected')
                else:
                    camera = log_line[1]
                    _state['last_log_times'][camera] = datetime.now()
                    captures += 1
                    previous_log_time = log_time
        else:
            logging.debug('Log line empty')

        time.sleep(check_interval)
        logging.debug(f'Captures: {captures}')


def server_worker(host, port, offline_diff_duration):
    handler = partial(GetHandler, offline_diff_duration)
    server = HTTPServer((host, port), handler)
    logging.info('Starting server, use <Ctrl-C> to stop')
    server.serve_forever()


class Command(BaseCommand):
    help = "Monitor Stream HEALTH through it's logs. " \
           "Exposes a single API endpoint that returns the Camera Status:"

    def add_arguments(self, parser):
        parser.add_argument("-c",
                            "--container",
                            type=str,
                            default='stream',
                            help="Stream Container Name or ID")
        # Server listening HOST and PORT
        parser.add_argument("-l",
                            "--listen",
                            type=str,
                            default='localhost',
                            help="Server listen address")
        parser.add_argument("-p",
                            "--port",
                            type=int,
                            default=8002,
                            help="Server listen port")
        # Check Interval
        parser.add_argument("-i",
                            "--interval",
                            type=int,
                            default=2,
                            help="Interval between reading logs in seconds")
        # Active Duration
        parser.add_argument(
            "-d",
            "--duration",
            type=int,
            default=20,
            help="Duration to use in considering a camera as "
                 "offline in seconds")

    def handle(self, *args, **options):
        # Container Name
        monitor = threading.Thread(target=monitor_worker,
                                   args=(
                                       options['container'],
                                       options['interval']
                                   ))
        server = threading.Thread(target=server_worker,
                                  args=(
                                      options['listen'],
                                      options['port'],
                                      options['duration'],
                                  ))
        monitor.start()
        server.start()

[danleyb2]

@lukenuyen02 Your version of the script works but doesn't list all the cameras. kindly try this instead:

# python stream_monitor.py --help
# python stream_monitor.py -c stream -i 30 -d 120
import re
import os
import time
import logging
import sys
import threading
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
import subprocess
from functools import partial
from datetime import datetime, timedelta
from django.core.management.base import BaseCommand


LOG_LEVEL = os.environ.get('LOGGING', 'INFO').upper()

logging.basicConfig(
    stream=sys.stdout,
    level=LOG_LEVEL,
    format='%(asctime)s %(levelname)s %(name)s %(threadName)s : %(message)s')

_state = {'container_active': False, 'last_log_times': {}}

log_line_regex = r'(\w+):([^:]*):(.*):\b'
compiled = re.compile(log_line_regex)


class GetHandler(BaseHTTPRequestHandler):

    def __init__(self, offline_diff_duration, *args, **kwargs):
        self.offline_diff_duration = offline_diff_duration
        super().__init__(*args, **kwargs)

    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type', 'application/json')
        self.end_headers()
        logging.debug('state: ')
        logging.debug(_state)

        online_time = datetime.now() - timedelta(
            seconds=self.offline_diff_duration)
        response = {'active': _state['container_running']}

        cameras = {}

        for k, v in _state['last_log_times'].items():
            logging.debug(f'Comparing times: [{v}] and [{online_time}]')
            if online_time < v:
                cameras[k] = {"status": "running"}
            else:
                cameras[k] = {"status": "offline"}

        response['cameras'] = cameras

        logging.debug('response')
        logging.debug(response)
        self.wfile.write(json.dumps(response).encode())
        self.wfile.write(b'\n')


def parse_log_line(line):
    """
    :param line: Stream log line
    :return: Log split into [level, camera, time]
    """
    m = re.match(log_line_regex, line)
    if m:
        groups = m.groups()
        return [groups[0], groups[1], groups[2]]


def monitor_worker(container_name, check_interval):
    captures = 0
    previous_log_time = None

    while True:
        result = subprocess.run(
            ['docker', 'logs', '--tail', '10', container_name],
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT)

        docker_log = result.stdout.decode('utf-8')
        logging.debug(f'docker_log: {docker_log}')

        if 'No such container' in docker_log:
            _state['container_running'] = False
            time.sleep(check_interval)
            continue
        else:
            _state['container_running'] = True

        if len(docker_log) > 0:
            for line in docker_log.splitlines():
                log_line = parse_log_line(line)
                logging.debug(f'log_line: {log_line}')
                if log_line:
                    log_time = log_line[2]
                    if previous_log_time and log_time == previous_log_time:
                        logging.debug('No new logs detected')
                    else:
                        camera = log_line[1]
                        _state['last_log_times'][camera] = datetime.now()
                        captures += 1
                        previous_log_time = log_time
        else:
            logging.debug('Log line empty')

        time.sleep(check_interval)
        logging.debug(f'Captures: {captures}')


def server_worker(host, port, offline_diff_duration):
    handler = partial(GetHandler, offline_diff_duration)
    server = HTTPServer((host, port), handler)
    logging.info('Starting server, use <Ctrl-C> to stop')
    server.serve_forever()



class Command(BaseCommand):
    help = "Monitor Stream HEALTH through it's logs. " \
           "Exposes a single API endpoint that returns the Camera Status:"

    def add_arguments(self, parser):
        parser.add_argument("-c",
                            "--container",
                            type=str,
                            default='stream',
                            help="Stream Container Name or ID")
        # Server listening HOST and PORT
        parser.add_argument("-l",
                            "--listen",
                            type=str,
                            default='localhost',
                            help="Server listen address")
        parser.add_argument("-p",
                            "--port",
                            type=int,
                            default=8002,
                            help="Server listen port")
        # Check Interval
        parser.add_argument("-i",
                            "--interval",
                            type=int,
                            default=2,
                            help="Interval between reading logs in seconds")
        # Active Duration
        parser.add_argument(
            "-d",
            "--duration",
            type=int,
            default=20,
            help="Duration to use in considering a camera as "
                 "offline in seconds")

    def handle(self, *args, **options):
        # Container Name
        monitor = threading.Thread(target=monitor_worker,
                                   args=(
                                       options['container'],
                                       options['interval']
                                   ))
        server = threading.Thread(target=server_worker,
                                  args=(
                                      options['listen'],
                                      options['port'],
                                      options['duration'],
                                  ))
        monitor.start()
        server.start()

[lukenuyen02]

@danleyb2 I will test this as soon as possible. if you dont heard back from for 1 or 2 days you can close the ticket

Thank you in advance for your time and assistance

[danleyb2]

@lukenuyen02 if it doesn't work let's jump on screen share session. It will make it easier

[danleyb2]

@lukenuyen02 I can confirm the code i had shared or this PR fixes the issue you were having with multiple cameras.
#125

I'll close the issue and we can re-open another if you still facing it