noobaa log is not performing logrotate about noobaa-core

[naveenpaul1]

@rkomandu please check the logrotation when it hits the size limit in your node, Last time when you did the upgrade log size was already larger than configured value and I believe logrotate failed because of that. If logrotate successful when noobaa.log size reaches 105M then this issue should be invalid otherwise we need to fix this issue.

[rkomandu]

Ran IO workload onto the app2 node serving the IP, logrotate didn't happen

[root@c83f1-app2 ~]# ls -lh /var/log/noobaa*
-rw-r--r--. 1 root root  18M Jun 18 10:14 /var/log/noobaa_events.log
-rw-r--r--. 1 root root 100M Jun 18 23:46 /var/log/noobaa.log
-rw-r--r--. 1 root root 6.1M Jun 14 06:44 /var/log/noobaa.log.1.gz
[root@c83f1-app2 ~]# date
Tue Jun 18 11:46:38 PM EDT 2024


Every 30.0s: ls -lh /var/log/noobaa*                                                                                      c83f1-app2: Wed Jun 19 00:04:49 2024

-rw-r--r--. 1 root root  18M Jun 18 23:46 /var/log/noobaa_events.log
-rw-r--r--. 1 root root 105M Jun 19 00:04 /var/log/noobaa.log
-rw-r--r--. 1 root root 6.1M Jun 14 06:44 /var/log/noobaa.log.1.gz

Every 30.0s: ls -lh /var/log/noobaa*                                                                                      c83f1-app2: Wed Jun 19 00:14:19 2024

-rw-r--r--. 1 root root  18M Jun 18 23:46 /var/log/noobaa_events.log
-rw-r--r--. 1 root root 106M Jun 19 00:13 /var/log/noobaa.log
-rw-r--r--. 1 root root 6.1M Jun 14 06:44 /var/log/noobaa.log.1.gz

[rkomandu]

rsyslog on the node

systemctl status rsyslog
● rsyslog.service - System Logging Service
     Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-06-14 06:44:09 EDT; 4 days ago
       Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
   Main PID: 1517556 (rsyslogd)
      Tasks: 3 (limit: 3297520)
     Memory: 267.3M
        CPU: 5min 57.981s
     CGroup: /system.slice/rsyslog.service
             └─1517556 /usr/sbin/rsyslogd -n

Jun 18 23:52:47 c83f1-app2 rsyslogd[1517556]: imjournal: 29754 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Jun 18 23:54:15 c83f1-app2 rsyslogd[1517556]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 18 23:55:16 c83f1-app2 rsyslogd[1517556]: imjournal from <c83f1-app2:node>: begin to drop messages due to rate-limiting
Jun 19 00:02:48 c83f1-app2 rsyslogd[1517556]: imjournal: 60938 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Jun 19 00:05:15 c83f1-app2 rsyslogd[1517556]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]
Jun 19 00:05:17 c83f1-app2 rsyslogd[1517556]: imjournal from <c83f1-app2:->: begin to drop messages due to rate-limiting
Jun 19 00:12:49 c83f1-app2 rsyslogd[1517556]: imjournal: 60688 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Jun 19 00:13:00 c83f1-app2 rsyslogd[1517556]: file size limit cmd for file '/var/log/noobaa.log' did no resolve situation  [v8.2102.0-117.el9]
Jun 19 00:15:16 c83f1-app2 rsyslogd[1517556]: imjournal from <c83f1-app2:node>: begin to drop messages due to rate-limiting
Jun 19 00:16:12 c83f1-app2 rsyslogd[1517556]: imjournal: journal files changed, reloading...  [v8.2102.0-117.el9 try https://www.rsyslog.com/e/0 ]