koyong / ticky_check

Generates two different reports from internal ticketing system log file i.e., syslog.log.

Ticky Check

Generate reports

Now, we're going to practice creating a script, named ticky_check.py, that generates two different reports from this internal ticketing system log file i.e., syslog.log. This script will create the following reports:

The ranking of errors generated by the system: A list of all the error messages logged and how many times each error was found, sorted by the most common error to the least common error. This report doesn't take into account the users involved. The user usage statistics for the service: A list of all users that have used the system, including how many info messages and how many error messages they've generated. This report is sorted by username.

To create these reports write a python script named ticky_check.py. Use nano or vim editor for this.

Python Script

nano ticky_check.py or vim ticky_check.py

Set interpreter.

#!/usr/bin/env python3

Here's your challenge:

Write a script to generate two different reports based on the ranking of errors generated by the system and the user usage statistics for the service. You'll write the script on your own, but we'll guide you throughout.

First:

Import all the Python modules that you'll use in this Python script. After importing the necessary modules, initialize two dictionaries: one for the number of different error messages and another to count the number of entries for each user (splitting between INFO and ERROR).

Second

Now, parse through each log entry in the syslog.log file by iterating over the file.

For each log entry, you'll have to first check if it matches the INFO or ERROR message formats. You should use regular expressions for this. When you get a successful match, add one to the corresponding value in the per_user dictionary. If you get an ERROR message, add one to the corresponding entry in the error dictionary by using proper data structure.

To complete this part reference exercise 1, first you'll see it how to do it using grep but we need to do this part using python regex (re) module:

Reference Exercise 1

We'll be working with a log file named syslog.log, which contains logs related to ticky.

You can view this file using:

cat syslog.log The log lines follow a pattern similar to the ones we've seen before. Something like this:

May 27 11:45:40 ubuntu.local ticky: INFO: Created ticket [#1234] (username)

Jun 1 11:06:48 ubuntu.local ticky: ERROR: Connection to DB failed (username)

When the service runs correctly, it logs an INFO message to syslog. It then states what it did and states the username and ticket number related to the event. If the service encounters a problem, it logs an ERROR message to syslog. This error message indicates what was wrong and states the username that triggered the action that caused the problem.

In this section, we'll search and view different types of error messages. The error messages for ticky details the problems with the file with a timestamp for when each problem occurred.

These are a few kinds of listed error:

Timeout while retrieving information The ticket was modified while updating Connection to DB failed Tried to add information to a closed ticket Permission denied while closing ticket Ticket doesn't exist To grep all the logs from ticky, use the following command:

grep ticky syslog.log

In order to search all the ERROR logs, use the following command:

grep "ERROR" syslog.log

To enlist all the ERROR messages of specific kind use the below syntax.

• Syntax: grep ERROR [message] [file-name]

grep "ERROR Tried to add information to closed ticket" syslog.log

We can also grep the ERROR/INFO messages in a pythonic way using a regular expression. Let's now write a few regular expressions using a python3 interpreter.

Now using Regex:

Import the regular expression module (re).

import re

• line = "May 27 11:45:40 ubuntu.local ticky: INFO: Created ticket [#1234] (username)" To match a string stored in line variable, we use the search() method by defining a pattern.

re.search(r"ticky: INFO: ([\w ]*) ", line) Output:

• <_sre.SRE_Match object; span=(29, 57), match='ticky: INFO: Created ticket '> You can also get the ERROR message as we did for the INFO log above from the ERROR log line.

• line = "May 27 11:45:40 ubuntu.local ticky: ERROR: Error creating ticket [#1234] (username)" To match a string stored in a line variable, we use the search() method by defining a pattern.

re.search(r"ticky: ERROR: ([\w ]*) ", line) Output:

• <_sre.SRE_Match object; span=(29, 65), match='ticky: ERROR: Error creating ticket '> Now that you know how to use regular expressions with Python, start fetching logs of ticky for a specific username. We'll need them in later sections.

Third

After you've processed the log entries from the syslog.log file, you need to sort both the per_user and error dictionary before creating CSV report files.

Keep in mind that:

The error dictionary should be sorted by the number of errors from most common to least common. The user dictionary should be sorted by username. Insert column names as ("Error", "Count") at the zero index position of the sorted error dictionary. And insert column names as ("Username", "INFO", "ERROR") at the zero index position of the sorted per_user dictionary.