angoca / log4db2

A logging utility like log4j for IBM Db2 SQL PL

log4db2

Log4db2 is a logging utility for DB2 LUW that uses SQL instructions with SQL PL code.

Its purpose is to provide an easy way to write messages from a SQL routine, with the possibility to query these messages directly from the database, allowing to monitor the progression of a process. This utility aims to reduce the time used for developing, testing, debugging and monitoring SQL routines, by centralizing the messages produced by the code.

The idea and architecture of this utility are based on the popular Java logging utilities, such as Log4j and slf4j/logback.

• Log4j - http://logging.apache.org/log4j
• Logback/SLF4J - http://logback.qos.ch/ - http://www.slf4j.org/

The licenses of this project are:

• For the source code is "BSD 2-Clause license".
• For the documentation is "FreeBSD Documentation license."

With these two licenses you are free to use, modify and distribute any part of this utility.

Author:

Andres Gomez Casanova (@AngocA)

Links for more information

These are some useful links:

• The source code is hosted at: https://github.com/angoca/log4db2
• The wiki is at: https://github.com/angoca/log4db2/wiki
• The last released version is published at: https://github.com/angoca/log4db2/releases
• The issue tracker for comments and bugs is at: https://github.com/angoca/log4db2/issues

---

Installation

One variable needs to the specified in order to run the install and example scripts.

LOG4DB2_PATH

This variable is initialized via the init script.

Before installing the scripts in a database, a connection to it has to be established. If not, an error will be raised.

Linux/UNIX/MAC OS:

Just follow these steps:

tar -zxvf log4db2.tar.gz
cd log4db2
. ./install

Make sure to put the dot (source command) before the script. This will source the values and use the current connection.

Windows Terminal (CMD - db2clp):

First, unzip the file log4db2.zip, and then:

cd log4db2
init.bat
install.bat

Windows PowerShell:

First, unzip the file log4db2.zip, and then:

cd log4db2
.\init.ps1
.\install.ps1

Check install

After the install, all statements should have been successful.

A more detailed guide to install the utility can be found in the Install section of the wiki: https://github.com/angoca/log4db2/wiki/Install

You can also install the utility from the sources and run the examples and tests: https://github.com/angoca/log4db2/wiki/Install%20from%20sources

Once the utility is installed, you can customize the utility. For more information, please visit the configuration section: https://github.com/angoca/log4db2/wiki/Configuration

---

Usage

1. Write the code

This could be the structure of your routine's code (Procedure or function).

CREATE ... HELLO_WORLD ()
 MODIFIES SQL
 BEGIN
  DECLARE LOGGER_ID SMALLINT;
  -- Your declarations

  CALL LOGGER.GET_LOGGER('Your.Hierarchy', LOGGER_ID);
  -- Your code
  CALL LOGGER.INFO(LOGGER_ID, 'Your message');
  -- Your code
 END @

As you can see, there is a call to GET_LOGGER to register the logger, and get its id. Then, you write messages by providing the id and the text. That's all.

2. Invoke the code

Then, you invoke your code (Depending if it is a stored procedure or a function.)

CALL HELLO_WORLD(); -- Stored procedure.
VALUES HELLO_WORLD(); -- Function.

3. Check the results

This is the easiest way to check the log messages.

CALL LOGADMIN.LOGS();

From the CLP is:

db2 "CALL LOGADMIN.LOGS()"

Check the Usage section for more information about the levels, how to access the messages and configure the utility: https://github.com/angoca/log4db2/wiki/Usage

---

Files and directories

These are the files included in the released version:

• COPYING.txt -- License for the code (BSD license - OpenSource).
• init* -- Environment configuration.
• install* -- Installation files.
• README.txt -- This file.
• reinstall* -- Reinstallation files.
• tail_logs* -- Script to show the most recent logs.
• uninit* -- Clean environment.
• uninstall* -- Uninstallation files.
• doc directory -- Documentation directory (ErrorCode, ER diagram).
• examples directory -- Examples ready to run.
• sql-pl directory -- Directory for all objects: DDL, DML, routines definition.
  • 00-AdminObjects.sql -- Create admin objects: BP, TS and schema.
  • 05-Tables.sql -- Tables, DML statement and runstats.
  • 10-LogsTable.sql -- Logs table - regular.
  • 10-LogsTablePartitioned.sql -- Logs table - partitioned.
  • 15-UtilityHeader.sql -- Headers of the core tools.
  • 20-UtilityBody.sql -- Body of the core tools.
  • 25-Appenders.sql -- Definition of the appenders.
  • 30-Log.sql -- Log procedure definition with its macros.
  • 35-Get_Logger.sql -- GetLogger procedure definition for v10.1 or upper.
  • 40-Trigger.sql -- Trigger of the different tables.
  • 50-AdminHeader.sql -- Headers of the administration tools.
  • 55-AdminBody.sql -- Body of the administration tools.
  • 60-TablesTimeTravel.sql -- Modifications for Time Travel.
  • 65-Version.sql -- Version of the utility.
  • 96-CleanTriggers.sql -- Remove all triggers.
  • 97-CleanObjects.sql -- Remove all objects.
  • 98-CleanTables.sql -- Remove all tables.
  • 99-CleanAdmin.sql -- Remove all tables.
  • Appender_CGTT_Create.sql -- Appender for temporary tables.
  • Appender_CGTT_Drop.sql -- Drops appender for temporary tables.
  • Appender_UTL_FILE_Create.sql -- Appender to write in a file
  • Appender_UTL_FILE_Drop.sql -- Drop appender that write in a file.
• xml directory -- Directory for XML Schemas, XML files and related scripts for appenders configuration.
  • 45-AppendersXML.sql -- Registers the XML Schema.
  • Appender_UTL_FILE.csv -- Description of the appender..
  • Appender_UTL_FILE.xml -- Configuration for LOG_UTL_FILE appender.
  • conf_appender_X.xsd -- XML Schema for the X release.

The * in the install-related files means that several files for each name can be found:

• .bat -- Windows Batch file for CMD.exe
• .ps1 -- Windows PowerShell
• .sql -- For DB2 CLPPlus.
• No extension -- For Linux/UNIX/Mac OS X.

log4db2's Issues

Get diagnostics - DB2_SQL_NESTING_LEVEL

Use this command to get the nesting level to prevent error codes:

http://publib.boulder.ibm.com/infocenter/db2luw/v10r1/topic/com.ibm.db2.luw.sql.ref.doc/doc/r0005647.html

Hello world example

Make a very simple hello world example, with a mini manual to see what it does.

Intermediate loggers

There are two tables conf_loggers and conf_logger_effective. Put some loggers in the conf_loggers without the whole path. This is to test the use of conf_loggers_effective.

https://github.com/angoca/log4db2/wiki/Configuration#wiki-conf_loggers_effective
Probably the documentation is wrong, and just intermediate levels are defined but not associated with any level.

Modify plugin Assembly configuration to generate two files

Generate two files with the same content for each release

• log4db2.zip
• log4db2-1A.zip

This will allow to publish easily, leaving a trace of all published files (with version), and the other as the most recent version to download (the manual does not have to change)

Put comments in all objects

Stored procedures
Variables
Modules
Routines

The comments should be the same as the documentation.

http://pic.dhe.ibm.com/infocenter/db2luw/v10r5/topic/com.ibm.db2.luw.sql.ref.doc/doc/r0000901.html

Create a procedure to check the IDs of both logger tables

-- TODO Check the IDs of the conf table and the effective table, in order to
-- see if the effective ids correspond to conf ids.

Reduce the IO operations in LOG procedure

This method has two selects:

DECLARE APPENDERS CURSOR FOR
SELECT APPENDER_ID, CONFIGURATION, PATTERN
FROM LOGDATA.CONF_APPENDERS
WITH UR;

SELECT C.LEVEL_ID INTO CURRENT_LEVEL_ID
FROM LOGDATA.CONF_LOGGERS_EFFECTIVE C
WHERE C.LOGGER_ID = LOG_ID
WITH UR;

These two configurations can be retrieved from arrays, and be updated periodically.

Arrays can be used: http://www.ibm.com/developerworks/data/library/techarticle/dm-0710arocena/

Test layout

Write some layout test to see if they write what they should write.

Expected the handle
Actual the handle
Expected like the current date
Actual the current date
Expected a given level
Actual the given level

Put 2014 in lincese

Test for LOGS table with Not logged

Probably, if the table LOGS is not logged, the transaction will be less expensive. Also, when the transaction does a roll back, the logs still are in the table. They are not rollbacked.

Check the cache for the configuration table and other tables

It seems that the cache is only for loggers, not for the other components.

Test modify conf for inner logger

There could be any quantity of levels in the logger hierarchy. However, when the level of a logger is changed, the change applies to all descendants. This is a recursive call, and I think we can arrive to the limit of 16 cascade calls.

Perform a test of defining a logger with a big hierarchy, for example
A.B.C.D.E.F.G.H.I.J.K.L.M.N.O.P.Q.R.S.T.U.V.W.X.Y.Z
Y despues cambiar el root logger, para que tenga que cambar todos los niveles intermedios.

Si falla en el cambio,cambiar la documentacion e indicar que solo se permiten X cantidad de niveles

Error codes in the wiki

Copy and improve the content of the Error codes.

Appender with logger level

Appender configuration should include a level to use it. Not all messages could be send to any configured appender.
This reduces the consumption of an appender just by sending few messages.
For example a mail appender could be configured, but just few messages will be send there.

Add the option optimized for in the queries

-- TODO Add the optimized for

Create ChangeLog page

Create a page change log to describe the changes.

It will start once the alpha version is released.

Modify README file

The content of that file has been published in this wiki.

The current content should be the:

• Project description
• Installation guide (not from sources)
• Hello world guide

LOG procedure perform a 'select' on the level table

The LOG method should not make a select to the levels table in order to retrieve the name of the level. Those names should be in an associative array, and the content should be refreshed as part of the configuration refresh.

-- Inserts the level.
SET NEW_MESSAGE = REPLACE(NEW_MESSAGE, '%p', (
SELECT UCASE(COALESCE(L.NAME,'UNK'))
FROM LOGDATA.LEVELS L
WHERE L.LEVEL_ID = LEV_ID
WITH UR));

Procedure to insert a hierarchy

-- TODO Create a SP that register a logger with a given level. This will create
-- all the levels in the conf_loggers, and the relations.

Wiki features with support

For each feature, put links to the code to show what it is talking about.

Create a function to check the levels in the loggers table

• TODO Check if the logger levels between the conf and effective table are the
  -- same. In conf could be INFO but in effective could be WARN.

Branch: Remove the cache option

Remove the cache option, and use it as default.
In this way the operations are reduced.

This can be created as a new branch.
It is recommended to do this, once the refactoring has been done.

Check the default root level in getLogger

-- TODO To check the value defaultRootLevelId before assign Warn as default.
-- If the root logger is not defined, then set the default level: WARN-3.

Add the option fetch first n rows only in queries

-- TODO Add the fetch first N rows only

Create tickets for TODOs in code

There are many TODOs in the code.
They should be defined here as tickets.

Check multiples appenders

• Check if the definition of multiples appenders woks.
• Check that the reference table is taken into account.
• Check if only a part of the hierarchy could log only by defining it in the references table.
• Check multiple appenders (after implementing another). They both should write messages.

DemoBank splitted

The Demo Bank example has big anonymous block. Create stored procedures for each operation.

Write db2diag appender

db2AdminMsgWrite

Bufferpool & tablespace for configuration

The configuration is very small but used from time to time.

It is better to have a dedicated bufferpool to keep the values in memory and ensure that they will always be there. The arrays as for sessions, and they will get the information from the bufferpools. It is better to ensure that the information is in the bufferpool, and not cleaned implying a re-read from disk.

Put specific name for each routine

Each routine has to have a specific name

Test this function for the App ID

APPLICATION_ID
http://pic.dhe.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=%2Fcom.ibm.db2.luw.sql.rtn.doc%2Fdoc%2Fr0011856.html

Modify LOGGER.LOGS to retrieve the next logs

By defining a global variable, probably we can store the more recent log (probably the date), and then when the same procedure is executed it will return the rows generated after that (date).
It helps to execute this procedure several times and check the progress of a process (the new logs)

Variable usage with parallel execution

Tests if the variables are kept between different executions.
The modules define global variables, but I do not know if they are for all applications or just for the current one.
If they are for the current one, the cache is useless to have that mechanism to refresh it.
The performance will be impacted because each connection has to create its own set of variables and fill them with the corresponding values.

Create a procedure to check the hierarchy of the loggers

-- TODO Check the logger structure, in order to have different names for the
-- sons of a given father. root>toto root>tata root>tata is an error, and should
-- remove this duplicate. Probably implement with a check constraint.
-- 20140106 - This is already checked if the values are introduced via the
-- GET_LOGGER procedure. Probably, the conf table has to be checked in order
-- to prevent this kind of problems.

This could be triggered when the configuration is modified.

Procedure to check the logger name lenght

-- TODO Check the registered loggers in the database, calculating the maximum
-- length of the concatenated inner levels, and this lenght should be less than
-- 256 chars. foo.bar.toto

Arrays without cardinality

CREATE TYPE unboundedPhoneList AS VARCHAR(12) ARRAY[];

Conf values TODO

checkHierarchy: Checks the configuration defined in the CONF_LOGGERS table before load it. TODO This feature has not been yet implemented.
checkLevels: Checks the configruation defined in the LEVELS table before load it. TODO This feature has not been yet implemented.

Isolation level for queries

-- TODO Add the isolation level.

System output appender

This is send the messages to the standard output.

Write DB2 logger appender

http://www.ibm.com/developerworks/data/library/techarticle/dm-0601khatri/

Define a release procedure

Create a Wiki page to explain how to make a new release

• Execute tests
• Before release a new version, update the version variable.
• Publish the zip in sourceforge
• Write a note in the main page of github and ohloh about the new version.

Write file appender

It should receive the directory and filename

However, those values should be stored in variables in order to reduce the overhead of processing an XML each time this appender in used.
But, when the configuration is modified, the modifications are not reflected in these variables.

Write Java appender

XML
Java_logger
procedure name

Reorganize sources

Currently, the sql-pl part has many different files, and they do no reflect the internal organization.

Refactor those files

Put "made in" in all files

As part of the header documentation

Put utility

Replace

• framework
• application
• program
• software
• tool

TO utility

Configuration array created from select

the configuration array could be created directly from select

SET phoneList = ARRAY[SELECT phone FROM authors WHERE city = 'Toronto'];

Reqs to use this utility: modifies sql data

Describe in the usage that the routines should be configured as modifies sql data.

If not, an error will be raised, because log4db2 modifies sql data.

Test for auto generated column in LOGS

-- TODO make tests in order to check in a auto generated column for an id
-- does not impact the performance, and provides a better way to sort messages.
-- This ID column could be hidden to the user. The benefit is that the logs
-- could be accessed via an index, but it impacts the writes, because this
-- structure has to be maintained.

Check reference in LOG

-- TODO Verificar esto, ya que aqui se puede usar la tabla references, si root
-- no esta activo, ya que diferentes loggers pueden tener diferente conf.