How to get templates in parse_by_Official() function of class HDFSLoader about plelog

[LeonYang95]

Hi @tunglambk,
Sure, its my pleasure!

Such "ground truth" of HDFS log messages are downloaded from the homepage of Prof. Wei Xu. And we manually moved it into the python project.

According to our experiences with log parsers, I think it is safe to say that those were manually written or extracted from the source code of the target system. Log parsers are highly likely to introduce noises.

Hope this will help, if anything, feel free to comment and I am happy to answer :P

[tunglambk]

Hi Dr.@YangLin-George ,

Thank you so much for your enthusiastic support.

From your explanation, I understand that if we can extract the log templates from the source code, we should use them like the parse_by_Official() function to avoid noise from the log parser. Otherwise, we can use IBM Drain3 to parse logs like the parse_by_IBM() function. Is this right?

[LeonYang95]

In my opinion, while doing research, log parsing seems necessary. For example, the widely used BGL dataset does not have such a "ground truth" like HDFS.

However, while you are using an anomaly detection approach in industry, if the real templates are available, its better to use them directly.

[tunglambk]

Hi Dr. @YangLin-George,

Yeah, I understood. Besides, I have one more question.

For example, the widely used BGL dataset does not have such a "ground truth" like HDFS.

The parse_by_Official() function of the BGLLoader class also has templates like HDFSLoader. How do you extract it? Thank you so much.

[LeonYang95]

Those were written based on the tempaletes generated by Drain. We tried to study the impact of parsing and prepared those.

[tunglambk]

Thank you so much for your support and great work, Dr. @YangLin-George, I close this issue here.