LogPAI

Loghub

A large collection of system log datasets for AI-driven log analytics [ISSRE'23].

Logparser

A machine learning toolkit for log parsing [ICSE'19].

Loglizer

A machine learning toolkit for log-based anomaly detection [ISSRE'16].

Logzip

An optimized log compression tool via iterative clustering [ASE'19].

DOWNLOAD DATASETS FOLLOW ON GITHUB

PUBLICATIONS

[ISSTA'24] Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Jieming Zhu, Michael R. Lyu. A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?, in Proc. of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2024.

[FSE'24] Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu and Michael R. Lyu. LILAC: Log Parsing using LLMs with Adaptive Parsing Cache, in Proc. of The ACM International Conference on the Foundations of Software Engineering (FSE), 2024.

[ISSRE'23] Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, and Michael R. Lyu. Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics, in Proc. of IEEE International Symposium on Software Reliability Engineering (ISSRE), 2023.

[ArXiv'21] Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael R. Lyu. Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection, arXiv preprint, arXiv:2107.05908, 2021.

[ASE'19] Jinyang Liu, Jieming Zhu, Shilin He, Pinjia He, Zibin Zheng, Michael R. Lyu. Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression, in Proc. of IEEE/ACM International Conference on Automated Software Engineering (ASE), 2019.

[ICSE'19] Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, Michael R. Lyu. Tools and Benchmarks for Automated Log Parsing, in Proc. of International Conference on Software Engineering (ICSE), 2019.

[FSE'18] Shilin He, Qingwei Lin, Jian-Guang Lou, Hongyu Zhang, Michael R.Lyu, Dongmei Zhang. Identifying Impactful Service System Problems via Log Analysis, in Proc. of the 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2018.

[ASE'18] Pinjia He, Zhuangbin Chen, Shilin He, Michael R. Lyu. Characterizing the Natural Language Descriptions in Software Logging Statements, in Proc. of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE), 2018.

[TDSC'18] Pinjia He, Jieming Zhu, Shilin He, Jian Li, Michael R. Lyu, Towards Automated Log Parsing for Large-Scale Log Data Analysis, IEEE Transactions on Dependable and Secure Computing (TDSC), 2018.

[ICWS'17] Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu. Drain: An Online Log Parsing Approach with Fixed Depth Tree, in Proc. of the 24th International Conference on Web Services (ICWS), 2017.

[DSN'16] Pinjia He, Jieming Zhu, Shilin He, Jian Li, Michael R. Lyu. An Evaluation Study on Log Parsing and Its Use in Log Mining, in Proc. of IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), 2016.

[ISSRE'16] Shilin He, Jieming Zhu, Pinjia He, Michael R. Lyu. Experience Report: System Log Analysis for Anomaly Detection, in Proc. of IEEE International Symposium on Software Reliability Engineering (ISSRE), 2016.

[ICSE'15] Jieming Zhu, Pinjia He, Qiang Fu, Hongyu Zhang, Michael R. Lyu, Dongmei Zhang. Learning to Log: Helping Developers Make Informed Logging Decisions, in Proc. of International Conference on Software Engineering (ICSE), 2015.

[ICSE'14] Qiang Fu, Jieming Zhu, Wenlu Hu, Jian-Guang Lou, Rui Ding, Qingwei Lin, Dongmei Zhang, and Tao Xie. Where Do Developers Log? An Empirical Study on Logging Practices in Industry, in Proc. of International Conference on Software Engineering (ICSE), 2014.

IMPACT

The ultimate goal of LogPAI is to build an open-source AI platform for automated log analysis. Towards this goal, we benchmark a set of research work as well as release open datasets and tools for log analysis research. With both datasets and source code available, we hope that our LogPAI project could benefit both researchers and practitioners in the community.

The loghub datasets have received a total of

by more than 450 organizations from both industry and academia. The related publications have been cited more than 1,500 times according to Google Scholar.

Our log analysis datasets and tools have been used by industrial teams in Microsoft, Huawei, IBM, Ant Group.