LogPAI

Loghub

A large collection of system log datasets for AI-powered log analytics.

Logparser

A toolkit for automated log parsing [ICSE'19, TDSC'18, DSN'16].

Loglizer

A log analysis toolkit for automated anomaly detection [ISSRE'16].

Log3C

Log-based impactful problem identification using machine learning [FSE'18].

DOWNLOAD DATASETS FOLLOW GITHUB REPO

PUBLICATIONS

[ArXiv'21] Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael R. Lyu. Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection, arXiv preprint, arXiv:2107.05908 (2021).

[ASE'19] Jinyang Liu, Jieming Zhu, Shilin He, Pinjia He, Zibin Zheng, Michael R. Lyu. Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression, in Proc. of IEEE/ACM International Conference on Automated Software Engineering (ASE), 2019.

[ICSE'19] Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, Michael R. Lyu. Tools and Benchmarks for Automated Log Parsing, in Proc. of International Conference on Software Engineering (ICSE), 2019.

[FSE'18] Shilin He, Qingwei Lin, Jian-Guang Lou, Hongyu Zhang, Michael R.Lyu, Dongmei Zhang. Identifying Impactful Service System Problems via Log Analysis, in Proc. of the 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), 2018.

[ASE'18] Pinjia He, Zhuangbin Chen, Shilin He, Michael R. Lyu. Characterizing the Natural Language Descriptions in Software Logging Statements, in Proc. of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE), 2018.

[TDSC'18] Pinjia He, Jieming Zhu, Shilin He, Jian Li, Michael R. Lyu, Towards Automated Log Parsing for Large-Scale Log Data Analysis, IEEE Transactions on Dependable and Secure Computing (TDSC), 2018.

[ICWS'17] Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu. Drain: An Online Log Parsing Approach with Fixed Depth Tree, in Proc. of the 24th International Conference on Web Services (ICWS), 2017.

[DSN'16] Pinjia He, Jieming Zhu, Shilin He, Jian Li, Michael R. Lyu. An Evaluation Study on Log Parsing and Its Use in Log Mining, in Proc. of IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), 2016.

[ISSRE'16] Shilin He, Jieming Zhu, Pinjia He, Michael R. Lyu. Experience Report: System Log Analysis for Anomaly Detection, in Proc. of IEEE International Symposium on Software Reliability Engineering (ISSRE), 2016.

[ICSE'15] Jieming Zhu, Pinjia He, Qiang Fu, Hongyu Zhang, Michael R. Lyu, Dongmei Zhang. Learning to Log: Helping Developers Make Informed Logging Decisions, in Proc. of International Conference on Software Engineering (ICSE), 2015.

[ICSE'14] Qiang Fu, Jieming Zhu, Wenlu Hu, Jian-Guang Lou, Rui Ding, Qingwei Lin, Dongmei Zhang, and Tao Xie. Where Do Developers Log? An Empirical Study on Logging Practices in Industry, in Proc. of International Conference on Software Engineering (ICSE), 2014.

IMPACT

The ultimate goal of LogPAI is to build an open-source AI platform for automated log analysis. Towards this goal, we benchmark a set of research work as well as release open datasets and tools for log analysis research. With both datasets and source code available, we hope that our LogPAI project could benefit both researchers and practitioners in the community.

The loghub datasets have been downloaded 70,000+ times by more than 380 organizations from both industry and academia. The related publications have been cited more than 1,500 times according to Google Scholar.

Our log analysis tools have been used by industrial teams in Microsoft, Huawei, and IBM.