Welcome to our deep dive into LogAI, a comprehensive open-source library designed to enhance your log analytics and intelligence capabilities. Let’s explore the functionalities of LogAI and guide you through the installation and usage in a friendly manner.
Table of Contents
- Introduction
- Installation
- Getting Started
- Documentation
- Technical Report and Citing LogAI
- Contact
- License
Introduction
LogAI serves as a versatile tool for a variety of log analytics tasks like summarization, clustering, and anomaly detection, all while supporting the OpenTelemetry data model for seamless compatibility with other log management platforms. With its user-friendly GUI and integrated models, LogAI simplifies complex processes, helping both academics and industry professionals alike.
Installation
Quick Install
Installing LogAI is straightforward. Here’s how you can do it:
# Check out LogAI code repo from Github
git clone https://github.com/salesforce/logai.git
cd logai
# [Optional] Create virtual environment
python3 -m venv venv
source venv/bin/activate
# Install LogAI
pip install logai
Install Optional Dependencies
If you want to take advantage of extended functionalities, here’s how you can install optional dependencies:
- Deep Learning Log Analysis:
pip install logai[deep-learning] - Enable LogAI GUI Portal:
pip install logai[gui] - LogAI Development:
pip install logai[dev] - Complete Installation:
pip install logai[all]
Note: You may encounter the warning about “Resource punkt not found.” If that happens, simply execute: python -m nltk.downloader punkt.
Getting Started
To get a feel for LogAI, you can start by exploring its GUI portal.
Explore LogAI GUI Portal
Launch the GUI portal with the following commands:
# Check out LogAI code repo from Github
git clone https://github.com/salesforce/logai.git
cd logai
# [Optional] Create virtual environment
python3 -m venv venv # create virtual environment
source venv/bin/activate # activate virtual env
# Install LogAI and GUI dependencies
pip install .[dev]
pip install .[gui]
# Start LogAI service
export PYTHONPATH=. # make sure to add current root to PYTHONPATH
python3 gui/application.py # Run local plotly dash server
Then, open the LogAI portal using http://localhost:8050 in your browser.
Control Panel
The control panel features applications for:
- Log Summarization: Summarizes and groups raw logs according to specific patterns.
- Log Clustering: Groups logs into clusters based on semantic representation.
- Anomaly Detection: Detects anomalies within logs through various algorithms.
Run Simple Time-series Anomaly Detection Application
Below is a configuration setup example for running an anomaly detection application:
{
"open_set_data_loader_config": {
"dataset_name": "HDFS",
"filepath": ""
},
"preprocessor_config": {
"custom_delimiters_regex": []
},
"log_parser_config": {
"parsing_algorithm": "drain",
"parsing_algo_params": {
"sim_th": 0.5,
"depth": 5
}
},
"feature_extractor_config": {
"group_by_category": ["Level"],
"group_by_time": "1s"
},
"log_vectorizer_config": {
"algo_name": "word2vec"
},
"categorical_encoder_config": {
"name": "label_encoder"
},
"anomaly_detection_config": {
"algo_name": "one_class_svm"
}
}
Then, simply run the following Python script to execute the application:
import json
from logai.applications.application_interfaces import WorkFlowConfig
from logai.applications.log_anomaly_detection import LogAnomalyDetection
# Load configuration file
json_config = ""
with open(json_config) as file:
config = json.load(file)
# Create Log Anomaly Detection Application
workflow_config = WorkFlowConfig.from_dict(config)
app = LogAnomalyDetection(workflow_config)
# Execute App
app.execute()
After execution, check the detection results with app.anomaly_results.
Build Customized LogAI Applications
You can create tailored log analysis applications using LogAI, for which tutorials are readily available. Here are two brilliant examples:
Deep-learning Anomaly Detection Benchmarking
LogAI also facilitates the benchmarking of deep-learning algorithms for anomaly detection, with tutorials available for guidance.
Documentation
For comprehensive detail about the LogAI library, visit the LogAI Documentation.
Technical Report and Citing LogAI
Find more details in our technical report.
Contact
If questions or suggestions arise, please contact us at logai@salesforce.com.
License
This project is licensed under the BSD 3-Clause License.
Troubleshooting
If you run into issues, here are some tips:
- Ensure compatibility with OpenTelemetry data models.
- If you face resource errors, try updating your NLTK library as mentioned earlier.
- Check configuration files for any syntax errors before running the application.
For more insights, updates, or to collaborate on AI development projects, stay connected with **fxis.ai**.
At **fxis.ai**, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.
Analogy for Understanding LogAI Structure
Think of LogAI as a Swiss army knife, specifically designed for log analytics. Just as a Swiss army knife includes multiple tools for different scenarios, LogAI offers a variety of functionalities ranging from log summarization to anomaly detection.
- Log Summarization: Like the blade of the knife, it cuts through messy logs to extract key patterns.
- Log Clustering: As the screwdriver helps fit things together, this function clusters logs based on their semantics.
- Anomaly Detection: Similar to the magnifying glass feature, this sharp-eyed tool allows you to spot deviations and irregularities in your logs.
Embrace the power of LogAI to streamline your log analytics and intelligence efforts!
