# Real-Time Elasticsearch-Kibana Logs Monitoring with Microsoft Teams & Slack Alerts

In today's world, it's crucial to monitor your application logs and get alerted about suspicious or critical activities. If you are using Kibana for log monitoring and want to get real-time alerts on Microsoft Teams or Slack, this guide is for you. This script will help you automate log monitoring and send notifications to your desired platform.

## Why is this Needed?

When you are monitoring logs from applications, it’s important to be aware of potential security breaches, errors, or unexpected behavior. For example, if your application generates an alert like `Elastic-Alert-Critical`, you want to be notified immediately so you can take action. This script can be customized to monitor various patterns like `Elastic-Alert-High`, `Elastic-Alert-Medium`, etc., based on your requirements.

### Prerequisites

* Kibana server access to create rules for indexes.
    
* A Kibana instance generating logs at `/var/log/kibana/kibana.log` or another location.
    
* Microsoft Teams and/or Slack Webhook URL for receiving alerts.
    
* Basic understanding of shell scripting.
    

## Script to Monitor Logs and Send Alerts

Below is the script that monitors your Kibana logs and sends alerts to both Microsoft Teams and Slack. You can choose to remove either part if you only want to use one of them.

```bash
#!/bin/bash

# Microsoft Teams Webhook URL
TEAMS_WEBHOOK_URL="https://webltd.webhook.office.com/webhookb2/..."

# Slack Webhook URL
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXX/XXXXX/XXXXX"

# Log file to monitor
LOG_FILE="/var/log/kibana/kibana.log"

# Function to send message to Microsoft Teams
send_to_teams() {
    local message=$1

    escaped_message=$(echo "$message" | sed 's/\\/\\\\/g' | sed 's/"/\\"/g')

    local payload=$(cat <<EOF
{
    "@type": "MessageCard",
    "@context": "http://schema.org/extensions",
    "summary": "Log Alert",
    "themeColor": "0076D7",
    "title": "Kibana-Alert from prod",
    "text": "$escaped_message"
}
EOF
)

    curl -H "Content-Type: application/json" -d "$payload" $TEAMS_WEBHOOK_URL
}

# Function to send message to Slack
send_to_slack() {
    local message=$1

    local payload=$(cat <<EOF
{
    "text": "$message"
}
EOF
)

    curl -X POST -H "Content-Type: application/json" -d "$payload" $SLACK_WEBHOOK_URL
}

# Monitor log file for patterns

tail -F "$LOG_FILE" | grep --line-buffered -E "Elastic-Alert|Elastic-Alert-Critical|Elastic-Alert-High|Elastic-Alert-Medium" | while read line
    do
        send_to_teams "$line"
        send_to_slack "$line"
    done
```

### Explanation

* The script continuously monitors the Kibana log file using `tail -F` and filters lines containing patterns like `Elastic-Alert`, `Elastic-Alert-Critical`, etc.
    
* It sends notifications to both Microsoft Teams and Slack by making `curl` requests with formatted messages.
    
* You can easily customize this to look for other patterns or integrate other tools.
    

#### Modify the script to send only `Elastic-Alert-Critical` to Slack

```bash
tail -F "$LOG_FILE" | grep --line-buffered -E "Elastic-Alert-Critical" | while read line
    do
         send_to_slack "$line"
    done
```

### Creating a Background Service

If you want this script to run continuously, even after a reboot, you can create a systemd service.

```bash
sudo vim /etc/systemd/system/kibana-alert.service
```

Add the following content:

```ini
[Unit]
Description=Kibana Log Monitoring Service
After=network.target

[Service]
ExecStart=/bin/bash /path/to/your-script.sh
Restart=always

[Install]
WantedBy=multi-user.target
```

### Enable and Start the Service

```bash
sudo systemctl daemon-reload
sudo systemctl enable kibana-alert.service
sudo systemctl start kibana-alert.service
```

### Checking Service Status

```bash
sudo systemctl status kibana-alert.service
```

## Customization Tips

* If you only want to use Microsoft Teams or Slack, simply remove the corresponding part of the script.
    
* For example, if you want to match other critical tags like `Elastic-Alert-Critical`, just add them to the `grep` command as shown above.
    

## Steps to Add Rules in Kibana Server UI

1. **Log in to Kibana**  
    Open your browser and go to the Kibana URL (e.g., [`http://localhost:5601`](http://localhost:5601)).
    
2. **Navigate to Stack Management**
    
    * Click on the menu icon (☰) in the top-left corner.
        
    * Go to **"Stack Management"** under the **"Management"** section.
        
        ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742662929755/b022be0b-c612-4489-a2a3-c39579330195.png align="center")
        
3. **Create Detection Rules**
    
    * Click on **"Alerts and Insights"**.
        
    * Go to **"Rules"**.
        
        ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742662959884/a10771b9-a3a1-4143-8cfd-bfa69c397e4d.png align="center")
        
4. **Create a New Rule**
    
    * Click the **"Create new rule"** button.
        
    * Choose the type of rule you want to create. For example:
        
        * **"Custom Query"** if you want to search logs with specific keywords like `Elastic-Alert` or `Elastic-Alert-Critical`.
            
        * **"Threshold"** if you want alerts triggered when specific conditions are met.
            
        * I am selecting `Elasticsearch query`
            
            ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742663488589/876d9236-786c-4732-9de1-5a55bbc89c68.png align="center")
            
5. **Define the Rule**
    
    * **Rule Name:** Give your rule a meaningful name, like `Elastic-Alert-Detection`.
        
    * **Index pattern:** Select your Kibana index pattern, usually `filebeat-*` or `logstash-*`.
        
        ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742663516027/5e1b908d-66f7-47f3-af36-43bbebadc681.png align="center")
        
    * **Custom Query:** Add your query, e.g.,
        
        ```bash
        kqlCopyEditmessage: "Elastic-Alert" OR message: "Elastic-Alert-Critical"
        ```
        
    * **Rule Schedule:** Set how often the rule will check for alerts (e.g., every 1 minute).
        
        ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742663521407/59e74e83-6272-45f4-a6bc-70b80cab573c.png align="center")
        
6. **Set Up Actions (Optional)**
    
    * You can add actions to trigger when a rule matches.
        
    * For example, you can send notifications to an email, webhook, or other services.
        
        ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742663536890/9e7efd10-3ffd-4855-bc59-852863ce8af8.png align="center")
        
7. S**elect the connector type.**
    
    * Select the connector type as **Server log.**
        
        ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742663993441/15ccb213-f9db-438d-ab52-088a8e223476.png align="center")
        
8. **Review and Create**
    
    * Review your configuration and click on **"Create rule"**.
        
9. **Monitor and Test**
    
    * Go to **"Detections & Alerts"** to monitor triggered alerts.
        
    * Test by generating logs with the keywords you set in the rule.
        

Feel free to copy this script and make changes according to your requirements. If you face any issues or need help setting this up, drop a comment below! 😊  

#### Useful Links

* [GitHub](https://github.com/yogeshgowdagr)
    
* [Blog](https://blogs.q21.in/)
    
* [Portfolio](https://yogesh.q21.in)
    
* [LinkedIn](https://www.linkedin.com/in/yogesh-gowda-gr/)
    
* [Buy Me a Coffee](https://buymeacoffee.com/yogeshgowdq)