Skip to content

KIBANA_VISUALIZATION_GUIDE.md

Latest commit

 

History

History
528 lines (400 loc) · 13.5 KB

KIBANA_VISUALIZATION_GUIDE.md

File metadata and controls

528 lines (400 loc) · 13.5 KB

Kibana Live Metrics Visualization Guide

Table of Contents

  1. Accessing Kibana
  2. Setting Up Index Patterns
  3. Creating Visualizations
  4. Building Dashboards
  5. Real-time Metrics to Track
  6. Auto-refresh Configuration
  7. Cloud vs Local Differences

Accessing Kibana

Local Docker Deployment

  1. Start Services:

    docker-compose up -d kibana

  2. Access Kibana:

    • URL: http://localhost:5601
    • Wait 1-2 minutes for initial startup
    • No authentication required (local setup)

Cloud Elasticsearch Deployment

  1. Get Kibana URL:

    • Log into Elastic Cloud Console
    • Navigate to your deployment
    • Copy the Kibana endpoint (e.g., https://xxxxx.kb.us-east-1.aws.found.io:9243)

  2. Access Kibana:

    • Open the Kibana URL in your browser
    • Login with your Elastic Cloud credentials
    • Or use API Key authentication

Setting Up Index Patterns

Step 1: Create Index Pattern

  1. Navigate to Index Patterns:

    • Go to Stack Management → Index Patterns
    • Click Create index pattern

  2. Configure Pattern:

    • Index pattern name: phish-*
    • This matches all indices: phish-us, phish-eu, phish-asia, phish-global
    • Click Next step

  3. Select Time Field:

    • Time field: Select timestamp
    • This enables time-based filtering and time-series visualizations
    • Click Create index pattern

Step 2: Verify Index Pattern

  • You should see all matching indices listed
  • Verify field mappings are correct:
    • url (keyword)
    • risk_score (float)
    • region (keyword)
    • timestamp (date)

Creating Visualizations

Visualization 1: Risk Score Distribution (Histogram)

Purpose: Show distribution of risk scores across all reports

  1. Create Visualization:

    • Go to Analytics → Visualize Library
    • Click Create visualization
    • Select Vertical Bar chart

  2. Configure:

    • Data source: Select phish-* index pattern
    • Y-axis:
      • Aggregation: Count
      • Custom label: "Number of Reports"
    • X-axis:
      • Aggregation: Histogram
      • Field: risk_score
      • Interval: 0.1 (buckets of 0.1)
      • Custom label: "Risk Score"

  3. Save:

    • Click Save (top right)
    • Name: "Risk Score Distribution"

Visualization 2: Regional Distribution (Pie Chart)

Purpose: Show distribution of reports by region

  1. Create Visualization:

    • Create visualization → Pie chart

  2. Configure:

    • Data source: phish-*
    • Slice by:
      • Aggregation: Terms
      • Field: region.keyword
      • Size: 10
      • Custom label: "Region"

  3. Save:

    • Name: "Regional Distribution"

Visualization 3: High-Risk URLs Over Time (Line Chart)

Purpose: Track high-risk URLs (risk_score >= 0.7) over time

  1. Create Visualization:

    • Create visualization → Line chart

  2. Configure:

    • Data source: phish-*
    • Y-axis:
      • Aggregation: Count
      • Custom label: "High-Risk Reports"
    • X-axis:
      • Aggregation: Date Histogram
      • Field: timestamp
      • Interval: Auto (or 1 minute for real-time)
      • Custom label: "Time"
    • Filters (click Add filter):
      • Field: risk_score
      • Operator: is greater than or equal
      • Value: 0.7

  3. Save:

    • Name: "High-Risk URLs Over Time"

Visualization 4: Top Phishing URLs (Data Table)

Purpose: List URLs with highest risk scores

  1. Create Visualization:

    • Create visualization → Data Table

  2. Configure:

    • Data source: phish-*
    • Metrics:
      • Aggregation: Count
    • Buckets:
      • Split rows:
        • Aggregation: Terms
        • Field: url.keyword
        • Size: 20
        • Order: Top
        • Order by: Count (descending)
    • Filters:
      • Field: risk_score
      • Operator: is greater than or equal
      • Value: 0.8

  3. Save:

    • Name: "Top High-Risk URLs"

Visualization 5: Reports by Region Over Time (Area Chart)

Purpose: Compare report volumes across regions over time

  1. Create Visualization:

    • Create visualization → Area chart

  2. Configure:

    • Data source: phish-*
    • Y-axis:
      • Aggregation: Count
      • Custom label: "Reports"
    • X-axis:
      • Aggregation: Date Histogram
      • Field: timestamp
      • Interval: Auto
    • Split series:
      • Sub aggregation: Terms
      • Field: region.keyword
      • Size: 10

  3. Save:

    • Name: "Reports by Region Over Time"

Visualization 6: Average Risk Score by Region (Metric)

Purpose: Show average risk score per region

  1. Create Visualization:

    • Create visualization → Metric

  2. Configure:

    • Data source: phish-*
    • Metric:
      • Aggregation: Average
      • Field: risk_score
      • Custom label: "Average Risk Score"
    • Group by:
      • Aggregation: Terms
      • Field: region.keyword
      • Size: 10

  3. Save:

    • Name: "Average Risk Score by Region"

Visualization 7: Risk Score Heatmap (Heat Map)

Purpose: Visualize risk score distribution by region and time

  1. Create Visualization:

    • Create visualization → Heat Map

  2. Configure:

    • Data source: phish-*
    • Value:
      • Aggregation: Average
      • Field: risk_score
    • X-axis:
      • Aggregation: Date Histogram
      • Field: timestamp
      • Interval: Auto
    • Y-axis:
      • Aggregation: Terms
      • Field: region.keyword

  3. Save:

    • Name: "Risk Score Heatmap by Region"

Building Dashboards

Step 1: Create Dashboard

  1. Navigate to Dashboards:

    • Go to Analytics → Dashboards
    • Click Create dashboard

  2. Add Visualizations:

    • Click Add → Add an existing visualization
    • Select all visualizations created above:
      • Risk Score Distribution
      • Regional Distribution
      • High-Risk URLs Over Time
      • Top High-Risk URLs
      • Reports by Region Over Time
      • Average Risk Score by Region
      • Risk Score Heatmap by Region

  3. Arrange Layout:

    • Drag and drop visualizations to arrange
    • Resize as needed
    • Recommended layout: 
      ┌─────────────────────┬─────────────────────┐
│ Risk Score Dist.    │ Regional Dist.      │
├─────────────────────┼─────────────────────┤
│ High-Risk Over Time │ Avg Risk by Region  │
├─────────────────────┴─────────────────────┤
│ Reports by Region Over Time                │
├────────────────────────────────────────────┤
│ Risk Score Heatmap                         │
├────────────────────────────────────────────┤
│ Top High-Risk URLs                        │
└────────────────────────────────────────────┘

Step 2: Configure Auto-Refresh

  1. Enable Auto-Refresh:

    • Click the Time picker (top right)
    • Click Auto-refresh toggle
    • Select interval: 30 seconds (for real-time monitoring)
    • Or 1 minute (for less frequent updates)

  2. Set Time Range:

    • Quick: Select "Last 1 hour", "Last 24 hours", etc.
    • Relative: Custom time range
    • Absolute: Specific start/end times

Step 3: Add Filters

  1. Add Global Filters:

    • Click Add filter (top of dashboard)
    • Example filters:
      • risk_score >= 0.7 (high-risk only)
      • region: US (US region only)
      • timestamp >= now-1h (last hour)

  2. Save Filters:

    • Filters persist when dashboard is saved

Step 4: Save Dashboard

  1. Save:

    • Click Save (top right)
    • Name: "PhishNChips Live Metrics Dashboard"
    • Description: "Real-time phishing threat intelligence dashboard"

  2. Share:

    • Click Share → Permalink (for sharing URL)
    • Or Embed (for embedding in other pages)

Real-time Metrics to Track

1. Throughput Metrics

Visualization: Line chart showing reports per minute

  • Metric: Count of documents
  • Time interval: 1 minute
  • Filter: None (all reports)

Kibana Query:

{
  "size": 0,
  "aggs": {
    "reports_per_minute": {
      "date_histogram": {
        "field": "timestamp",
        "calendar_interval": "1m"
      }
    }
  }
}

2. Risk Score Trends

Visualization: Line chart with multiple series

  • Series 1: Average risk_score (all reports)
  • Series 2: Count of high-risk (risk_score >= 0.7)
  • Series 3: Count of critical-risk (risk_score >= 0.9)

3. Regional Performance

Visualization: Metric cards

  • US Region: Count, Avg Risk Score
  • EU Region: Count, Avg Risk Score
  • ASIA Region: Count, Avg Risk Score

4. Cluster Health Metrics

Note: Requires Elasticsearch monitoring indices

If available, create visualizations for:

  • Cluster Status: Green/Yellow/Red
  • Active Shards: Count over time
  • Node Count: Number of nodes
  • Indexing Rate: Documents/second

5. Query Performance

Visualization: Line chart

  • Metric: Average query latency (if logged)
  • Time interval: 1 minute
  • Note: Requires custom logging/metrics

Auto-refresh Configuration

Recommended Settings

For Real-time Monitoring:

  • Refresh interval: 30 seconds
  • Time range: Last 1 hour
  • Auto-apply: Enabled

For Historical Analysis:

  • Refresh interval: 5 minutes (or disabled)
  • Time range: Last 24 hours or custom
  • Auto-apply: Enabled

Setting Auto-refresh

  1. In Dashboard:

    • Click Time picker (top right)
    • Toggle Auto-refresh ON
    • Select interval (30s, 1m, 5m, etc.)

  2. In Individual Visualizations:

    • Some visualizations support auto-refresh independently
    • Check visualization settings

Cloud vs Local Differences

Local Docker

Access:

  • URL: http://localhost:5601
  • No authentication
  • Direct connection to local ES nodes

Configuration:

  • Kibana connects to: http://elasticsearch-us:9200
  • All services on same Docker network

Cloud Elasticsearch

Access:

  • URL: Provided by Elastic Cloud (e.g., https://xxxxx.kb.us-east-1.aws.found.io:9243)
  • Authentication required (username/password or API key)
  • HTTPS connection

Configuration:

  • Kibana pre-configured to connect to cloud ES
  • Managed by Elastic Cloud
  • May have additional features (ML, Security, etc.)

Differences:

  • Authentication: Cloud requires login
  • SSL/TLS: Cloud uses HTTPS
  • Features: Cloud may have additional Kibana features enabled
  • Performance: Cloud may have better performance (managed infrastructure)

Quick Start Checklist

  • Kibana is running and accessible
  • Index pattern phish-* created with timestamp as time field
  • At least 3-5 visualizations created
  • Dashboard created with all visualizations
  • Auto-refresh enabled (30 seconds recommended)
  • Time range set appropriately (Last 1 hour for real-time)
  • Dashboard saved and named
  • Data is flowing (check Discover tab)

Troubleshooting

No Data Showing

  1. Check Index Pattern:

    • Verify phish-* pattern matches your indices
    • Check if indices exist: curl http://localhost:9200/_cat/indices/phish-*

  2. Check Time Range:

    • Ensure time range includes data timestamps
    • Try "Last 7 days" or "All time"

  3. Check Field Mappings:

    • Verify fields exist in index pattern
    • Check field types (keyword vs text)

Visualizations Not Updating

  1. Check Auto-refresh:

    • Ensure auto-refresh is enabled
    • Try manual refresh (refresh button)

  2. Check Time Range:

    • Ensure time range includes recent data
    • Check if data is being indexed

  3. Check Filters:

    • Remove filters temporarily to see all data
    • Verify filter logic is correct

Performance Issues

  1. Reduce Time Range:

    • Use shorter time windows (Last 1 hour vs Last 7 days)
    • Reduces data volume

  2. Optimize Queries:

    • Use filters instead of queries where possible
    • Limit aggregation sizes

  3. Check Cluster Health:

    • Verify ES cluster is healthy
    • Check for slow queries in ES logs

Advanced: Custom Metrics

Creating Custom Aggregations

Example: Count of unique URLs per region

{
  "size": 0,
  "aggs": {
    "by_region": {
      "terms": {
        "field": "region.keyword"
      },
      "aggs": {
        "unique_urls": {
          "cardinality": {
            "field": "url.keyword"
          }
        }
      }
    }
  }
}

Using Scripted Fields

  1. Create Scripted Field:

    • Stack Management → Index Patterns → phish-*
    • Scripted Fields → Add scripted field
    • Example: Risk category based on risk_score

  2. Use in Visualizations:

    • Select scripted field in aggregations

Document Version: 1.0
Last Updated: 2025-01-26
Author: PhishNChips Development Team