overhaul

Author: Demfier

DATASET.md

@@ -0,0 +1,101 @@
+# InsightBench Dataset Schema
+
+This document describes the structure and conventions of the InsightBench dataset.
+
+## Overview
+
+InsightBench contains **97 flags** (analytical tasks) with a total of **409 ground-truth insights** for evaluating data analytics agents. Each flag consists of:
+- A CSV dataset (simulated ServiceNow data)
+- A Jupyter notebook with ground-truth analysis code and insights
+- A JSON metadata file used by the evaluation framework
+
+## JSON Schema
+
+Each `flag-{N}.json` file has this structure:
+
+```json
+{
+    "dataset_csv_path": "data/notebooks/csvs/flag-{N}.csv",
+    "user_dataset_csv_path": "data/notebooks/csvs/flag-{N}-sysuser.csv | null",
+    "metadata": {
+        "goal": "Analytical goal description",
+        "role": "Analyst role (e.g., 'L2 Support Agent')",
+        "category": "One of the canonical categories",
+        "dataset_description": "Description of the dataset fields",
+        "header": "Task title",
+        "difficulty": 1-4 (optional, null if not assigned)
+    },
+    "insight_list": [
+        {
+            "data_type": "descriptive | diagnostic | predictive | ...",
+            "insight": "The key finding (ground truth)",
+            "insight_value": { ... },
+            "plot": { ... },
+            "question": "The analytical question",
+            "actionable_insight": "Recommended action",
+            "code": "Python analysis code"
+        }
+    ],
+    "insights": ["insight text 1", "insight text 2", ...],
+    "summary": "Summary of all findings"
+}
+```
+
+## Categories
+
+| Category | Count | Description |
+|----------|-------|-------------|
+| Incident Management | 31 | ServiceNow incident analysis |
+| Finance Management | 28 | Expense and financial analysis |
+| Goal Management | 20 | Organizational goal tracking |
+| User Management | 8 | HR and personnel analysis |
+| Asset Management | 6 | IT asset management |
+| Asset Management & User Management | 2 | Cross-domain analysis |
+| Finance Management & User Management | 2 | Cross-domain analysis |
+
+## Data Types (Insight Classification)
+
+| Type | Count | Description |
+|------|-------|-------------|
+| descriptive | 188 | What happened? Distribution, averages, totals |
+| diagnostic | 108 | Why did it happen? Correlations, root cause |
+| comparative | 33 | How do groups differ? Cross-group analysis |
+| predictive | 22 | What will happen? Trends, forecasts |
+| analytical | 23 | Deep analysis of patterns and relationships |
+| time_series | 17 | Temporal patterns and trends |
+| correlative | 6 | Statistical correlations |
+| trend diagnosis | 4 | Trend identification and analysis |
+| frequency | 4 | Frequency and count analysis |
+| distribution | 2 | Distribution shape analysis |
+| prescriptive | 1 | What should be done? Recommendations |
+| categorical | 1 | Categorical data analysis |
+
+## Adversarial Flags (80-87)
+
+Flags 80-87 are **deliberately adversarial** test cases. Their CSVs intentionally lack the columns required by the analysis. The ground-truth insights are error messages like "There was no column X to conduct any analysis."
+
+These flags test whether data analytics agents:
+1. Correctly identify when required data is unavailable
+2. Report the limitation instead of hallucinating insights
+3. Do not fabricate analysis results
+
+## Difficulty Levels
+
+Difficulty is assigned to 12 flags (from notebook metadata):
+
+| Level | Flags |
+|-------|-------|
+| 1 | 12, 59 |
+| 2 | 3, 49 |
+| 3 | 2, 48 |
+| 4 | 1, 47, 94, 95, 96, 97 |
+
+## Evaluation
+
+The evaluation framework uses the `insights` array (list of insight text strings) for scoring via ROUGE-1 or G-EVAL. The `insight_list` provides the full structured ground truth.
+
+Benchmark tiers:
+- `"toy"`: flags 1-5 (5 flags)
+- `"standard"`: flags 1-30 (30 flags)
+- `"full"`: flags 1-97 excluding duplicates (97 flags)
+

README.md

@@ -10,7 +10,9 @@ Insight-Bench is a benchmark dataset designed to evaluate end-to-end data analyt
 
 ## Data
 
-All groundtruth notebooks are in `data/notebooks`. 
+The benchmark contains **97 flags** (analytical tasks) with **409 ground-truth insights** across 7 categories: Incident Management, Finance Management, Goal Management, User Management, Asset Management, and two cross-domain categories. Flags 80-87 are adversarial test cases that evaluate whether agents correctly identify when required data columns are unavailable.
+
+All ground-truth notebooks are in `data/notebooks`. See [DATASET.md](DATASET.md) for full schema documentation.
 
 An example notebook can be found here: `data/notebooks/flag-1.ipynb`
 

data/notebooks/flag-1.json

@@ -4,17 +4,18 @@
     "metadata": {
         "goal": "Find the discrepancy and imbalance in distribution of incidents assigned across categories",
         "role": "L2 Support Agent",
-        "category": "Incidents Management",
+        "category": "Incident Management",
         "dataset_description": "The dataset comprises 500 entries simulating ServiceNow incidents table, detailing various attributes such as category, state, open and close dates, involved personnel, and incident specifics like location, description, and priority. It captures incident management activities with fields like 'opened_at', 'closed_at', 'assigned_to', 'short_description', and 'priority', reflecting the operational handling and urgency of issues across different locations and categories.",
-        "header": "Hardware Incident Dataset (Flag 1)"
+        "header": "Hardware Incident Dataset (Flag 1)",
+        "difficulty": 4
     },
     "insight_list": [
         {
             "data_type": "descriptive",
             "insight": "hardware incidents is significantly higher than others",
             "insight_value": {
                 "x_val": "Hardware",
-                "y_val": 335
+                "y_val": 336
             },
             "plot": {
                 "plot_type": "histogram",
@@ -168,7 +169,8 @@
                 "plot description": "The line plot displays the trend of incidents across different categories over time. Each line represents a category and the height of the line corresponds to the number of incidents in that category at a given time. The 'Hardware' category does not show any significant increasing trend. It is relatively stable and has a higher count than other categories."
             },
             "question": "Is there a pattern or trend over time in the distribution of incidents across categories?",
-            "code": "# Convert 'opened_at' to datetime\ndf['opened_at'] = pd.to_datetime(df['opened_at'])\n\n# Resample the data by month and category, and count the number of incidents\ndf_resampled = df.groupby([pd.Grouper(key='opened_at', freq='M'), 'category']).size().unstack()\n\n# Plot the resampled data\nplot = df_resampled.plot(kind='line')\n\n# Set plot title\nplt.title('Incidents Over Time by Category')\n\n# Set x-axis label\nplt.xlabel('Time')\n\n# Set y-axis label\nplt.ylabel('Number of Incidents')\n\n# Display the figure\nplt.show()"
+            "code": "# Convert 'opened_at' to datetime\ndf['opened_at'] = pd.to_datetime(df['opened_at'])\n\n# Resample the data by month and category, and count the number of incidents\ndf_resampled = df.groupby([pd.Grouper(key='opened_at', freq='M'), 'category']).size().unstack()\n\n# Plot the resampled data\nplot = df_resampled.plot(kind='line')\n\n# Set plot title\nplt.title('Incidents Over Time by Category')\n\n# Set x-axis label\nplt.xlabel('Time')\n\n# Set y-axis label\nplt.ylabel('Number of Incidents')\n\n# Display the figure\nplt.show()",
+            "actionable_insight": ""
         },
         {
             "data_type": "diagnostic",

data/notebooks/flag-10.json

@@ -77,7 +77,8 @@
             },
             "question": "Is the increase in ttr uniform across all categories of incidents or is it more pronounced in a specific category?",
             "actionable_insight": "The uniform increase in TTR across all categories suggests that the issue is not specific to any particular category. This could indicate a systemic issue in the incident management process. It would be beneficial to investigate the overall process and identify areas for improvement to reduce the TTR.",
-            "code": "# Group by category and opened_at date, then calculate average ttr\ncategory_ttr_trend = df.groupby(['category', df['opened_at'].dt.date])['ttr_days'].mean().reset_index()\n\n# Plot the trend for each category\nfig, ax = plt.subplots(figsize=(10,6))\n\nfor category in category_ttr_trend['category'].unique():\n    ax.plot(category_ttr_trend[category_ttr_trend['category'] == category]['opened_at'], \n            category_ttr_trend[category_ttr_trend['category'] == category]['ttr_days'], \n            label=category)\n\nplt.title('Trend of TTR Across Categories Over Time')\nplt.xlabel('Opened At')\nplt.ylabel('Average TTR (Days)')\nplt.legend(loc='best')\nplt.grid(True)\nplt.show()"
+            "code": "# Group by category and opened_at date, then calculate average ttr\ncategory_ttr_trend = df.groupby(['category', df['opened_at'].dt.date])['ttr_days'].mean().reset_index()\n\n# Plot the trend for each category\nfig, ax = plt.subplots(figsize=(10,6))\n\nfor category in category_ttr_trend['category'].unique():\n    ax.plot(category_ttr_trend[category_ttr_trend['category'] == category]['opened_at'], \n            category_ttr_trend[category_ttr_trend['category'] == category]['ttr_days'], \n            label=category)\n\nplt.title('Trend of TTR Across Categories Over Time')\nplt.xlabel('Opened At')\nplt.ylabel('Average TTR (Days)')\nplt.legend(loc='best')\nplt.grid(True)\nplt.show()",
+            "insight_value": {}
         },
         {
             "data_type": "descriptive",
@@ -97,7 +98,8 @@
             },
             "question": "Are there any trends in the productivity of the human agents over time? For instance, is there a decrease in the number of incidents resolved per agent over time?",
             "actionable_insight": "The uniform productivity across all agents suggests that the workload is evenly distributed and all agents are equally productive. This is a positive indicator of good workload management. However, it would still be beneficial to continually monitor agent productivity and workload to ensure this balance is maintained.",
-            "code": "agent_incident_count = df.groupby('assigned_to')['number'].count()\n\n# Plot the histogram\nagent_incident_count.plot(kind='bar', figsize=(10,6))\n\nplt.title('Number of Incidents Resolved Per Agent')\nplt.xlabel('Agent')\nplt.ylabel('Number of Incidents Resolved')\nplt.grid(True)\nplt.xticks(rotation=45)\nplt.show()"
+            "code": "agent_incident_count = df.groupby('assigned_to')['number'].count()\n\n# Plot the histogram\nagent_incident_count.plot(kind='bar', figsize=(10,6))\n\nplt.title('Number of Incidents Resolved Per Agent')\nplt.xlabel('Agent')\nplt.ylabel('Number of Incidents Resolved')\nplt.grid(True)\nplt.xticks(rotation=45)\nplt.show()",
+            "insight_value": {}
         }
     ],
     "insights": [

data/notebooks/flag-100.json

@@ -4,9 +4,10 @@
     "metadata": {
         "goal": "Find the discrepancy and imbalance in distribution of incidents assigned across categories",
         "role": "L2 Support Agent",
-        "category": "Incidents Management",
+        "category": "Incident Management",
         "dataset_description": "The dataset comprises 500 entries simulating ServiceNow incidents table, detailing various attributes such as category, state, open and close dates, involved personnel, and incident specifics like location, description, and priority. It captures incident management activities with fields like 'opened_at', 'closed_at', 'assigned_to', 'short_description', and 'priority', reflecting the operational handling and urgency of issues across different locations and categories.",
-        "header": "Hardware Incident Dataset (Flag 100)"
+        "header": "Hardware Incident Dataset (Flag 100)",
+        "difficulty": 4
     },
     "insight_list": [
         {
@@ -73,7 +74,7 @@
         },
         {
             "data_type": "diagnostic",
-            "insight": "There are specific mentions of printers in the incident descriptions.",
+            "insight": "There are no specific mentions of the word 'Printer' in the incident descriptions.",
             "insight_value": {
                 "x_val": "Printer",
                 "y_val": 0
@@ -162,18 +163,22 @@
                 "plot description": "The line plot displays the trend of incidents across different categories over time. Each line represents a category and the height of the line corresponds to the number of incidents in that category at a given time. The 'Hardware' category does not show any significant increasing trend. It is relatively stable and has a higher count than other categories."
             },
             "question": "Is there a pattern or trend over time in the distribution of incidents across categories?",
-            "code": "# Convert 'opened_at' to datetime\ndf['opened_at'] = pd.to_datetime(df['opened_at'])\n\n# Resample the data by month and category, and count the number of incidents\ndf_resampled = df.groupby([pd.Grouper(key='opened_at', freq='M'), 'category']).size().unstack()\n\n# Plot the resampled data\nplot = df_resampled.plot(kind='line')\n\n# Set plot title\nplt.title('Incidents Over Time by Category')\n\n# Set x-axis label\nplt.xlabel('Time')\n\n# Set y-axis label\nplt.ylabel('Number of Incidents')\n\n# Display the figure\nplt.show()"
+            "code": "# Convert 'opened_at' to datetime\ndf['opened_at'] = pd.to_datetime(df['opened_at'])\n\n# Resample the data by month and category, and count the number of incidents\ndf_resampled = df.groupby([pd.Grouper(key='opened_at', freq='M'), 'category']).size().unstack()\n\n# Plot the resampled data\nplot = df_resampled.plot(kind='line')\n\n# Set plot title\nplt.title('Incidents Over Time by Category')\n\n# Set x-axis label\nplt.xlabel('Time')\n\n# Set y-axis label\nplt.ylabel('Number of Incidents')\n\n# Display the figure\nplt.show()",
+            "actionable_insight": ""
         },
         {
             "insight": "",
             "question": "What are the most effective resolution methods for different types of incidents?",
-            "code": "import pandas as pd\nfrom matplotlib import pyplot as plt\n\ndf = pd.read_csv('csvs/flag-100.csv')\n\n# Analyze resolution methods\nresolution_counts = df['resolution_method'].value_counts()\nmost_common_method = resolution_counts.index[0]\nleast_common_method = resolution_counts.index[-1]\n\nresolution_by_category = df.groupby('category')['resolution_method'].agg(lambda x: x.value_counts().index[0])\n\n# Plotting code\nplt.figure(figsize=(12, 6))\nresolution_counts.plot(kind='bar')\nplt.title('Frequency of Resolution Methods')\nplt.xlabel('Resolution Method')\nplt.ylabel('Number of Incidents')\nplt.xticks(rotation=45, ha='right')\nplt.tight_layout()\n\n# Save the plot\nplt.show()\nplt.close()"
+            "code": "import pandas as pd\nfrom matplotlib import pyplot as plt\n\ndf = pd.read_csv('csvs/flag-100.csv')\n\n# Analyze resolution methods\nresolution_counts = df['resolution_method'].value_counts()\nmost_common_method = resolution_counts.index[0]\nleast_common_method = resolution_counts.index[-1]\n\nresolution_by_category = df.groupby('category')['resolution_method'].agg(lambda x: x.value_counts().index[0])\n\n# Plotting code\nplt.figure(figsize=(12, 6))\nresolution_counts.plot(kind='bar')\nplt.title('Frequency of Resolution Methods')\nplt.xlabel('Resolution Method')\nplt.ylabel('Number of Incidents')\nplt.xticks(rotation=45, ha='right')\nplt.tight_layout()\n\n# Save the plot\nplt.show()\nplt.close()",
+            "insight_value": {},
+            "actionable_insight": "",
+            "data_type": "descriptive"
         }
     ],
     "insights": [
         "The incidents are equally distributed across all categories.",
         "There are no specific issues mentioned in the incident descriptions for each category.",
-        "There are specific mentions of printers in the incident descriptions.",
+        "There are no specific mentions of the word 'Printer' in the incident descriptions.",
         "There is no specific location where hardware incidents are concentrated.",
         "There is no significant increasing trend in the 'Hardware' or any other category.",
         ""

data/notebooks/flag-12.json

@@ -4,9 +4,10 @@
     "metadata": {
         "goal": "Find the discrepancy and imbalance in incidents assigned",
         "role": "L1 Agent",
-        "category": "Incidents Management",
+        "category": "Incident Management",
         "dataset_description": "The dataset comprises 500 entries simulating ServiceNow incidents table, detailing various attributes such as category, state, open and close dates, involved personnel, and incident specifics like  description, and priority. It captures incident management activities with fields like 'opened_at', 'closed_at', 'assigned_to', 'short_description', and 'priority', reflecting the operational handling and urgency of issues across different locations and categories.",
-        "header": "Hardware Incident Easy Dataset (Flag 12)"
+        "header": "Hardware Incident Easy Dataset (Flag 12)",
+        "difficulty": 1
     },
     "insight_list": [
         {
@@ -116,7 +117,8 @@
                 "plot description": "The line plot displays the trend of incidents across different categories over time. Each line represents a category and the height of the line corresponds to the number of incidents in that category at a given time. The 'Hardware' category shows a significant increasing trend."
             },
             "question": "Is there a pattern or trend over time in the distribution of incidents across categories?",
-            "code": "# Convert 'opened_at' to datetime\ndf['opened_at'] = pd.to_datetime(df['opened_at'])\n\n# Resample the data by month and category, and count the number of incidents\ndf_resampled = df.groupby([pd.Grouper(key='opened_at', freq='M'), 'category']).size().unstack()\n\n# Plot the resampled data\nplot = df_resampled.plot(kind='line')\n\n# Set plot title\nplt.title('Incidents Over Time by Category')\n\n# Set x-axis label\nplt.xlabel('Time')\n\n# Set y-axis label\nplt.ylabel('Number of Incidents')\n\n# Display the figure\nplt.show()"
+            "code": "# Convert 'opened_at' to datetime\ndf['opened_at'] = pd.to_datetime(df['opened_at'])\n\n# Resample the data by month and category, and count the number of incidents\ndf_resampled = df.groupby([pd.Grouper(key='opened_at', freq='M'), 'category']).size().unstack()\n\n# Plot the resampled data\nplot = df_resampled.plot(kind='line')\n\n# Set plot title\nplt.title('Incidents Over Time by Category')\n\n# Set x-axis label\nplt.xlabel('Time')\n\n# Set y-axis label\nplt.ylabel('Number of Incidents')\n\n# Display the figure\nplt.show()",
+            "actionable_insight": ""
         }
     ],
     "insights": [