Merge branch 'feature/add-config-version-to-metering' into 'develop'

Adding config verison in schema provider

See merge request genaiic-reusable-assets/engagement-artifacts/genaiic-idp-accelerator!623

Author: rstrahan

CHANGELOG.md

@@ -7,7 +7,7 @@ SPDX-License-Identifier: MIT-0
 
 ### Added
 
-- **Configuration Version in Metering Database** — Added `config_version` field to the metering database to enable cost tracking and analytics per configuration version. The metering Glue table now includes a `config_version` column, and all metering Parquet files store the configuration version used for each document. Enables Athena queries to compare costs across different configurations, support A/B testing analytics, and optimize per-version costs. Documents without a config version default to "default".
+- **Configuration Version Tracking Across All Analytics Tables** — Added `config_version` field to all analytics tables (metering, document_evaluations, section_evaluations, attribute_evaluations, and document_sections_*) to enable comprehensive tracking and analytics per configuration version. All Glue tables now include a `config_version` column, and all Parquet files store the configuration version used for each document. Enables direct filtering and comparison queries without complex JOINs - users can query "show me W2 documents processed with config v2.1" or "compare accuracy for configs v2.0 vs v2.1" with simple WHERE clauses. Supports cost analysis, A/B testing, quality comparison, and data lineage tracking. Documents without a config version default to "default".
 
 ### Fixed
 

docs/reporting-database.md

@@ -46,6 +46,7 @@ The `document_evaluations` table contains document-level evaluation metrics:
 | false_alarm_rate | double | False alarm rate (0-1) |
 | false_discovery_rate | double | False discovery rate (0-1) |
 | execution_time | double | Time taken to evaluate (seconds) |
+| config_version | string | Configuration version used for processing |
 
 This table is partitioned by date (YYYY-MM-DD format).
 
@@ -65,6 +66,7 @@ The `section_evaluations` table contains section-level evaluation metrics:
 | false_alarm_rate | double | Section false alarm rate (0-1) |
 | false_discovery_rate | double | Section false discovery rate (0-1) |
 | evaluation_date | timestamp | When the evaluation was performed |
+| config_version | string | Configuration version used for processing |
 
 This table is partitioned by date (YYYY-MM-DD format).
 
@@ -87,6 +89,7 @@ The `attribute_evaluations` table contains attribute-level evaluation metrics:
 | confidence | string | Confidence score from extraction |
 | confidence_threshold | string | Confidence threshold used |
 | evaluation_date | timestamp | When the evaluation was performed |
+| config_version | string | Configuration version used for processing |
 
 This table is partitioned by date (YYYY-MM-DD format).
 
@@ -144,6 +147,7 @@ The `metering` table captures detailed usage metrics and cost information for ea
 | unit_cost | double | Cost per unit in USD (e.g., cost per token, cost per page) |
 | estimated_cost | double | Calculated total cost in USD (value × unit_cost) |
 | timestamp | timestamp | When the operation was performed |
+| config_version | string | Configuration version used for processing |
 
 This table is partitioned by date (YYYY-MM-DD format).
 
@@ -214,6 +218,7 @@ Document sections are stored in dynamically created tables based on the section
 | section_classification | string | Type/class of the section |
 | section_confidence | double | Confidence score for the section classification |
 | timestamp | timestamp | When the document was processed |
+| config_version | string | Configuration version used for processing |
 
 **Dynamic Data Columns:**
 The remaining columns are dynamically inferred from the JSON extraction results and vary by section type. Common patterns include:
@@ -502,6 +507,68 @@ GROUP BY
   context
 ORDER BY 
   total_cost DESC;
+
+-- Cost analysis by configuration version
+SELECT 
+  config_version,
+  COUNT(DISTINCT document_id) as document_count,
+  SUM(estimated_cost) as total_cost,
+  AVG(estimated_cost) as avg_cost_per_record
+FROM 
+  metering
+WHERE 
+  date >= '2024-01-01'
+GROUP BY 
+  config_version
+ORDER BY 
+  total_cost DESC;
+```
+
+**Configuration version analysis:**
+```sql
+-- Compare accuracy across configuration versions
+SELECT 
+  config_version,
+  AVG(accuracy) as avg_accuracy,
+  AVG(f1_score) as avg_f1_score,
+  COUNT(DISTINCT document_id) as document_count
+FROM 
+  document_evaluations
+WHERE 
+  date >= '2024-01-01'
+GROUP BY 
+  config_version
+ORDER BY 
+  avg_f1_score DESC;
+
+-- Filter documents by configuration version
+SELECT 
+  document_id,
+  section_classification,
+  timestamp
+FROM 
+  document_sections_w2
+WHERE 
+  config_version = 'v2.1'
+  AND date >= '2024-01-01'
+LIMIT 100;
+
+-- Cost vs quality by configuration version
+SELECT 
+  m.config_version,
+  AVG(e.weighted_overall_score) as avg_quality,
+  SUM(m.estimated_cost) as total_cost,
+  COUNT(DISTINCT m.document_id) as document_count
+FROM 
+  document_evaluations e
+JOIN 
+  metering m ON e.document_id = m.document_id AND e.config_version = m.config_version
+WHERE 
+  e.date >= '2024-01-01'
+GROUP BY 
+  m.config_version
+ORDER BY 
+  avg_quality DESC;
 ```
 
 ### Creating Dashboards

lib/idp_common_pkg/idp_common/agents/analytics/assets/db_description.md

@@ -5,22 +5,22 @@ The solution creates several predefined tables in the Glue Data Catalog:
 
 1. Document Evaluations Table (document_evaluations)
    * Contains document-level evaluation metrics
-   * Columns include: document_id, input_key, evaluation_date, accuracy, precision, recall, f1_score, false_alarm_rate, false_discovery_rate, execution_time
+   * Columns include: document_id, input_key, evaluation_date, accuracy, precision, recall, f1_score, false_alarm_rate, false_discovery_rate, execution_time, config_version
    * Partitioned by date (YYYY-MM-DD format)
 
 2. Section Evaluations Table (section_evaluations)
    * Contains section-level evaluation metrics
-   * Columns include: document_id, section_id, section_type, accuracy, precision, recall, f1_score, false_alarm_rate, false_discovery_rate, evaluation_date
+   * Columns include: document_id, section_id, section_type, accuracy, precision, recall, f1_score, false_alarm_rate, false_discovery_rate, evaluation_date, config_version
    * Partitioned by date (YYYY-MM-DD format)
 
 3. Attribute Evaluations Table (attribute_evaluations)
    * Contains attribute-level evaluation metrics
-   * Columns include: document_id, section_id, section_type, attribute_name, expected, actual, matched, score, reason, evaluation_method, confidence, confidence_threshold, evaluation_date
+   * Columns include: document_id, section_id, section_type, attribute_name, expected, actual, matched, score, reason, evaluation_method, confidence, confidence_threshold, evaluation_date, config_version
    * Partitioned by date (YYYY-MM-DD format)
 
 4. Metering Table (metering)
    * Captures detailed usage metrics for document processing operations
-   * Columns include: document_id, context, service_api, unit, value, number_of_pages, timestamp
+   * Columns include: document_id, context, service_api, unit, value, number_of_pages, timestamp, config_version
    * Partitioned by date (YYYY-MM-DD format)
 
 5. Rule Validation Summary Table (rule_validation_summary)
@@ -39,7 +39,7 @@ In addition to the predefined tables, the solution also creates dynamic tables f
 
 * Tables are automatically created by an AWS Glue Crawler based on the section classification
 * Each section type gets its own table (e.g., document_sections_invoice, document_sections_receipt)
-* Common columns include: section_id, document_id, section_classification, section_confidence, timestamp
+* Common columns include: section_id, document_id, section_classification, section_confidence, timestamp, config_version
 * Additional columns are dynamically inferred from the JSON extraction results
 * Tables are partitioned by date (YYYY-MM-DD format)
 

lib/idp_common_pkg/idp_common/agents/analytics/schema_provider.py

@@ -54,6 +54,7 @@ def get_metering_table_description() -> str:
 - `unit_cost` (double): Cost per unit in USD
 - `estimated_cost` (double): Calculated total cost (value × unit_cost)
 - `timestamp` (timestamp): When the operation was performed
+- `config_version` (string): Configuration version used for processing (defaults to "default" if not specified)
 
 **Partitioned by**: date (YYYY-MM-DD format)
 
@@ -89,12 +90,21 @@ def get_metering_table_description() -> str:
 ORDER BY total_cost DESC
 
 -- Token usage by model
-SELECT "service_api", 
+SELECT "service_api",
        SUM(CASE WHEN "unit" = 'inputTokens' THEN "value" ELSE 0 END) as input_tokens,
        SUM(CASE WHEN "unit" = 'outputTokens' THEN "value" ELSE 0 END) as output_tokens
-FROM metering 
+FROM metering
 WHERE "unit" IN ('inputTokens', 'outputTokens')
 GROUP BY "service_api"
+
+-- Cost by configuration version
+SELECT "config_version",
+       SUM("estimated_cost") as total_cost,
+       COUNT(DISTINCT "document_id") as document_count,
+       SUM("estimated_cost") / COUNT(DISTINCT "document_id") as avg_cost_per_doc
+FROM metering
+GROUP BY "config_version"
+ORDER BY total_cost DESC
 ```
 """
 
@@ -187,6 +197,7 @@ def get_evaluation_tables_description() -> str:
 - Use `document_id` to join between all three tables
 - Use `section_id` and `document_id` to join section and attribute evaluations
 - Join with metering table on `document_id` for cost vs accuracy analysis
+- `config_version` is available directly in all evaluation tables (no join needed)
 
 ### Sample Queries:
 ```sql
@@ -211,13 +222,40 @@ def get_evaluation_tables_description() -> str:
 WHERE "confidence" IS NOT NULL
 GROUP BY confidence_band
 
--- Cost per accuracy point by document type  
+-- Cost per accuracy point by document type
 SELECT se."section_type",
        AVG(se."accuracy") as avg_accuracy,
        SUM(m."estimated_cost") / COUNT(DISTINCT m."document_id") as avg_cost_per_doc
 FROM section_evaluations se
-JOIN metering m ON se."document_id" = m."document_id"  
+JOIN metering m ON se."document_id" = m."document_id"
 GROUP BY se."section_type"
+
+-- Filter by config_version (available directly in evaluation tables)
+SELECT "document_id",
+       "accuracy",
+       "f1_score",
+       "config_version"
+FROM document_evaluations
+WHERE "config_version" = 'your-config-version'
+
+-- Compare accuracy across configuration versions
+SELECT "config_version",
+       AVG("accuracy") as avg_accuracy,
+       AVG("f1_score") as avg_f1_score,
+       COUNT(DISTINCT "document_id") as document_count
+FROM document_evaluations
+GROUP BY "config_version"
+ORDER BY avg_f1_score DESC
+
+-- Cost vs quality analysis by config version
+SELECT e."config_version",
+       AVG(e."weighted_overall_score") as avg_quality_score,
+       SUM(m."estimated_cost") as total_cost,
+       SUM(m."estimated_cost") / AVG(e."weighted_overall_score") as cost_per_quality_point
+FROM document_evaluations e
+JOIN metering m ON e."document_id" = m."document_id"
+GROUP BY e."config_version"
+ORDER BY avg_quality_score DESC
 ```
 """
 
@@ -371,6 +409,7 @@ def get_dynamic_document_sections_description(
                 "  - `timestamp` (timestamp): When the document was processed\n"
             )
             description += "  - `date` (string): Partition key in YYYY-MM-DD format\n"
+            description += "  - `config_version` (string): Configuration version used for processing\n"
             description += (
                 "  - Various `metadata.*` columns (strings): Processing metadata\n"
             )
@@ -450,6 +489,16 @@ def get_dynamic_document_sections_description(
 JOIN metering m ON ds."document_id" = m."document_id"
 WHERE ds."document_class.type" = 'W2'
 GROUP BY ds."section_classification", ds."document_class.type"
+
+-- CORRECT: Filter by configuration version
+SELECT "document_id",
+       "document_class.type",
+       "config_version",
+       "timestamp"
+FROM document_sections_w2
+WHERE "config_version" = 'fake_w2'
+AND date >= '2024-01-01'
+ORDER BY "timestamp" DESC
 ```
 
 **This schema information is generated from your actual configuration and shows exactly what tables and columns exist in your deployment.**
@@ -757,12 +806,13 @@ def _get_specific_document_sections_table_info(
 
 #### Standard Columns (present in all document_sections tables):
 - `"document_id"` (string): Unique identifier for the document
-- `"section_id"` (string): Unique identifier for the section  
+- `"section_id"` (string): Unique identifier for the section
 - `"section_classification"` (string): Type/class of the document section
 - `"section_confidence"` (string): Confidence score for classification
 - `"explainability_info"` (string): JSON with extraction field confidence scores and geometry
 - `"timestamp"` (timestamp): When document was processed in YYYY-MM-DD hh:mm:ss.ms format
 - `"date"` (string): Partition key in YYYY-MM-DD format
+- `"config_version"` (string): Configuration version used for processing
 
 #### Columns specific to this table:
 """

lib/idp_common_pkg/idp_common/reporting/save_reporting_data.py

@@ -599,6 +599,7 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
                 ("correctly_classified_pages", pa.int32()),
                 ("correctly_split_without_order", pa.int32()),
                 ("correctly_split_with_order", pa.int32()),
+                ("config_version", pa.string()),
             ]
         )
 
@@ -615,6 +616,7 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
                 ("false_discovery_rate", pa.float64()),
                 ("weighted_overall_score", pa.float64()),
                 ("evaluation_date", pa.timestamp("ms")),
+                ("config_version", pa.string()),
             ]
         )
 
@@ -634,6 +636,7 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
                 ("confidence_threshold", pa.string()),
                 ("weight", pa.float64()),
                 ("evaluation_date", pa.timestamp("ms")),
+                ("config_version", pa.string()),
             ]
         )
 
@@ -730,6 +733,7 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
                 if doc_split_metrics
                 else None
             ),
+            "config_version": document.config_version or "default",
         }
 
         # Save document metrics in Parquet format
@@ -768,6 +772,7 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
                     "weighted_overall_score", 0.0
                 ),
                 "evaluation_date": evaluation_date,  # Use document's initial_event_time
+                "config_version": document.config_version or "default",
             }
             section_records.append(section_record)
 
@@ -804,6 +809,7 @@ def save_evaluation_results(self, document: Document) -> Optional[Dict[str, Any]
                     ),
                     "weight": weight,  # Explicitly handle None values
                     "evaluation_date": evaluation_date,  # Use document's initial_event_time
+                    "config_version": document.config_version or "default",
                 }
                 attribute_records.append(attribute_record)
                 logger.debug(
@@ -1292,6 +1298,9 @@ def save_document_sections(self, document: Document) -> Optional[Dict[str, Any]]
                     flattened_data["section_classification"] = section.classification
                     flattened_data["section_confidence"] = section.confidence
                     flattened_data["timestamp"] = timestamp
+                    flattened_data["config_version"] = (
+                        document.config_version or "default"
+                    )
 
                     section_records.append(flattened_data)
 
@@ -1311,6 +1320,9 @@ def save_document_sections(self, document: Document) -> Optional[Dict[str, Any]]
                         )
                         flattened_item["section_confidence"] = section.confidence
                         flattened_item["record_index"] = i
+                        flattened_item["config_version"] = (
+                            document.config_version or "default"
+                        )
 
                         section_records.append(flattened_item)
                 else:
@@ -1321,6 +1333,7 @@ def save_document_sections(self, document: Document) -> Optional[Dict[str, Any]]
                         "section_classification": section.classification,
                         "section_confidence": section.confidence,
                         "value": str(extraction_data),
+                        "config_version": document.config_version or "default",
                     }
                     section_records.append(record)
 

lib/idp_common_pkg/tests/unit/reporting/test_save_reporting_data.py

@@ -446,3 +446,46 @@ def test_save_with_multiple_data_types(
         mock_save_metering.assert_called_once_with(document_with_sections)
         mock_save_sections.assert_called_once_with(document_with_sections)
         assert len(results) == 2
+
+    def test_config_version_in_schemas_and_records(self, mock_s3_client):
+        """Test that config_version is included in schemas and populated in records."""
+        import pyarrow as pa
+
+        # Test metering schema
+        metering_schema = pa.schema(
+            [
+                ("document_id", pa.string()),
+                ("context", pa.string()),
+                ("service_api", pa.string()),
+                ("unit", pa.string()),
+                ("value", pa.float64()),
+                ("number_of_pages", pa.int32()),
+                ("unit_cost", pa.float64()),
+                ("estimated_cost", pa.float64()),
+                ("timestamp", pa.timestamp("ms")),
+                ("config_version", pa.string()),
+            ]
+        )
+        assert "config_version" in [field.name for field in metering_schema]
+
+        # Test document evaluation schema (check in save_evaluation_results source)
+        # Create a test document with config_version
+        doc_with_config = Document(
+            id="test-doc", input_key="test/doc.pdf", config_version="test-v1.0"
+        )
+        doc_without_config = Document(id="test-doc2", input_key="test/doc2.pdf")
+
+        # Verify config_version fallback behavior
+        assert doc_with_config.config_version == "test-v1.0"
+        assert doc_without_config.config_version is None
+
+        # Simulate record creation with fallback
+        record_with_config = {
+            "config_version": doc_with_config.config_version or "default"
+        }
+        record_without_config = {
+            "config_version": doc_without_config.config_version or "default"
+        }
+
+        assert record_with_config["config_version"] == "test-v1.0"
+        assert record_without_config["config_version"] == "default"