Classification confidence hardcoded to 1.0 and prompts lack schema field names

[joaothomaz-built]

Summary

Two issues in ClassificationService reduce classification accuracy and prevent downstream consumers from identifying uncertain matches for human review.

Both affect Pattern 2 (and the shared idp_common library used across patterns).

Issue 1: Bedrock classification confidence is hardcoded to 1.0

Bedrock models return meaningful confidence scores in their classification responses, but classify_page_bedrock discards them and hardcodes confidence=1.0:

lib/idp_common_pkg/idp_common/classification/service.py ~line 1337:

return PageClassification(
    page_id=page_id,
    classification=DocumentClassification(
        doc_type=doc_type,
        confidence=1.0,  # Default confidence
        metadata={...},
    ),
)

This pattern propagates through multiple paths in the same file:

Path	Approx. line	Confidence value
Bedrock classification	~1337	1.0
SageMaker/UDOP	~1433	1.0
Holistic/packet classification (segment pages)	~2601	1.0
Regex shortcut	~1170	1.0
Max-pages fallback	~326	1.0
Single-class config	~2500	1.0
Section splitting disabled	~2080	1.0

The Section model also defaults to 1.0:

# lib/idp_common_pkg/idp_common/models.py ~line 57
classification: str
confidence: float = 1.0

Impact: Consumers of the classification output (UIs, evaluation pipelines, downstream automation) cannot distinguish high-confidence from low-confidence classifications. This prevents flagging uncertain sections for human review, which is critical for document processing accuracy.

Suggestion: Parse and propagate the confidence value from Bedrock's response. For paths where the model genuinely doesn't return confidence (SageMaker, regex shortcuts), the 1.0 default is reasonable, but it should be distinguished from real model confidence (e.g., via a confidence_source metadata field).

Issue 2: Classification prompt lacks schema field names

The class list injected into Bedrock prompts only includes type name + description:

lib/idp_common_pkg/idp_common/classification/service.py ~line 741:

def _format_classes_list(self) -> str:
    return "\n".join(
        [
            f"{doc_type.type_name}  \t[ {doc_type.description} ]"
            for doc_type in self.document_types
        ]
    )

The holistic classification path (_format_classes_and_descriptions, ~line 2332) similarly uses only type + description in a markdown table.

Schema field names (e.g., property_address, appraised_value, inspection_date) from the JSON Schema properties are available in config.classes but are never included in the prompt. These field names are the strongest semantic signal for disambiguation — when Bedrock sees page text containing "Appraised Value: $450,000" and the class list includes fields like appraised_value, effective_date, borrower_name, the match becomes significantly more accurate.

Impact: When multiple document types have similar names or descriptions (e.g., "Appraisal Reports" vs "Inspection Reports"), the model lacks sufficient signal to disambiguate, resulting in misclassification.

Suggestion: Append a subset of schema field names (e.g., top 10-15) to each class entry in the prompt. This is a low-risk change since it only adds content to the prompt without changing its structure, and stays well within model context limits.

Environment

• Version: v0.4.16
• File: lib/idp_common_pkg/idp_common/classification/service.py

[rstrahan]

Thanks for the detailed write-up, @joaothomaz-built!

Re Issue 1 (Confidence scores): The premise that "Bedrock models return meaningful confidence scores in their classification responses" is not accurate. The Bedrock Converse API returns generated text, stop reasons, and usage metrics — but no confidence or probability scores. Additionally, our classification prompt asks the model to return class, classification_reason, and document_boundary — there's no confidence field in the response.

We could hypothetically add a confidence field to the prompt and ask the LLM to self-assess, but LLM self-reported confidence is notoriously poorly calibrated — models often report high confidence even when wrong. We'd rather not surface an unreliable signal that gives users a false sense of reliability.

For cases where you need confidence-based routing to human review, we'd suggest using the assessment service which provides extraction-quality confidence scores with configurable thresholds, or implementing a post-processing Lambda hook that applies your own business rules.

The 1.0 hardcoding is intentional — it signals "no confidence information available" rather than pretending to have a calibrated score.

Re Issue 2 (Attribute names in classification prompts): This is an interesting idea. Including schema field names could theoretically help disambiguate visually similar document types. However, including all attributes for all classes in every classification prompt would significantly increase token consumption (and cost/latency), and including only a subset would bias the classifier.

Current mitigations include: (a) writing detailed class descriptions that capture distinguishing characteristics, and (b) using few-shot classification examples which are very effective at disambiguating similar types.

That said, we can see the value as an opt-in feature for power users. We'll consider adding an optional {CLASS_AND_ATTRIBUTE_NAMES_AND_DESCRIPTIONS} placeholder that users could include in custom classification prompts when they need richer class context. This would keep it opt-in so it doesn't impact token costs for users who don't need it. Would that work for you?

Thanks for raising these great topics!

[joaothomaz-built]

Hi Bob, thanks for the thoughtful clarification. Understood issue 1, and an extendable placeholder on issue 2 would be a great opt-in addition for sure!

…

On Thu, Apr 16, 2026 at 4:03 PM Bob Strahan ***@***.***> wrote: *rstrahan* left a comment (aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws#262) <#262 (comment)> Thanks for the detailed write-up, @joaothomaz-built <https://github.com/joaothomaz-built>! *Re Issue 1 (Confidence scores):* The premise that "Bedrock models return meaningful confidence scores in their classification responses" is not accurate. The Bedrock Converse API returns generated text, stop reasons, and usage metrics — but no confidence or probability scores. Additionally, our classification prompt asks the model to return class, classification_reason, and document_boundary — there's no confidence field in the response. We could hypothetically add a confidence field to the prompt and ask the LLM to self-assess, but LLM self-reported confidence is notoriously poorly calibrated <https://arxiv.org/abs/2306.13063> — models often report high confidence even when wrong. We'd rather not surface an unreliable signal that gives users a false sense of reliability. For cases where you need confidence-based routing to human review, we'd suggest using the *assessment service* which provides extraction-quality confidence scores with configurable thresholds, or implementing a post-processing Lambda hook that applies your own business rules. The 1.0 hardcoding is intentional — it signals "no confidence information available" rather than pretending to have a calibrated score. *Re Issue 2 (Attribute names in classification prompts):* This is an interesting idea. Including schema field names could theoretically help disambiguate visually similar document types. However, including all attributes for all classes in every classification prompt would significantly increase token consumption (and cost/latency), and including only a subset would bias the classifier. Current mitigations include: (a) writing detailed class descriptions that capture distinguishing characteristics, and (b) using *few-shot classification examples* which are very effective at disambiguating similar types. That said, we can see the value as an opt-in feature for power users. We'll consider adding an optional {CLASS_AND_ATTRIBUTE_NAMES_AND_DESCRIPTIONS} placeholder that users could include in custom classification prompts when they need richer class context. This would keep it opt-in so it doesn't impact token costs for users who don't need it. Would that work for you? Thanks for raising these great topics! — Reply to this email directly, view it on GitHub <#262 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B2V6SQXMCF2WLSXHUCOG6IT4WEU73AVCNFSM6AAAAACXGOQU7SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DENRSGY4TCOBXGQ> . You are receiving this because you were mentioned.Message ID: <aws-solutions-library-samples/accelerated-intelligent-document-processing-on-aws/issues/262/4262691874 @github.com>