Prompt caching when Bedrock used for OCR and Extraction

[kaleko]

In the situation where the IDP stack is using Bedrock for OCR and also Bedrock for extraction, no prompt caching is used. Assuming the same LLM is used for each, this is an opportunity for improvement.

Implementing this would involve restructuring the OCR and Extraction prompts to start with the same prefix (probably something like: "here is an image of a page: {IMAGE}"), then adding a cache point immediately after.

I expect this will be easy to implement but perhaps more arduous to verify that it doesn't negatively impact accuracy.

https://deepwiki.com/search/if-i-use-bedrock-for-ocr-it-wi_3259f96e-c288-4288-b00c-80967d8708c1?mode=fast

[rstrahan]

Hi @kaleko - <> tags are currently used in the default task prompt templates in both Classification and Extraction prompts.. but these are intended to maximize caching across different documents using the same config..

I think if we try to cache across classification and extraction by caching the doc content, we'd end up writing cache once, reading it once (minimal savings), at the cost of optimizing the repeated static content across all docs that we do today. See more detail below..

https://aws-solutions-library-samples.github.io/accelerated-intelligent-document-processing-on-aws/configuration/#how-it-works-1

Here's the default classification prompt

<task-description> Analyze the provided document using both its visual layout and textual content to determine its document type and whether this page begins a new document or continues the previous one. </task-description>
<document-types> {CLASS_NAMES_AND_DESCRIPTIONS} </document-types>
<classification-instructions> Follow these steps to classify the document: 1. Examine the visual layout: headers, logos, formatting, structure, and visual organization 2. Analyze the textual content: key phrases, terminology, purpose, and information type 3. Identify distinctive features that match the document type descriptions 4. Decide if this page starts a new document (output "start") or continues the previous document (output "continue") 5. Consider both visual and textual evidence together to determine the best match 6. CRITICAL: Only use document types explicitly listed in the <document-types> section </classification-instructions>
<output-format> {
  "classification_reason": "Detailed reasoning including specific visual and textual evidence that led to this classification",
  "class": "exact_document_type_from_list",
  "document_boundary": "start or continue"
} </output-format>

<<CACHEPOINT>>

<document-ocr-data> {DOCUMENT_TEXT} </document-ocr-data>
<document-image> {DOCUMENT_IMAGE} </document-image>
<final-instructions> Analyze the document above by: 1. Applying the <classification-instructions> to examine both visual and textual features 2. Selecting ONLY from document types in <document-types> 3. Providing clear reasoning with specific evidence 4. Outputting in the exact JSON format specified in <output-format> </final-instructions>

Just one <<CACHEPOINT>> inserted here.. Above this point, the prompt is static.. Below this point, the doc/page content is inserted and since it varies, not worth caching.

Our 'Granular' assessment prompt has two <<CACHEPOINT>> because we run it many times with the same doc content (in an effort to get it to do better assessments by focusing on small numbers of extractions at a time)

<snip>
IMPORTANT:   - For LIST attributes like "Transactions", assess EACH individual item in the list separately with individual bounding boxes - Each transaction should be assessed as a separate object in the array with its own spatial coordinates - Do NOT provide aggregate assessments for list items - assess each one individually with precise locations - Include assessments AND bounding boxes for ALL attributes present in the extraction results - Match the exact structure of the extracted data - Provide page numbers for all bounding boxes (starting from 1) </final-instructions>
<<CACHEPOINT>>
<document-image> {DOCUMENT_IMAGE} </document-image>
<ocr-text-confidence-results> {OCR_TEXT_CONFIDENCE} </ocr-text-confidence-results>
<<CACHEPOINT>>
<attributes-definitions> {ATTRIBUTE_NAMES_AND_DESCRIPTIONS} </attributes-definitions>
<extraction-results> {EXTRACTION_RESULTS} </extraction-results>

You can see effects in the Estimated Cost of any document.. I ran a Test Execution of 50 W2s - you can see the Extraction token usage here across the whole test set.  The static part of the prompt consumed cacheWriteTokens the first time (more expensive) but cacheReadTokens every other time (cheaper). Second screen shot is from just one of the W2s.. all cacheRead, noCacheWrite.