Rule Validation

Rule Validation

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

Overview

Rule Validation automatically checks if your documents meet specific business rules and compliance requirements. It uses AI to evaluate documents against predefined criteria, making it useful for any industry that needs to validate documents against policies or regulations.

Common Uses:

• Healthcare: Checking prior authorizations, validating medical coding rules
• Financial Services: Verifying loan applications, checking compliance
• Legal: Reviewing contract clauses, ensuring regulatory compliance
• Insurance: Validating claims, checking policy compliance
• Manufacturing: Quality control checks, specification compliance

The healthcare examples we provide show what’s possible, but you can customize this for any industry.

See it as a packaged vertical: the Health Insurance Review sample feature wraps this rule-validation capability in an installable Feature Platform extension — it ships a prior-auth config preset, a postRuleValidation hook that turns rule results into a deterministic claim status, and a UI that visualizes both the per-rule results and a live Rules Discovery flow.

Getting Started

How to Enable Rule Validation

Rule Validation is available in Pipeline mode (the default processing mode). You can enable it in two ways:

Option 1: During Stack Deployment

• When deploying the CloudFormation stack, select the rule-validation configuration preset
• In the configuration dropdown, select rule-validation
• The stack deploys with rule validation enabled

Option 2: Import After Deployment

• Open the IDP-ACC Web UI
• Navigate to Configuration → Import
• Select rule-validation from the Config Library
• Toggle on Rule Validation

Two-Step Process for Any Industry

Rule validation works for any industry using this two-step process:

Step 1: Extract Rules from Your Policy Documents (Optional)

What you need:

• A policy document containing your rules (PDF format)
  • Examples: compliance manuals, regulatory guidelines, coding policies, underwriting rules

How to do it: Use Policy Discovery — see docs/policy-discovery.md for the full walkthrough. In short:

1. In the Web UI, open the Discovery tab and switch to Policy Discovery mode.
2. Upload your policy PDF. Policy Discovery parses it into structured rules and writes them to the active configuration version’s policy_classes field.
3. Review the extracted rules in Configuration → Policy Schema, then save the version.

Skip this step if: You already have your rules in a structured format — paste them directly into the Policy Schema editor.

Step 2: Validate Your Documents Against Rules

What you need:

• Rules to validate against (from Step 1 or your own structured rules)
• Documents to validate (PDF format)
  • Examples: applications, claims, authorization requests, contracts

How to do it:

1. Enable Rule Validation

   • Option A: Deploy stack with the rule-validation configuration preset
   • Option B: Import rule-validation from Config Library in the UI

2. Configure Document Schema

   • Go to Configuration → Document Schema tab
   • Define your document sections (e.g., Applicant Info, Financial Data, Supporting Docs)
   • Specify what attributes to extract from each section
   • These extracted attributes provide context to the AI for better validation

3. Configure Rule Schema

   • Go to Configuration → Rule Schema tab
   • Paste the rules from Step 1 (or your own rules)
   • Organize into rule types
   • Add detailed descriptions for each rule

4. Upload and Process Documents

   • Upload documents to validate
   • System automatically processes and validates against your rules
   • View results showing Pass/Fail for each rule with detailed reasoning

Coming Soon: We’re working on combining both steps into a single unified application.

Quick Start with Healthcare Example

Want to see it in action first? We provide a complete healthcare example with sample documents and pre-configured rules. See the Healthcare Example section below for step-by-step instructions using:

• Sample prior authorization document: samples/rule-validation/respiratory_pa_packet.pdf
  • Synthetic multi-page respiratory therapy prior authorization request
  • Contains multiple sections: patient information, clinical information, evidence documents, operative logs, and claims data
• Sample NCCI policy manual: samples/rule-validation/NCCI Medicare Policy Manual.pdf
  • Source: CMS NCCI Policy Manual Chapter 5 (2024)
  • Contains medical coding rules and guidelines
• Pre-configured Policy Discovery and rule validation configs

Key Features

• Multi-Level Validation Workflow: Policy classification, section-level evaluation, rule type consolidation, and orchestrated summary generation
• Dual Validation Engines: Choose LLM (semantic) or Z3 (symbolic/deterministic) per rule via x-aws-idp-validation-engine
• Policy Classification: Automatically determines which policy types apply to a document before validation
• Asynchronous Processing: Concurrent evaluation of multiple rules with built-in rate limiting
• Intelligent Chunking: Page-aware text chunking that preserves page boundaries and context
  • chunks_created: 0 = No chunking (section processed as whole)
  • chunks_created: 2+ = Section split into multiple chunks with 10% overlap
• Customizable Recommendations: User-defined recommendation options (e.g., Pass/Fail, Compliant/Non-Compliant)
• Dynamic Statistics: Automatic generation of recommendation counts based on actual results
• Comprehensive Tracking: Token usage, timing metrics, supporting page references, and chunking metadata
• Dual Output Formats: JSON for programmatic access and Markdown for human review
• Robust Error Handling: Graceful degradation with fallback responses

Architecture

Rule Validation Workflow

1. Section-Level Evaluation: Each document section is evaluated against all configured rule types

   • Rules are processed concurrently with semaphore-based rate limiting
   • Page-aware chunking handles large sections while preserving context
   • Results stored in S3 for each section

2. Rule Type Consolidation: Multiple section responses are consolidated per rule

   • LLM analyzes all evidence across sections
   • Generates single consolidated recommendation per rule
   • Aggregates supporting page references

3. Orchestrated Summary Generation: Final summary with statistics and reports

   • Dynamic recommendation counts (e.g., {“Pass”: 10, “Fail”: 2})
   • Overall statistics by rule type
   • JSON and Markdown output formats

State Machine Integration

Rule validation is integrated into the pipeline mode workflow after extraction:

OCR → Classification → Extraction → Policy Classification → Rule Validation (Map) → Orchestration

The workflow includes:

1. Policy Classification Step: Determines which policy types apply to the document
2. Rule Validation Map State: Processes sections in parallel for matched policy types only
3. Orchestration Step: Consolidates results and makes final compliance decisions

Human-in-the-Loop (HITL)

HITL review for rule validation results is not supported in the current version and will be added in a future release. When HITL is enabled for extraction, rule validation automatically skips on the second execution and reuses results from the first run.

No-Policy-Match Behavior

When a document doesn’t match any configured policy class (e.g., the document name and page content don’t match any policy_classes patterns), rule validation is skipped gracefully:

• A formatted Markdown summary is saved to S3 and displayed in the Web UI
• The message includes the document name and guidance on configuring policy classes
• The Step Functions workflow bypasses the rule validation Map state — no errors are raised

Configuration

Three-Step Rule Validation Approach

Rule validation uses a three-step approach to improve accuracy and handle large documents effectively:

1. Policy Classification: Determines which policy types apply to the document
2. Fact Extraction: Extracts relevant facts from document sections for matched policies
3. Orchestrator: Consolidates facts and makes final compliance decisions

This separation provides several benefits:

• Only relevant policy types are processed, reducing cost and time
• Large documents can be processed in chunks without losing context
• Fact-finding is separated from decision-making for clearer reasoning
• Multiple pieces of evidence are synthesized into accurate compliance determinations

Basic Configuration

Configure rule validation in your pattern configuration file:

rule_validation:
  enabled: true
  semaphore: 5  # Max concurrent API calls
  max_chunk_size: 8000  # Characters per chunk
  overlap_percentage: 10  # Chunk overlap for context

  recommendation_options: |
    Pass: The requirement criteria are fully met.
    Fail: The requirement is partially met or requires additional information.
    Information Not Found: No relevant data exists in the user history.

  # Step 1: Fact Extraction Configuration
  fact_extraction:
    model: us.anthropic.claude-sonnet-4-6
    temperature: 0.0
    top_k: 20
    top_p: 0.01
    max_tokens: 4096
    system_prompt: |
      You are a specialized fact extraction assistant...
    task_prompt: |
      Extract relevant facts from the document text for the given rule.
      Document Text: {DOCUMENT_TEXT}
      Policy Type: {policy_type}
      Rule: {rule}

  # Step 2: Orchestrator Configuration
  rule_validation_orchestrator:
    model: us.anthropic.claude-sonnet-4-6
    temperature: 0.0
    top_k: 20
    top_p: 0.01
    max_tokens: 4096
    system_prompt: |
      You are a compliance decision orchestrator...
    task_prompt: |
      Based on the extracted evidence, determine compliance.
      Extracted Evidence: {extracted_evidence}
      Policy Class: {policy_class}
      Rule: {rule}

Configuration Parameters

Common Parameters (top level):

• enabled: Turns rule validation on or off

• semaphore: Maximum number of concurrent Bedrock calls per document, in both the section-level fact-extraction step and the consolidation step (default: 5). It is a per-invocation bound, and the account-level call rate is this value times the number of documents in flight — so with MaxConcurrentWorkflows at its default of 100, a semaphore of 5 permits up to 500 concurrent calls from the consolidation step alone. That multiplier is the reason to be careful raising it.

  ⚠️ The consolidation step now honours it and previously did not, so a deployment that left the default at 5 will see that step issue fewer concurrent calls and take longer. What bounded it before was the event loop’s default thread pool — os.cpu_count() + 4 threads, capped at 32 — rather than this setting, so the width it actually ran at depended on the container rather than on the configuration. For the deployed 4,096 MB RuleValidationOrchestrationFunction that is roughly 6 to 7, so a 14-rule document previously ran in about two waves and now runs in three (ceil(14 / 5)). Raising the value is how to take that throughput back deliberately, bearing the multiplier above in mind. The end-to-end figures available are an upper bound on the change rather than the deployed one: measured against Claude Sonnet 4.5 on a 16-CPU machine, where the default pool is 20 threads wide and all 14 rules therefore ran at once, consolidation took 16.9 s at semaphore: 5 against 5.7 s unbounded.

• max_chunk_size: Maximum tokens per chunk (default: 8000). Multiplied by token_size, the assumed characters per token (default: 4), to get the character budget a chunk is measured against — 32,000 characters with the defaults

• overlap_percentage: How much of the previous chunk is repeated at the start of the next, to keep a fact that spans the boundary readable (0-100, default: 10). ⚠️ It does not govern every chunk boundary. Chunking is page-aware, and when the previous chunk held more than one complete page the whole of its last page is repeated regardless of this setting — which is the usual case for a multi-page document. The percentage applies where the previous chunk held a single page, i.e. on documents whose pages are large relative to max_chunk_size, and to the character-based fallback. In both of those, 0 repeats nothing. Values above 50 are reduced to 50 by the character fallback, which logs that it did, because a smaller stride multiplies the number of model calls rather than improving context

• recommendation_options: Custom recommendation categories for your use case

Fact Extraction Parameters:

• model: The LLM model to use for extracting facts
• temperature: Controls randomness in responses (0.0 = fully deterministic)
• system_prompt: Defines the role and behavior of the fact extraction assistant
• task_prompt: Instructions for extracting facts, with these placeholders:
  • {DOCUMENT_TEXT}: The actual document content
  • {policy_type}: The category of policy class being evaluated
  • {rule}: The specific rule text

Orchestrator Parameters:

• model: The LLM model to use for making compliance decisions
• temperature: Controls randomness in responses (0.0 = fully deterministic)
• system_prompt: Defines the role and behavior of the compliance orchestrator
• task_prompt: Instructions for making decisions, with these placeholders:
  • {extracted_evidence}: Facts gathered from all chunks and sections
  • {policy_class}: The category of policy class being evaluated
  • {rule}: The specific rule text

Policy Classes

Define policy types and specific rules to evaluate:

policy_classes:
  - policy_type: "global_periods"
    questions:
      - "If a procedure has a global period of 000 or 010 days, it is defined as a minor surgical procedure..."
      - "If a procedure has a global period of 090 days, it is defined as a major surgical procedure..."

  - policy_type: "same_day_service_rules"
    questions:
      - "Since National Correct Coding Initiative (NCCI) Procedure-to-Procedure (PTP) edits are applied..."

Document Classes

Specify which document types should be validated:

classes:
  - name: "PA-Administrative"
    description: "Prior Authorization administrative information"
    attributes:
      - name: "patient_name"
        description: "Full name of the patient"
      - name: "insurance_policy_number"
        description: "Insurance policy or member ID"

Customizing Recommendation Options

The default recommendation options are Pass/Fail/Information Not Found, but you can customize these for your specific use case:

Healthcare Compliance Example

rule_validation:
  recommendation_options: |
    Compliant: Fully meets regulatory requirements.
    Non-Compliant: Does not meet requirements.
    Requires Review: Manual review needed.
    Not Applicable: Rule does not apply to this case.

Financial Audit Example

rule_validation:
  recommendation_options: |
    Approved: All criteria satisfied.
    Rejected: Criteria not met.
    Pending: Additional documentation required.

policy_classes:
  - policy_type: "loan_eligibility"
    questions:
      - "Applicant must have minimum credit score of 650..."
      - "Debt-to-income ratio must not exceed 43%..."

  - policy_type: "documentation_requirements"
    questions:
      - "Two years of tax returns must be provided..."
      - "Proof of employment must be current within 30 days..."

The statistics in the final summary will automatically use your custom options:

{
  "recommendation_counts": {
    "Compliant": 15,
    "Non-Compliant": 3,
    "Requires Review": 2
  }
}

Rule Configuration Best Practices

Writing Effective Rules

1. Be Specific: Include clear criteria and conditions

   questions:
     - "If a procedure has a global period of 090 days AND an E&M service is performed on the same date..."

2. Provide Context: Include relevant definitions and examples

   questions:
     - "CPT code 31500 describes an emergency endotracheal intubation. For example, if intubation is performed in a rapidly deteriorating patient..."

3. Structure Complex Rules: Use JSON format for rules with multiple components

   questions:
     - |
       {
         "cpt_codes_affected": ["31500"],
         "rule_text": "If laryngoscopy is required...",
         "bundled_services": ["laryngoscopy for elective or emergency placement"],
         "separately_reportable_conditions": ["intubation in rapidly deteriorating patient"]
       }

Organizing Policy Types

Group related rules into logical policy types:

policy_classes:
  - policy_type: "eligibility_rules"
    questions:
      - "Patient must be enrolled in insurance plan..."
      - "Coverage must be active on date of service..."

  - policy_type: "medical_necessity"
    questions:
      - "Procedure must be medically necessary..."
      - "Documentation must support diagnosis..."

Output Formats

JSON Output

Located at s3://{bucket}/{document_id}/rule_validation/consolidated/consolidated_summary.json:

{
  "document_id": "doc_123",
  "overall_status": "COMPLETE",
  "total_policy_types": 4,
  "overall_statistics": {
    "total_rules": 15,
    "recommendation_counts": {
      "Pass": 12,
      "Fail": 2,
      "Information Not Found": 1
    }
  },
  "rule_summary": {
    "global_periods": {
      "status": "COMPLETE",
      "total_rules": 2,
      "Pass": 2,
      "Fail": 0
    }
  },
  "supporting_pages": ["1", "2", "3", "5"]
}

Two things to know if you read this file programmatically.

supporting_pages is always a list of strings. Page references reach the summary from two engines and from model output, so they arrive as strings and as numbers; the document-level list canonicalises them to strings, drops duplicates, and orders numeric references by value followed by anything non-numeric by codepoint (so 'Zebra' precedes 'apple'). The per-rule lists under rule_details are not canonicalised — they hold exactly what each rule’s response returned, which is the record of the evidence cited, and their order is whichever engine produced them.

overall_status is "ERROR" with an error field when consolidation did not complete. The statistics alongside it are real and consistent with each other — a rule is counted once it has been read, and pass_percentage is computed over the rules counted — but they cover only what was reached before the failure, so read the counts as a floor on what was evaluated rather than as the document’s total. The Markdown report states this above the statistics table. A document whose per-section validation failed is a different case and is reported per section — see Where a failed rule validation shows up.

Markdown Output

Located at s3://{bucket}/{document_id}/rule_validation/consolidated/consolidated_summary.md:

# Rule Validation Summary

**Document ID:** doc_123
**Status:** COMPLETE
**Total Rule Types:** 4

## Overall Statistics

**Total Rules:** 15
**Pass:** 12
**Fail:** 2
**Information Not Found:** 1

## Rule Type: global_periods

**Total Rules:** 2
**Pass:** 2

### Rules

| Rule | Recommendation | Reasoning | Supporting Pages |
|------|----------------|-----------|------------------|
| Minor surgery rule | Pass | Evidence found on page 1... | 1, 3 |

Performance Optimization

Rate Limiting

Control concurrent API calls to prevent throttling:

rule_validation:
  semaphore: 5  # Adjust based on your API limits

Prompt Caching

Place static content before the <<CACHEPOINT>> marker and dynamic content after:

task_prompt: |
  You are an insurance evaluator...

  <<CACHEPOINT>>

  {recommendation_options}

  <user_history>
  {DOCUMENT_TEXT}
  </user_history>

This caches the static instructions and only processes the dynamic document content, reducing costs.

Chunking Configuration

Adjust chunking parameters for your document sizes:

rule_validation:
  max_chunk_size: 8000  # Increase for longer documents
  overlap_percentage: 10  # Increase for more context preservation

Integration with Extraction

Rule validation works seamlessly with extraction results:

1. Extraction Phase: Extracts structured data from documents
2. Rule Validation Phase: Validates extracted data against business rules
3. Combined Output: Both extraction and validation results available in document object

Access both results:

# Extraction results
extraction_uri = section.extraction_result_uri
extraction_data = s3.get_json_content(extraction_uri)

# Rule validation results
validation_uri = document.rule_validation_result.output_uri
validation_data = s3.get_json_content(validation_uri)

Monitoring and Debugging

CloudWatch Metrics

Monitor rule validation performance:

• Execution duration per section
• Token usage and costs
• Error rates and types

Logs

Check CloudWatch logs for:

• rule-validation-policy-classification-function: Policy classification logs
• rule-validation-function: Section-level evaluation logs
• rule-validation-orchestration-function: Orchestration logs

Where a failed rule validation shows up

When rule validation fails, the failure is recorded on the sections it affects, so the document’s Sections panel names them instead of leaving the explanation only in the Step Functions cause and the CloudWatch log. The Status column reads Failed, and hovering the status shows the issue with the technical cause behind it.

Three codes are written, all at error severity, because they describe different situations:

Code	Written by	Meaning
rule_validation_failed	the per-section rule validation step	This section’s own rule validation did not complete, so it has no compliance verdict. The cause is whatever the rule-validation service recorded — a missing page, a solver timeout, a model error.
rule_validation_not_consolidated	the orchestration step	Every section was validated, but the step that turns those results into the document’s single compliance decision failed. No section has a verdict, so every section carries this issue.
section_processing_failed	the collate step	That step found this section’s earlier processing had failed.

⚠️ Look on the document’s own page, not the document list. The Sections panel always shows these, because it reads the section’s issues directly. The document list’s Processing Issues badge is only refreshed by the two steps that write the whole document — the orchestration and collate steps — so a rule_validation_failed recorded by the per-section step can leave that badge showing its previous value, commonly zero. The per-section write updates one section atomically and cannot know the document-wide total: it does not read the stored item, and ten sections are being written concurrently. extraction_failed reaches the badge the same way.

The exception is unchanged by this: the Step Functions cause still carries the same message it always did. What changed is that the document’s own record now carries it too.

A transient failure is not marked, and it is retried. A throttle, a read timeout, a dropped connection or a model that is not ready is retried by the state machine — eight attempts at 2.5× backoff from ten seconds, about 2.8 hours of backoff, on both the per-section and orchestration steps — so flagging the sections would show them failed for as long as that ladder runs and then clear itself.

The two decisions come from the same place, which is what makes the suppression trustworthy. Step Functions decides whether to retry from the Python exception’s class name, while transience is a property of the error code and the exception’s cause chain — a question no retry list can ask. So rule validation classifies the failure once, in idp_common.utils.transient_errors, and re-raises every transient cause under one class name that all three rule-validation states list. A record is withheld exactly when a retry is genuinely coming.

If a ladder exhausts every attempt, the document fails with the explanation in the Step Functions cause and the sections are not flagged. A step cannot tell its last attempt from its first — Step Functions does not tell it which attempt it is on — so the final one suppresses the record for the same reason the first did. The failure is still counted by the failure alarms and the dead-letter queue; what is missing is the per-section diagnosis. Extraction and assessment behave identically.

A transient fault is never answered with a verdict

Information Not Found is a verdict: it is one of the configured recommendation_options, it is counted in the document’s rule-validation summary, and downstream consumers act on it — the sample health-insurance review feature reads a document made mostly of them as insufficient documentation. So it has to mean “the document does not evidence this rule”, and not “we could not reach the model”.

Rule validation therefore separates the two everywhere it could previously conflate them. A deterministic failure for one rule still yields Information Not Found with the reason in its reasoning field, which is what keeps one unparseable rule from discarding every other rule’s answer. A transient failure raises instead, so the step is retried and the rule gets a real answer. The same split applies to the Z3 engine’s value-extraction call, to the summarisation step that turns per-section facts into the document’s decision, and to the consolidation step — which previously returned an empty result rather than failing, so a throttle could finish a document with no verdicts, no failed status and nothing recorded anywhere.

A related fault is handled the same way, because it silently changed a result rather than reporting a failure: a page whose text could not be read is a page whose policy-matching regexes never ran, so a policy type evidenced only on that page went unmatched and none of its rules were validated. It is now retried rather than absorbed.

The cleanup that removes the previous run’s per-section results is best-effort, and consolidation does not depend on it. Consolidation reads the explicit list of per-section outputs this run produced, passed to it by the orchestration step, so an object the cleanup failed to delete is not in that list and is not read. A surviving object costs storage and makes the prefix confusing to read by hand; it cannot reach a verdict. A transient failure of the cleanup is still retried, because a retry is the right answer to a throttle on its own terms.

⚠️ If you call the consolidation service directly, pass section_uris. Omitting it falls back to listing the prefix, which is the right behaviour for re-consolidating an existing prefix by hand and the wrong behaviour for a run of your own: on a reprocessed document that prefix can still hold the previous run’s verdicts for the same document, which are plausible enough to be consolidated and acted on. See the module README for the parameter’s three values.

A document-scope explanation has no section to attach to, and is not invented one. The collate step also collects free-text errors that belong to the document as a whole rather than to any one section — page-level OCR problems, for example. Those are not attributed to a section, because picking one would be a claim the pipeline cannot support; they remain in the Step Functions cause and the step’s log. Per-page classification failures behind them are separately recorded on their own sections by the classification stage.

A very long explanation is abridged in the middle, on the same 4 KB bound every processing issue’s technical cause carries, with the error count written at the front so it survives the abridgement. The unabridged text is in the step’s CloudWatch log.

Common Issues

High Token Usage:

• Reduce max_chunk_size
• Optimize rule descriptions
• Use prompt caching effectively

Slow Processing:

• Increase semaphore value — it bounds both the fact-extraction and the consolidation step, so raising it shortens both. Watch for Bedrock throttling as you do: the account-level call rate is this value times the number of documents in flight.
• Reduce number of rules
• Use faster model (e.g., Claude Haiku)

Inconsistent Results:

• Set temperature: 0 for deterministic output
• Improve rule clarity and specificity
• Add more context in rule descriptions

Cost Considerations

Rule validation costs depend on:

• Number of rules evaluated
• Document size and chunking
• Model selection
• Prompt caching effectiveness

Cost Optimization Tips:

1. Use prompt caching (can reduce costs by 50-90%)
2. Choose appropriate model (Haiku for simple rules, Sonnet for complex)
3. Minimize rule redundancy
4. Optimize chunk sizes to reduce API calls

API Reference

For detailed API documentation, see the Rule Validation Module README.

Examples

Healthcare Prior Authorization Example

We provide a complete healthcare example demonstrating prior authorization validation against NCCI medical coding rules.

Sample Documents:

• Prior Authorization Document: samples/rule-validation/respiratory_pa_packet.pdf

  • Synthetic multi-page respiratory therapy prior authorization request
  • Contains multiple sections:
    • Patient Information (demographics, insurance details)
    • Clinical Information (diagnoses, medical history)
    • Evidence Documents (supporting clinical documentation)
    • Operative Logs (procedure details, CPT codes)
    • Claims Data (billing information, service dates)

• NCCI Policy Manual: samples/rule-validation/NCCI Medicare Policy Manual.pdf

  • National Correct Coding Initiative policy reference
  • Source: CMS NCCI Policy Manual Chapter 5 (2024)
  • Contains medical coding rules, bundling guidelines, and compliance requirements

Configuration Files:

• Step 1 - Policy Discovery: extracted rules are written into policy_classes on the active config version (see docs/policy-discovery.md)
• Step 2 - Rule Validation: config_library/unified/rule-validation/config.yaml

This example includes:

• NCCI coding rules
• Global period validation
• Same-day service rules
• Bundling and component service rules
• Prior authorization document classes

Note: This is a reference implementation. You can replace these rules with your own domain-specific requirements.

Testing with Notebooks

For hands-on examples, see: notebooks/misc/e2e-holistic-packet-classification-rule-validation.ipynb

This notebook demonstrates:

• Loading and configuring rule validation
• Processing document sections
• Consolidating results
• Viewing output formats

Skipping Rule Validation for Excluded Classes

If a document class is marked with x-aws-idp-exclude-from-processing: true (see Excluding Static Pages in the Classification docs), RuleValidationService.validate_document_async short-circuits for any section classified as that class:

• No LLM call is made inside the per-section process_one_section coroutine — it returns immediately with (section_responses={}, 0, False) meaning “no rule responses, zero chunks created, no chunking needed”.
• No output file is written — rule validation has no per-section artifact of its own, and the extraction stub (result.json) already records that this section was skipped.
• Rule validation aggregation simply sees zero responses for the skipped section and proceeds normally with the active sections.

This keeps rule validation clean on mixed documents (e.g. a DS-11 passport application package where only the 2 form pages need rule checks — the 4 boilerplate pages are noise and produce no rule triggers).

Best Practices

1. Start Simple: Begin with a few clear rules and expand gradually
2. Test Thoroughly: Validate rules against known good/bad examples
3. Monitor Costs: Track token usage and optimize as needed
4. Use Caching: Always place static content before <<CACHEPOINT>>
5. Clear Recommendations: Define unambiguous recommendation options
6. Document Rules: Include context and examples in rule descriptions
7. Version Control: Track rule changes and their impact on results
8. Regular Review: Periodically review and update rules based on results