Prompt-Based Extraction: How to Use LLMs Within Your Governed Document Processing Pipeline

ABBYY's Document AI platform Vantage now supports prompt-based extraction, a feature that lets you connect large language models (LLMs) directly to your document processing workflows and extract structured data using natural language instructions. No custom coding, no separate platform, no upfront model training required. This post walks through what the feature does, supported LLM providers, how to connect them, and the practical capabilities that make it enterprise-ready from day one.

Key takeaways

• ABBYY Vantage supports a "Bring Your Own Model" (BYOM) approach, with native connections to OpenAI, Microsoft Foundry, and Google AI Studio (Gemini).
• Prompt-based extraction is a drag-and-drop activity in Advanced Designer that fits directly into existing document skills.
• Pre-engineered prompts are included to accelerate deployment, with full support for editing and refinement.
• You can test and refine prompts against real documents before publishing a skill to production.
• Selective document snippet sending gives you control over what data leaves your organization and how many tokens you consume.

 

What is prompt-based extraction in the ABBYY Vantage platform?

Prompt-based extraction allows you to use natural language instructions to extract structured data from documents using an LLM of your choice. Instead of training a traditional machine learning model, you describe the fields you want to extract, define the output format, and let the LLM handle the rest.

The feature lives inside ABBYY Vantage's Advanced Designer as a dedicated activity. You add it to a workflow canvas the same way you would add any other extraction activity, connect it to an LLM provider, write or adjust a prompt, and run it against your documents. The result maps directly to the output fields you have defined in the skill.

This matters because it extends what the platform can do. Purpose-built machine learning models remain the right choice for high-volume, structured, repeatable documents. Prompt-based extraction fills the gap for documents with high layout variability, low-volume high-value cases, and scenarios where you have no labeled training data to start from.

LLM connectivity: Bring your own model

ABBYY Vantage takes a model-agnostic approach. You supply the LLM credentials; the platform handles the rest. Connections are configured once at the tenant level and are immediately available to all skill designers in your organization.

Which LLM providers are currently supported?

Three providers are supported through native, pre-built connections:

• OpenAI ChatGPT: Direct connection using an API key and model name (e.g., gpt-4, gpt-4o). Setup takes a few minutes.
• Microsoft Foundry: For organizations running on Azure infrastructure, this option connects to a wide variety of models hosted within Azure. Configuration requires an API key, a full endpoint URL, and a model name. This route is well-suited to enterprises that need data residency controls or have existing Azure commitments.
• Google AI Studio (Gemini): Connects to Google's Gemini models using an API key and model selection (e.g., gemini-2.0-flash, gemini-2.5-pro). This is a straightforward option for teams already working within Google Cloud.

Each provider has different configuration requirements, but the setup process follows the same pattern: navigate to Configuration in the Admin section, open Connections, click "Add Connection," select your provider, enter your credentials, and run a connection test. A green checkmark confirms the connection is live. Tenant administrators can also override the default connection per environment after a skill is published, without needing to republish the skill.

What about LLM providers not on this list?

While we are working to extend the native connectivity within the prompt-based extraction feature, you can connect other models using the Custom Activity feature. This feature extends connectivity beyond the three pre-built providers, allowing you to invoke any third-party REST service from within a document skill. This makes it possible to route documents to other LLMs, including models like Anthropic Claude, self-hosted models, or proprietary enterprise LLMs. You define the document fields, text, and image formats before sending them externally, and you choose output formats beyond the defaults. This gives you control over what data you share and reduces coding effort significantly. We also offer pre-built connectors for additional providers through the ABBYY Marketplace.

Key feature highlights

Pre-engineered prompts for immediate use

Writing effective extraction prompts from scratch takes time and expertise. The prompt-based activity in ABBYY Vantage includes pre-engineered prompts built on decades of document processing experience. These are not generic templates. They encode specific knowledge about how business documents are structured, how field labels vary across languages and formats, and how to instruct an LLM to extract data verbatim without inferring or reformatting values.

A typical pre-engineered prompt includes four components:

• Role: Defines what the LLM should act as (e.g., "data extraction model") and sets the scope of the task.
• Output format: Specifies the exact JSON structure the LLM should return, with field names that map directly to the skill's output fields.
• Field-specific rules: Provides recognition patterns, format specifications, location hints, and exclusion rules for each field. For example, an IBAN field rule will list label variants across multiple languages ("Bankverbindung," "Coordonnées bancaires," "Payment Details"), specify that the full value should be extracted exactly as printed, and explicitly exclude IBAN values belonging to the customer rather than the vendor.
• Strictness rules: Prevents hallucination by instructing the LLM never to generate or infer values. If a field is not clearly present, it is omitted from the output rather than guessed.

You can adjust these prompts to add context, expand field rules, or introduce conditional extraction logic. The pre-engineered structure gives you a reliable starting point rather than a blank page.

Testing and optimizing prompts at design time

Before a skill reaches production, you can test prompt-based activities directly against real documents inside Advanced Designer. Select a document from your test set, run the activity, and review the extracted fields in the predicted view. Each extracted value shows the field name, the text pulled from the document, and the location within the document where the value was found.

This design-time testing loop lets you identify issues quickly and iterate without affecting live processing. Common refinements include:

• Adding more label variations when a field is missed due to formatting differences
• Strengthening strictness rules when the LLM attempts to fill empty fields with inferred data
• Specifying escape sequences for multiline values such as addresses
• Clarifying data ownership rules when the document contains both vendor and customer data for the same field type

The iterative approach, changing one rule at a time and testing against multiple documents, produces more consistent results than optimizing for a single example. Document format also affects results. The platform supports several options for how documents are sent to the LLM, including plain text, formatted text, and annotated text. Annotated text is the recommended format because it preserves document structure while being efficiently processed, and it avoids the higher token costs and limited support associated with sending raw PDF files.

Selective document snippet sending for token control and data privacy

Sending an entire document to an external LLM API on every extraction request creates two problems: cost and data exposure. The ABBYY Document AI platform addresses both through selective processing.

Because prompt-based extraction runs as one activity within a broader document skill, you can configure it to process only specific parts of a document. The platform's optical character recognition (OCR) and layout analysis runs first, producing a clean, structured text representation of the document. You then send only the relevant sections or fields to the LLM, rather than the full raw document.

The practical benefits are direct:

• Token consumption decreases because the LLM receives less input per request.
• Data privacy improves because sensitive fields or sections you do not want to send externally never leave the pipeline.
• Accuracy increases because the LLM receives clean, structured text from ABBYY's OCR rather than raw pixels or poorly formatted input.

This approach also reduces the risk of prompt injection, a known vulnerability when raw document content is passed directly to an LLM without preprocessing.

How prompt-based extraction fits into a hybrid workflow

Prompt-based extraction works alongside, not instead of, the platform's other extraction methods. A typical invoice skill might use a pre-trained ML extraction model for structured fields like line items and totals, then add a prompt-based activity specifically for vendor banking details that vary too much across document layouts for a trained model to handle reliably.

The hybrid pipeline means each extraction approach handles what it does best:

Outputs from prompt-based activities flow into the same validation, human-in-the-loop review, and export steps as any other extraction result. The LLM's response is grounded in the source document, and because field values are extracted verbatim and mapped to specific line locations, you get a traceable output that supports compliance requirements.

What's next for LLM integration in the ABBYY Document AI platform?

To give you a sneak peek at what we have in the works for you: we are broadening LLM ecosystem support to include general-purpose and self-hosted proprietary models beyond the current pre-built connections. We are also developing prompt management and optimization tooling, enhanced governance and hallucination guardrails for LLM outputs, and ABBYY-hosted language models optimized for document processing to enable efficient and reliable zero-shot extraction without prior training.

Start building with prompt-based extraction

Prompt-based extraction closes one of the most common gaps in document automation: the ability to handle documents that are too variable or unstructured for traditional model training, without building a custom integration from scratch. The combination of pre-built LLM connections, pre-engineered prompts, design-time testing, and selective data sending means you can go from connection setup to a working extraction skill in under 30 minutes.

For full technical guidance, see the documentation on configuring LLM connections and extracting data with prompt-based activities. To explore what's possible across the full platform, visit abbyy.com/ai-document-processing.