Evaluate AI extraction on construction documents with conflicting evidence

Choose one document family and a limited set of fields. An invoice, inspection record and drawing schedule have different structures and review implications. State what each extracted value will support: search, a comparison, a proposed match or another bounded task.

For each important field, define the expected format, unit, source location and validation rule. Identify the cost of a false value and the reviewer who handles uncertainty. A wrong supplier entity may matter more than a slightly inaccurate description, so one average accuracy score is a weak acceptance measure.

Keep the original file and its document identity. A page reference should lead to the exact source version used by the extraction run. If the source is replaced, retain the historical version or an approved immutable copy so the proposal remains auditable.

In this fictional example, Revision A shows 120 units and Revision B shows 128. The system should return both values with their revision references and evidence. If it cannot establish which document is approved for the intended use, it should flag the conflict and leave the selection to the authorised reviewer.

The reviewer checks the document register and relevant decision history. A later upload date alone does not prove that Revision B supersedes Revision A for this purpose. Once the reviewer chooses the applicable source, record the decision and the values excluded from the proposal.

Oracle Aconex's document metadata reference distinguishes revision, system version and review status. That is a useful model for preserving the evidence needed here. Extraction should not flatten those attributes into a single latest-file field.

For stable machine-readable forms, structured import or deterministic parsing may be more reliable and easier to maintain than a model. OCR can recover text from scans. A model can help interpret varying layouts or labels, while explicit rules check arithmetic, identifiers and allowed values.

Microsoft's invoice extraction documentation illustrates a specialised model that extracts invoice fields and line items. It does not establish the correctness of a construction workflow or the applicable revision of a project document. Test the selected product and version on your actual document family before extrapolating from a sample demonstration.

Keep each stage visible enough to diagnose failures. If OCR reads 1.20 as 120, the reviewer needs to see the source crop or page context. If the text is correct but the unit is wrong, that is a different failure and needs a different control.

Include clean digital files, poor scans, rotated pages, multi-page tables, missing fields, similar supplier names, conflicting revisions and unusual units. Add a duplicate file with a different filename and a genuine new document with a reused number. Split tuning material from evaluation material so the final score is not just a measure of memorised examples.

Have qualified reviewers establish expected outcomes, including cases where no single value should be accepted. Preserve disagreement for adjudication. A reference dataset containing guesses will reward the model for reproducing those guesses.

Report performance by field and failure type. Count false accepted values, missed conflicts, abstentions and reviewer corrections. Measure end-to-end handling effort as well. A proposal that reads most fields correctly but hides the source may take longer to verify than manual entry.

Confidence scores vary by product, model, field and input quality. Evaluate whether the score is useful for your sample before setting a routing threshold. There is no universal percentage at which a project quantity, supplier identity or inspection outcome becomes approved.

Use deterministic checks where possible: required fields, approved units, arithmetic consistency and valid record references. A failed check should create an explicit exception. Do not automatically repair an inconsistent number merely to make the total balance; preserve the source and ask for review.

For high-consequence fields, require the appropriate review even when the model is confident. Show a proposed value beside its evidence and make reject or cannot determine legitimate outcomes. The reviewer should never have to invent a value to clear the queue.

A scanned document may contain text that tells a model to ignore previous rules, retrieve another file or change an output. Such content must remain part of the untrusted source, not become authority for the application. OWASP's indirect prompt-injection guidance describes this risk with external files and recommends limiting privileges and separating untrusted content.

Give the extraction service only the access required for the bounded task. It should not hold credentials for approving drawings, posting invoices or sending project documents elsewhere. Validate output structure and permitted references outside the model, and require the authorised workflow for consequential actions.

Add malicious and accidental instruction-like text to the challenge set. Verify that it cannot expand access, change the approval route or suppress conflicting evidence. Record the result as a tested boundary, not a claim that all possible attacks have been eliminated.

Run a limited trial with the actual reviewers. Record how often they open the source, correct a field, reject a proposal or request a better document. Separate extraction defects from source-quality problems and unresolved business decisions. Each needs a different owner.

Stop the affected flow if it leaks data, hides a conflicting revision or performs an unauthorised action. Preserve the proposals and logs, use the manual process and reconcile the affected records before restarting. The AP pilot guide gives a concrete scope for this approach.

The useful result is evidence that reviewers can inspect and accept, including a clear admission when the system cannot decide.

1. Select one document family, field set and downstream decision.
2. Build a reviewed challenge set with conflicting revisions and ambiguous values.
3. Test field results, source traceability, abstention and access boundaries together.
4. Expand only after the review queue improves the work without hiding uncertainty.