An oil and gas knowledge assistant that cites the approved revision

Topics: oil and gas knowledge management; AI-assisted document search; permission-aware retrieval; approved document revisions.

A user asks where to find the current supplier-onboarding form. The assistant returns a fluent answer with a citation. The cited document is an obsolete revision copied into a shared folder, while the current form sits in the controlled repository.

A citation is useful only if it points to evidence that is both relevant and eligible for that user and question. An oil and gas knowledge-assistant pilot should prove those conditions before expanding its document population or the kinds of questions it answers.

Choose questions with a bounded consequence

Begin with administrative navigation: locating an approved form, identifying a document owner or finding the submission route stated in an authorised process document. Exclude operating instructions, engineering decisions, permit authority, emergency actions and technical limits from the pilot.

The document owner approves the corpus and question scope. Security approves access and retention. Business reviewers define correct answers and acceptable abstentions. Use synthetic, public or expressly authorised low-sensitivity documents at the start. Do not copy an entire operational repository into a vector database simply because a connector can reach it.

A conventional search page may already meet the need. Establish that baseline so the pilot can compare an assistant with keyword search and filtered document navigation, not just with a user guessing filenames.

Make eligibility part of retrieval

Each document needs a stable identifier, revision, approval status, effective scope, owner, source location and access policy. Preserve the relationship between current and superseded revisions. A historical question may legitimately need an old revision, but the first pilot can restrict itself to current administrative questions and decline historical interpretation.

Before retrieval, determine the user's permitted corpus and the applicable approved versions. Apply those controls in the application and retrieval layer, rather than asking the model to ignore restricted text it has already received. Carry the document identity and exact revision into the citation.

Run the pilot against a read-only document/search interface with a separate staging index. Its service identity has no permission to revise source documents or change approvals. Permit writes only to the controlled pilot index, evaluation results and minimal audit records. The document owner approves ingestion, while security verifies that the index and its caches enforce the intended access scope.

The answer should link to the relevant passage in the controlled source, subject to the user's access. Retain enough provenance to reconstruct which evidence was supplied to the model. W3C PROV offers general concepts for recording those relationships. W3C provenance overview

Two revisions and two users

Illustrative example. Administrative procedure AP-12 revision 2 directs supplier submissions to an old mailbox. Revision 3 directs them to an approved portal. Revision 2 is superseded. The current procedure is available to procurement staff; contractors have a separate approved public instruction.

A retrieval system that finds revision 2 because its wording resembles the question has failed the version test even if it quotes accurately. A system that gives the contractor the correct internal answer has failed the access test. Neither failure should be hidden inside an overall answer-quality score.

Check the surrounding interface as well as the answer. A restricted filename, search snippet, citation title or suggested follow-up can reveal information even when the document body is hidden. In the contractor test, inspect all of those surfaces and any downloadable transcript.

Build an evaluation set with deliberate abstentions

Collect typical questions from the approved workflow and have document owners label the supporting passages. Include ambiguous wording, absent answers, obsolete revisions, conflicting current records, inaccessible sources and questions outside the allowed scope.

A proposed pilot might use 80 distinct questions in three primary groups: 40 ordinary answerable questions, 20 ordinary questions requiring abstention and 20 access or revision challenge cases. Label the expected answer or abstention for each challenge case as well. These counts are illustrative planning choices. Secondary tags can overlap—for example, a challenge may involve both access and revision—but should not add another question to the total.

Keep development questions separate from held-out evaluation questions. Freeze the corpus snapshot, retrieval configuration, model version and scoring rules for the final run. Record later changes as new evaluation versions rather than comparing results from different corpora as though they were equivalent.

NIST's Generative AI Profile identifies risks including unsupported confident output and privacy failures. The pilot below turns those broad concerns into specific tests; it does not claim certification against the framework. NIST Generative AI Profile

Score the claims, citations and refusals

Measure whether each material answer claim is supported by the cited passage, whether the cited revision is eligible and whether the user can access it. A citation that opens the right document but does not support the statement is not a pass.

If an answer correctly identifies the portal but invents a submission deadline, record the unsupported deadline separately; the valid portal citation does not support the entire answer. Retain both claim-level results and an answer-level result showing whether any material claim failed. This prevents several easy factual statements from concealing one consequential invention.

Measure abstention in both directions: unsupported questions that were correctly declined, and answerable questions declined unnecessarily. Also record completion time and reviewer corrections compared with the search baseline. An assistant that refuses everything may avoid unsupported answers but provides little business value.

Keep the baseline comparison fair. Use comparable question sets and alternate which method reviewers try first, so time saved is not simply the benefit of having just found the answer by another method. Separate time to locate evidence from time to verify and correct the response; fluent text can shorten the first task while lengthening the second.

Agree acceptance criteria with the owners before the held-out run. Treat access leakage, invented source references and answering excluded operational questions as individual control failures, regardless of the average score. Zero observed leakage in the chosen tests supports a bounded decision; it is not a permanent guarantee.

Test hostile content and changing permissions

A retrieved document may contain instructions aimed at the assistant rather than information for the user. OWASP describes this indirect prompt-injection risk, including the limits of retrieval alone as a defence. OWASP prompt-injection guidance

Treat retrieved text as untrusted evidence. Give the pilot no tools to send messages, change records or execute operational actions. Test a document that asks the model to ignore its instructions, reveal other material or fabricate an approval. The expected result is continued adherence to the allowed task and access boundary, with the case logged for review.

Revoke a user's access and withdraw a document during testing. Verify that new retrievals, cached responses and re-opened answer histories enforce the current access decision. Do not rely on a new prompt to erase restricted content from an existing conversation: the application must control what earlier evidence is passed back to the model and what history is displayed. If permission freshness cannot be established, fail closed for the affected content and offer the authorised fallback.

Revocation cannot undo information already seen or remove a copy already downloaded by an authorised user. Record that limit in the design. Control future exports and retention under the agreed policy, and distinguish those controls from any claim to retrieve previously disclosed information.

Stop, recover and review the result

Stop the affected service on access leakage, unsupported operational guidance, loss of citation provenance or unresolved current-version conflicts. Preserve a minimal authorised incident record without spreading restricted text into broad logs. Disable the affected corpus or connector and return users to approved document search or the document owner.

After correction, reindex or invalidate caches as required, then rerun the affected tests and the core regression set. Review the results before resuming. Expansion should specify the next document family, users and question types; success on supplier forms does not establish suitability for technical procedures.

Summary and next actions

The pilot should show that users can find a supported answer in less time while still checking its source. Review claim support, current revisions, access controls and unnecessary refusals separately before deciding which document family to try next.

1. Approve a small administrative corpus and name its document, security and business owners.
2. Establish a search baseline and label answerable, absent, conflicting and restricted questions.
3. Test claim support, revision eligibility, access changes and hostile document content.
4. Demonstrate fallback and recovery, then decide whether the bounded result justifies expansion.

A pilot assessment can start with a document-type list, revision metadata, user-role map and synthetic questions. No operational manuals or sensitive repository export is needed for that first design discussion.