Build an extraction audit log that records each AI-produced field with its source location, then route a sample to human verification and record agreement and corrections. The log becomes the audit trail for what the AI produced, what humans confirmed, and what was corrected.
- Define the extraction fields and their allowed values before any AI runs. Use structured fields such as study identification, status, design, eligibility, locations, outcomes, and reported results, following the ClinicalTrials.gov data element definitions where they apply. For each field, specify the source location to record, such as a section, table, or page.
- Run the AI extraction and log every field with its source location. The log should capture the field name, the extracted value, and the exact location in the source document where the value was found. This creates a traceable link between the AI output and the evidence.
- Select a sample of extracted records for human verification. The sample can be all records or a subset; record the selection rule. For each sampled field, a human reviewer checks the source location and marks the value as confirmed, corrected, or unresolved.
- Record agreement and corrections in the log. For each verified field, note whether the human agreed with the AI value, and if not, the corrected value and the reason. This produces a measure of agreement and a record of changes.
- Summarize the audit log for reporting. Report the number of fields extracted, the number verified, the agreement rate, and the types of corrections. Use this summary to describe the extraction process in the review's methods, consistent with PRISMA 2020 reporting of methods and findings.
The audit log supports transparency about AI involvement and provides a basis for readers to assess the extraction. It also helps identify fields where AI performance is lower and may need closer review.