Checking whether every claim in a document was backed by its citation took reviewers weeks.
Check each claim and its citation as a pair, and give reviewers a list of issues instead of a document to reread.
Review went from weeks to hours. 1,000+ issues flagged.
- Review workflow
- Weeks → hours
- Issues flagged
- 1,000+
- Shipped from scratch
- 0 → 1
Context
- Company
- VectorShift (YC S23)
- My role
- Shipped it from zero to one
- Users
- Teams reviewing financial and diligence documents
- Framing
- Quality as a product problem, not a QA step
The problem
Financial and diligence documents make claims and cite sources for them. Someone has to check that the two actually match.
A citation being there says nothing about whether it supports the claim. A number gets updated in one place and not another. A source gets paraphrased until it says something different.
Reviewers were doing this by hand, and a review took weeks.
- Review workflows took weeks before Verification.
- Most of that time went into confirming claims that were fine, in order to find the few that were not.
Constraints & non-goals
Constraints
- The product has to judge the relationship between a statement and its evidence, not just whether a citation exists.
- False confidence is worse than no tool. The product flags. A reviewer decides.
- Every flag has to show the claim and the cited passage side by side.
Out of scope
- Certifying a document as correct.
- Rewriting the document automatically.
- Fact-checking against the open web.
Options I weighed
- NOT CHOSEN
A citation-presence checker
Easy to build and close to useless. The costly errors are cited claims that the source does not support.
- NOT CHOSEN
An AI summary for reviewers
A summary is one more text to trust. Reviewers needed fewer things to read, not a different thing.
- CHOSEN
Claim-and-citation checking with an issue queue
It targets the real failure and turns review from rereading into triage.
The spec
Users
- Analysts who review documents before they go out.
- Leads who sign off on them.
Requirements
- Pull claims and their citations out of a document.
- Check whether each citation supports its claim.
- Flag issues for review.
- Show the claim, the citation and the reason for the flag together.
Acceptance criteria
- Every flag links to the exact claim and cited passage.
- A reviewer can dismiss a flag in one action, and the dismissal is recorded.
- A document with no flags still shows what was checked.
Guardrails
- The product never labels a document “verified”. It reports the issues it found.
- Dismissal rate is tracked, so noisy checks get fixed instead of ignored.
What shipped
Extract
Claims and their citations are pulled from the document.
Pair
Each claim is matched to the passage it cites.
Check
AI checks whether the evidence supports the claim.
Triage
Reviewers work through flagged issues instead of rereading everything.
What changed
Review workflows went from weeks to hours.
- Verification has flagged 1,000+ issues.
- Quality became a surface reviewers use every day, not a checklist at the end.
HONEST CAVEATIssues flagged is not the same as errors prevented, and no flag list is guaranteed complete. Both numbers describe the workflow, not accuracy.
Reflection
Quality is a product problem, not a QA process. Treating it that way changed what we built.
If I did it again, I would instrument dismissals from the first release. The ratio of dismissed to accepted flags is the best single signal of whether reviewers trust the tool.
Last updated · Company tenure is VectorShift · Jul 2025–present, not a claim about individual project dates.