This note examines traceable evidence as a foundation for problem discovery and learning from feedback. See the AI-native organization agenda.
A research system becomes easier to evaluate when every important claim carries its own evidence record. Polished prose should be a view over that record, not its substitute.
The question
What would make an automated research note useful enough to inform a product decision? A long answer with many links is a weak acceptance criterion. The reader needs to know which claims are supported, which conclusions are inferred, and which missing facts could reverse the recommendation.
Our working hypothesis is that a claim-level evidence record can improve reviewability. The claim should remain identifiable even when a summary is rewritten, an article is shortened, or a source is updated. This is a design proposal, not a validated performance claim.
Two relevant research patterns
WikiChat: verify the factual content
WikiChat combines language-model generation with Wikipedia retrieval and factual grounding. Its pipeline retains grounded facts and supplements them with retrieved information. The paper evaluates factuality separately from conversational quality and reports results for simulated and human conversations. Its scope is Wikipedia-grounded dialogue; those findings do not establish reliability for an open-web research publisher or an agent taking actions. [1]
STORM: investigate before drafting
STORM treats pre-writing as a distinct research stage. It discovers perspectives, uses simulated question-and-answer conversations grounded in web sources, and organizes the collected information into an outline before article writing. The paper introduces FreshWiki and outline assessments to study that stage. The architectural lesson we take is to evaluate discovery and coverage before judging the final prose. This interpretation is narrower than claiming an automatically generated article is correct. [2]
A claim record for research notes
For this Lab, we propose storing the following fields for each material claim. The schema is our design, not a claimed implementation of either paper.
| Field | Purpose |
|---|---|
| Claim text and ID | The precise statement the article asks a reader to believe. |
| Source and version | An exact URL, publication date, relevant section, and retrieval date. |
| Evidence relationship | Supports, qualifies, contradicts, or provides background. |
| Statement type | Reported finding, our inference, or proposed experiment. |
| Scope and uncertainty | The conditions under which the statement applies and what is unknown. |
| Review decision | Accept, revise, omit, or hold, with a reason. |
A source can be authoritative and still fail to support a particular sentence. Review therefore asks two different questions: is this source appropriate, and does the cited passage entail the claim as written? A paper about dialogue factuality cannot by itself support a claim about the safety of tool execution.
A small, falsifiable test
Compare a citation-only draft with a claim-ledger draft on the same source packs. The intended outcome is faster detection of unsupported conclusions without losing useful coverage.
- Prepare a fixed task set. Choose ten narrowly scoped technical questions. Preserve the sources available on a fixed date. Include contradictory results, a revised paper, a missing answer, and a source that looks relevant but does not support the key claim.
- Keep conditions comparable. Use the same model version, source budget, and generation settings. One condition produces a cited draft directly. The other produces evidence records before the draft. Repeat runs and retain every output.
- Review independently. Give reviewers a common rubric. Ask them to mark unsupported, overstated, and contradicted claims, then measure the time required. Resolve disagreements separately from the first-pass scores.
- Evaluate utility alongside caution. Check whether each draft answers the original question and preserves important disagreements. A system that removes every substantive claim is not a useful research assistant.
- Report the full trade-off. Include review time, claim-level error counts, omitted relevant findings, generation cost, and variance across runs. Keep abstentions visible rather than silently removing them from the denominator.
What this would not establish
Ten questions would provide an initial diagnostic, not broad evidence of superiority. Source packs can contain shared errors; two sources may repeat the same underlying claim. Human review can also miss mistakes. More structured records may increase time and cost before they save reviewer effort.
We would need additional evaluation across domains, languages, source freshness, and reviewer expertise. The proposed test concerns research quality and reviewability. It does not authorize autonomous publication, redistribution of copyrighted material, or the use of private data.
Implication for Solo Unicorn Lab
The first useful automation should create a reviewable source pack and draft. Publication should remain a separate decision. If the evidence record cannot explain a conclusion, the system should revise it, hold it, or state the uncertainty.
That creates a practical research loop: improve the question, gather evidence, inspect the claims, publish an approved version, and retain corrections. The current Site is a static research edition; this workflow has not been connected to a scheduler.
Primary references
- Semnani et al. (2023). WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia. Findings of EMNLP 2023. Used for the factual-grounding and evaluation discussion.
- Shao et al. (2024). Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models. NAACL 2024. Used for the pre-writing research and outline-evaluation discussion.