Training-Free Fact Verification Framework
Provenance-Aware Prediction Caching, Leakage Audits & Retrieval Baselines
A modular research software framework for improving fact verification without model parameter updates, featuring immutable provenance-aware prediction caching, data-leakage auditing, fixed-ontology error profiling, and 8 baseline (B0–B8) / 4 retrieval (R1–R4) protocols covered by automated pytest suites.
The Technical Bottleneck
Evaluating training-free LLM fact verification methods (such as few-shot retrieval, self-reflection, and historical prediction memory) is frequently undermined by silent cache contamination, partition leakage, and inconsistent label ontologies.
Why Existing Baselines Fail
Before running expensive large-scale model evaluations on benchmarks like AVeriTeC, a strictly auditable, deterministic software harness is required to guarantee zero data leakage between historical memory banks and test claims.
Core Formulation & Insight
We engineered a provenance-first Python framework implementing immutable source-cache validation, grouped partition splitters, fixed-ontology Macro-F1/Accuracy evaluators, granular error profilers, and standardized baseline (B0–B8) and retrieval (R1–R4) interfaces verified by synthetic unit tests.
Architectural Design
Deterministic evaluation and caching architecture for training-free fact verification experiments.
1. Grouped Partitioning & Leakage Auditor
Ensures claim clusters and evidence sources never leak across memory-bank and evaluation splits.
2. Immutable Provenance-Aware Prediction Cache
Hashes prompt templates, model identifiers, and evidence states to guarantee reproducible offline replay.
3. Standardized Baselines (B0–B8) & Retrieval (R1–R4)
Provides unified interfaces for zero-shot, few-shot, and history-augmented verification workflows.
4. Fixed-Ontology Metrics & Error Profiler
Computes strict ontology-mapped Accuracy/Macro-F1 and slices failure modes by claim complexity.
Technical Stack & Systems Engineering
Implementation Highlights
- →Full public codebase with source-cache validation, grouped partitioning, and automated `pytest` verification.
Datasets & Protocol
Datasets & Corpora
- Synthetic Verification Test Suite (AVeriTeC model-backed server runs pending)
Evaluation Protocol
Unit and integration testing of all B0–B8 baselines, R1–R4 retrievers, and leakage auditors.
Tracked Evaluation Metrics
Complete, test-verified research framework (B0–B8 baselines, R1–R4 retrieval, immutable cache); large-scale AVeriTeC model-backed server runs are scheduled next.
9 standardized training-free verification configurations
4 evidence and prediction-history retrieval strategies
Full pytest suite passing; GPU AVeriTeC runs pending
Empirical Analysis & Interpretation
In accordance with strict research integrity, we report the verified software architecture and test suite without claiming unexecuted AVeriTeC model numbers.
What This System Contributes
- 01Designed and implemented the entire public repository, caching architecture, and pytest suite.
Boundary Conditions & Future Work
- ·Full AVeriTeC LLM-backed benchmark runs are pending server cache population.