ChronoSeek
Temporal-First Agentic RAG with Event Knowledge Graph for Lifelog Retrieval
A temporal-first agentic retrieval system transforming 725,000+ lifelog images into a Neo4j Event Knowledge Graph and Milvus vector store, orchestrated by a time-bound LangGraph agent. Achieved a perfect 1000.0/1000.0 score (Rank #1) on Task 3 (Question Answering) at ACM ICMR LSC 2026.
The Technical Bottleneck
Retrieving personal moments from continuous multimodal lifelog archives (hundreds of thousands of egocentric frames) breaks flat embedding retrievers (CLIP/SigLIP). Visually similar frames occur across different days and activities, while queries require temporal sequencing ('before', 'after', 'duration') and factual aggregation under a strict 4-minute live competition clock.
Why Existing Baselines Fail
Prior LSC systems added LLM re-ranking on top of frame-level similarity search, leaving their core index non-temporal. Conversely, prior graph-based lifelog systems suffered from prohibitive traversal latency. We hypothesized that indexing discrete human activity events in a property graph—paired with a time-bounded agentic critic—could simultaneously achieve high temporal precision and sub-10ms graph query latency.
Core Formulation & Insight
ChronoSeek introduces a dual-index architecture: an offline VLM pipeline segments image streams into ~3-minute activity events stored as nodes in Neo4j (connected by NEXT, SAME_PLACE, and SIMILAR_ACTIVITY edges) while SigLIP embeddings populate Milvus (IVF_PQ). Online, a LangGraph Agentic Orchestrator classifies query intent, performs Chain-of-Thought sub-query decomposition, executes parallel Cypher + ANN retrieval, fuses scores with temporal priority (α = 0.7), and uses an Agentic Critic to trigger a human-in-the-loop Fast-Path grid whenever confidence drops (<0.6) or elapsed time exceeds 70% of the budget.
Architectural Design
Decouples heavy multimodal event segmentation into an offline indexing pipeline so online agentic retrieval operates over compact event nodes and quantized vector indices within strict competition latency bounds.
ChronoSeek system architecture: Offline Indexing (CLIP boundary detection, Qwen2.5-VL event labeling, Neo4j EKG, Milvus) and Online Agentic Retrieval.
1. Two-Stage Event Segmentation (CLIP + Qwen2.5-VL-7B)
Detects candidate event boundaries when consecutive-frame CLIP cosine distance exceeds τ = 0.3, then samples K = 5 keyframes per segment for Qwen2.5-VL-7B to assign a 32-class compact activity taxonomy label, refined timestamps, and event caption.
2. Dual Indexing: Neo4j EKG + Milvus IVF_PQ
Constructs Event Nodes in Neo4j linked by Temporal (NEXT), Spatial (SAME_PLACE ≤50m), and Semantic (SIMILAR_ACTIVITY) edges alongside frame-level SigLIP-base embeddings in Milvus.
3. LangGraph Agentic Orchestrator & CoT Decomposition
Classifies incoming queries into Ad-hoc, Known-Item Search (KIS), or Question Answering (QA) and decomposes multi-step temporal constraints into parallel Cypher and Milvus predicates.
4. Cross-Modal Fusion & Agentic Critic Fast-Path
Computes S_final = 0.7·S_temporal + 0.3·S_semantic. An Agentic Critic monitors top-5 confidence (θ_conf = 0.6) and wall-clock time (≤170s), routing either to RAG answer synthesis or an instant Fast-Path interactive human-in-the-loop grid.
Technical Stack & Systems Engineering
Implementation Highlights
- →Processed 725,000+ lifelog images into structured temporal events; benchmarked 118K images / 33,471 event nodes with sub-6 ms Neo4j Cypher latency.
- →Implemented deterministic activity taxonomy consolidation (mapping synonymous VLM predictions into a reliable 32-class ontology).
- →Built a real-time competition web interface with DRES session integration, color-coded event timelines, context frame inspection, and 1-click submission.
Datasets & Protocol
Datasets & Corpora
- Official ACM LSC Dataset (18 months, 725,949 images)
- Controlled 5-Day Ablation Split (7,063 images, 19 temporal queries, 2 semantic queries)
- 3-Month Scalability Split (Jan–Mar 2019, 118,000 images, 33,471 events, 33 queries)
Evaluation Protocol
Evaluated static visual baselines (SigLIP, CLIP), Temporal-Only filtering, Hybrid Reranked, and Event-Aware retrieval under both ground-truth and zero-shot Qwen2.5-VL labels, plus window-size (1h vs. 2h vs. 4h) and synthetic label-noise (20%, 50%) stress tests.
Tracked Evaluation Metrics
Ranked #1 globally on Task 3 (Question Answering) at ACM ICMR LSC '26 with a perfect 1000.0/1000.0 score, and achieved a 10.8× MRR improvement over CLIP in controlled zero-shot ablations.
Rank #1 out of 15 international teams in live evaluation
10.8× over static CLIP (0.056) and SigLIP (0.00)
100% non-null QA coverage; mean 4.2s / p95 6.7s latency
Evaluated over 33,471 events (118K images)
Empirical Analysis & Interpretation
Ablation studies prove that event identification—not frame-level visual re-ranking—is the decisive bottleneck in temporal lifelog retrieval. Even under 50% synthetic activity label noise, Event-Aware retrieval maintains 0.842 MRR, outperforming all static embedding baselines by an order of magnitude.
ChronoSeek interactive competition UI with Fast-Path candidate grid, event timeline, and DRES submission integration.
What This System Contributes
- 01Architected and implemented the end-to-end ChronoSeek offline indexing and online LangGraph agentic retrieval pipelines.
- 02Designed the Neo4j Event Knowledge Graph schema and deterministic 32-class VLM activity mapping.
- 03Conducted all ablation, noise-robustness, and scalability benchmarks and authored the ACM ICMR LSC '26 paper.
Boundary Conditions & Future Work
- ·Zero-shot VLM event boundaries degrade when user queries specify loose 4-hour windows (MRR drops to 0.467).
- ·Source repository is currently private per competition/lab hosting policy; full methodology and reproducibility details are published in ACM LSC '26.
ChronoSeek: Temporal-First Agentic RAG with Event Knowledge Graph for Lifelog Retrieval
Van-Truong Le, Minh-Triet Tran, Viet-Tham Huynh