ChronoSeek: Temporal-First Agentic RAG with Event Knowledge Graph for Lifelog Retrieval
1University of Science, Vietnam National University Ho Chi Minh City (VNU-HCM) · SELab
The 9th Annual ACM Workshop on the Lifelog Search Challenge (LSC '26), co-located with ACM ICMR 2026, Amsterdam, Netherlands
Figure 1: Figure 1: High-level architecture of ChronoSeek combining offline Event Knowledge Graph + Milvus indexing with online LangGraph Agentic Orchestration and Fast-Path fallback.
Highest score among all 15 participating teams globally
10.8× improvement over static CLIP (0.056) and SigLIP (0.00)
Validated on both 7K-image dev set and 118K-image 3-month subset
Across 33,471 event nodes in Neo4j; full pipeline p95 ≈ 6.7 s
Paper Abstract
Retrieving personal moments from lifelong multimodal data remains challenging, especially when queries require temporal reasoning and multi-step contextual understanding. Existing lifelog search systems largely rely on static embedding-based retrieval, which fails to capture the narrative structure of daily activities. We propose ChronoSeek, a temporal-first agentic retrieval system that organizes lifelog data as an Event Knowledge Graph and uses a time-bound Agentic Orchestrator with a human-in-the-loop fallback. ChronoSeek constructs fine-grained events with Qwen2.5-VL, links them through temporal, spatial, and semantic edges in Neo4j, and performs hybrid retrieval over both the EKG and Milvus. With ground-truth activity labels, event-aware filtering provides an oracle upper bound of MRR = 1.00 across 19 temporal queries, while static SigLIP/CLIP baselines yield near-zero MRR. With Qwen2.5-VL zero-shot predictions and a cleaned activity taxonomy, MRR reaches 0.607—a 10.8× improvement over CLIP. Scalability tests on 118K images and 33,471 events show sub-6 ms Neo4j query latency, and ChronoSeek achieved a perfect score on LSC'26 Task 3 (Question Answering), which strongly emphasizes temporal reasoning.
Research Motivation
Top-performing lifelog retrieval systems rely on flat, frame-level CLIP/SigLIP cosine similarity. They struggle to distinguish visually similar but contextually distinct moments ('cooking' vs. 'cleaning the kitchen') and fail on multi-step temporal narratives ('the moment I left the restaurant after dinner') under live competition time limits.
Guiding Research Questions
- RQ1Can structuring continuous egocentric image streams into an Event Knowledge Graph (EKG) bridge the gap between static visual similarity and multi-step temporal reasoning?
- RQ2How can an agentic RAG pipeline balance Chain-of-Thought query decomposition and dual-channel retrieval (Cypher + ANN) within a strict 240-second live competition budget?
Methodology & Architecture
Offline indexing segments raw lifelog streams via a two-stage CLIP cosine-distance change detector (τ = 0.3) and Qwen2.5-VL-7B keyframe refinement (K = 5 keyframes) mapped to a compact 32-class activity taxonomy. Events are stored as nodes in Neo4j linked by NEXT (temporal), SAME_PLACE (spatial, ≤50m), and SIMILAR_ACTIVITY (semantic) edges, alongside SigLIP-base embeddings in Milvus (IVF_PQ). Online retrieval uses a LangGraph Agentic Orchestrator that classifies queries (Ad-hoc, KIS, QA), decomposes complex temporal constraints into parallel Cypher and Milvus sub-queries, fuses scores (α = 0.7 temporal weight), and employs an Agentic Critic (θ_conf = 0.6, 0.7 × T_max budget) with a Fast-Path human-in-the-loop fallback grid.
Two-phase architecture: (1) Offline Indexing (CLIP boundary proposal → Qwen2.5-VL event labeling → Neo4j Event Knowledge Graph + Milvus SigLIP IVF_PQ store) and (2) Online Agentic Orchestrator (LangGraph CoT decomposition → Parallel Cypher + Milvus retrieval → Cross-modal fusion S_final = 0.7 S_temporal + 0.3 S_semantic → Agentic Critic → RAG QA or Fast-Path UI fallback).
- · Official ACM LSC Dataset (18 months, 725,949 egocentric images)
- · Controlled 5-day Development Subset (7,063 images, 19 temporal + 2 semantic queries)
- · 3-Month Scalability Subset (Jan–Mar 2019, 118,000 images, 33,471 events, 33 temporal queries)
Mean Reciprocal Rank (MRR) · Recall@K · Keyword Hit@10 · Neo4j Cypher Latency (ms) · Official LSC'26 Normalized Task Scores
Additional Figures from the Manuscript
Figure 2: Figure 2: The Fast-Path human-in-the-loop fallback interface with color-coded event timeline, evidence inspector, and live competition timer.
Primary Contributions
- 01Designed a temporal-first Event Knowledge Graph (EKG) schema in Neo4j encoding NEXT, SAME_PLACE, and SIMILAR_ACTIVITY edges over segmented lifelog events.
- 02Built a time-bound LangGraph Agentic Orchestrator with parallel Cypher + Milvus retrieval and an Agentic Critic Fast-Path fallback.
- 03Demonstrated a 10.8× MRR gain over static CLIP baselines in controlled ablations and achieved a perfect 1000.0 score on Task 3 (Question Answering) at ACM ICMR LSC '26.
Analyzed Boundary Conditions
- ·Zero-shot VLM activity predictions remain sensitive to overly broad time windows (MRR drops from 1.00 at 1h to 0.467 at 4h).
- ·Live competition score on Task 2 (Known-Item Search: 54.5) was impacted by an operational submission-interface issue despite plausible backend candidates.
BibTeX Entry
@inproceedings{le2026chronoseek,
author = {Le, Van-Truong and Tran, Minh-Triet and Huynh, Viet-Tham},
title = {ChronoSeek: Temporal-First Agentic RAG with Event Knowledge Graph for Lifelog Retrieval},
booktitle = {Proceedings of the 9th Annual ACM Workshop on the Lifelog Search Challenge (LSC '26)},
year = {2026},
pages = {1--6},
publisher = {ACM},
address = {New York, NY, USA},
doi = {10.1145/3810984.3817067}
}