Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay
Computer-use agents pay full frontier inference to re-derive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline: it segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte-identical, cacheable, and mechanically auditable. On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms, and an agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture, a mid-tier model reading the block matching a frontier one. The same compiler doubles as a demand-side cost instrument. Read off passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters that agent-cost models assume but, to our knowledge, have not measured: the Routine Overhead Ratio R and the routine recurrence h. We report first values of R, a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, for a realistic all-fleet token ceiling near 8%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit. Schema, compiler, and evaluation harness are open.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Screening for Sparse Online Learning
Sparsity promoting regularizers are widely used to impose low-complexity structure (e.g. l1-norm for sparsity) to the regression coefficients of supervised learning. In the realm of deterministic optimization, the sequen…
Cross modal video representations for weakly supervised active speaker localization
An objective understanding of media depictions, such as inclusive portrayals of how much someone is heard and seen on screen such as in film and television, requires the machines to discern automatically who, when, how, …
Action DetectionActive Speaker LocalizationActivity DetectionEvent DetectionA Multi-Stream Convolutional Neural Network Framework for Group Activity Recognition
In this work, we present a framework based on multi-stream convolutional neural networks (CNNs) for group activity recognition. Streams of CNNs are separately trained on different modalities and their predictions are fus…
Activity RecognitionGroup Activity RecognitionPose EstimationBroad cross-reactivity of the T-cell repertoire achieves specific and sufficiently rapid target searching
The molecular recognition of T-cell receptors is the hallmark of the adaptive immunity. Given the finiteness of the T-cell repertoire, individual T-cell receptors are necessary to be cross-reactive to multiple antigenic …
Fullie and Wiselie: A Dual-Stream Recurrent Convolutional Attention Model for Activity Recognition
Multimodal features play a key role in wearable sensor based Human Activity Recognition (HAR). Selecting the most salient features adaptively is a promising way to maximize the effectiveness of multimodal sensor data. In…
Activity RecognitionHuman Activity Recognitionvalid