paper-with-me

Papers

Generative Pretrained Embedding and Hierarchical Irregular Time Series Representation for Daily Living Activity Recognition

2024-12-27 · Damien Bouchabou, Sao Mai Nguyen

Within the evolving landscape of smart homes, the precise recognition of daily living activities using ambient sensor data stands paramount. This paper not only aims to bolster existing algorithms by evaluating two distinct pretrained embeddings suited for ambient sensor activations but also introduces a novel hierarchical architecture. We delve into an architecture anchored on Transformer Decoder-based pre-trained embeddings, reminiscent of the GPT design, and contrast it with the previously established state-of-the-art (SOTA) ELMo embeddings for ambient sensors. Our proposed hierarchical structure leverages the strengths of each pre-trained embedding, enabling the discernment of activity dependencies and sequence order, thereby enhancing classification precision. To further refine recognition, we incorporate into our proposed architecture an hour-of-the-day embedding. Empirical evaluations underscore the preeminence of the Transformer Decoder embedding in classification endeavors. Additionally, our innovative hierarchical design significantly bolsters the efficacy of both pre-trained embeddings, notably in capturing inter-activity nuances. The integration of temporal aspects subtly but distinctively augments classification, especially for time-sensitive activities. In conclusion, our GPT-inspired hierarchical approach, infused with temporal insights, outshines the SOTA ELMo benchmark.

📄 PDF Abstract BibTeX arXiv:2412.19732

Code (1)

dbouchabou/Generative-Pretrained-Embedding-and-Hierarchical-Representation-to-Unlock-ADL-Rhythm-in-Smart-Homes 공식 구현 tf

Tasks

Activity RecognitionDecoderIrregular Time SeriesTime Series

Methods 이 논문이 사용한 방법론

Tanh Activation 설명 없음
Sigmoid Activation 설명 없음
Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…
Weight Decay 설명 없음
LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…
Linear Warmup With Cosine Annealing Linear Warmup With Cosine Annealing is a learning rate schedule where we increase the learning rate linearly for $n$ updates and then anneal according to a cosine schedule…
Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…
Discriminative Fine-Tuning Discriminative Fine-Tuning is a fine-tuning strategy that is used for ULMFiT type models. Instead of using the same learning rate…

Similar Papers 제목 키워드 기반

Integrating Textual Embeddings from Contrastive Learning with Generative Recommender for Enhanced Personalization

2025-04-13 · Yijun Liu

Recent advances in recommender systems have highlighted the complementary strengths of generative modeling and pretrained language models. We propose a hybrid framework that augments the Hierarchical Sequential Transduct…

Contrastive LearningRecommendation Systems

Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models

2015-07-17 · Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville 외

We investigate the task of building open domain, conversational dialogue systems based on large dialogue corpora using generative models. Generative models produce system responses that are autonomously generated word-by…

DecoderWord Embeddings

Two-Stage Aggregation with Dynamic Local Attention for Irregular Time Series

2023-11-13 · Xingyu Chen, Xiaochen Zheng, Amina Mollaysa, Manuel Schürch 외

Irregular multivariate time series data is characterized by varying time intervals between consecutive observations of measured variables/signals (i.e., features) and varying sampling rates (i.e., recordings/measurement)…

Irregular Time SeriesTime Series

Text Compression for Efficient Language Generation

2025-03-14 · David Gu, Peter Belcak, Roger Wattenhofer

We challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation. To this end, we propose the "Generative Pretrained Thoughtformer" (GPTHF), a hierarchical transformer …

Language ModelingLanguage ModellingSentenceSentence Embeddings+2

Transformer Embeddings of Irregularly Spaced Events and Their Participants

2021-12-31 · ICLR 2022 4 · Chenghao Yang, Hongyuan Mei, Jason Eisner

The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in w…