VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling

2021-12-10 · Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani, Alexey Gritsenko

User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language. The tasks are also diverse, including object detection, language generation and grounding. In this paper, we present VUT, a Versatile UI Transformer that takes multimodal input and simultaneously accomplishes 5 distinct tasks with the same model. Our model consists of a multimodal Transformer encoder that jointly encodes UI images and structures, and performs UI object detection when the UI structures are absent in the input. Our model also consists of an auto-regressive Transformer model that encodes the language input and decodes output, for both question-answering and command grounding with respect to the UI. Our experiments show that for most of the tasks, when trained jointly for multi-tasks, VUT substantially reduces the number of models and footprints needed for performing multiple tasks, while achieving accuracy exceeding or on par with baseline models trained for each individual task.

📄 PDF Abstract BibTeX arXiv:2112.05692

Code (0)

등록된 구현이 없습니다.

Tasks

object-detection Object Detection Question Answering Text Generation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Multi-Head Attention 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Adam 설명 없음

VUT: Versatile UI Transformer for Multimodal Multi-Task User Interface Modeling

2021-09-29 · Yang Li, Gang Li, Xin Zhou, Mostafa Dehghani 외

object-detection Object Detection Question Answering Text Generation

4M: Massively Multimodal Masked Modeling

2023-12-11 · NeurIPS 2023 11 · David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo 외

Current machine learning models for vision are often highly specialized and limited to a single modality and task. In contrast, recent large language models exhibit a wide range of capabilities, hinting at a possibility …

Decoder

MolMix: A Simple Yet Effective Baseline for Multimodal Molecular Representation Learning

2024-10-10 · Andrei Manolache, Dragos Tantaru, Mathias Niepert

In this work, we propose a simple transformer-based baseline for multimodal molecular representation learning, integrating three distinct modalities: SMILES strings, 2D graph representations, and 3D conformers of molecul…

molecular representation Representation Learning

OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation

2025-10-22 · Guowei Xu, Yuxuan Bian, Ailing Zeng, Mingyi Shi 외 arxiv

This paper introduces OmniMotion-X, a versatile multimodal framework for whole-body human motion generation, leveraging an autoregressive diffusion transformer in a unified sequence-to-sequence manner. OmniMotion-X effic…

Enhancing CT Image synthesis from multi-modal MRI data based on a multi-task neural network framework

2023-12-13 · Zhuoyao Xin, Christopher Wu, Dong Liu, Chunming Gu 외

Image segmentation, real-value prediction, and cross-modal translation are critical challenges in medical imaging. In this study, we propose a versatile multi-task neural network framework, based on an enhanced Transform…

Image Generation Image Segmentation Segmentation Semantic Segmentation+1