Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

2021-02-02 · Federico A. Galatolo, Mario G. C. A. Cimino, Gigliola Vaglini

In this research work we present CLIP-GLaSS, a novel zero-shot framework to generate an image (or a caption) corresponding to a given caption (or image). CLIP-GLaSS is based on the CLIP neural network, which, given an image and a descriptive caption, provides similar embeddings. Differently, CLIP-GLaSS takes a caption (or an image) as an input, and generates the image (or the caption) whose CLIP embedding is the most similar to the input one. This optimal image (or caption) is produced via a generative network, after an exploration by a genetic algorithm. Promising results are shown, based on the experimentation of the image Generators BigGAN and StyleGAN2, and of the text Generator GPT2

📄 PDF Abstract BibTeX arXiv:2102.01645

Code (3)

galatolofederico/clip-glass 공식 구현 pytorch

armaank/archlectures pytorch

yk/clip_music_video pytorch

Tasks

DescriptiveImage Generation

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Adam 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Six Ways To Communicate To Someone At Expedia Via Phone And Email's. To communicate or get human at Expedia, the quickest option is typically to call their customer service at ＋１－８８８－８２９－０８８１　ｏｒ　＋１（８０５）　３３０　（４０５６）. You can also use the live chat…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Feedforward Network A Feedforward Network, or a Multilayer Perceptron (MLP), is a neural network with solely densely connected layers. This is the classic neural network architecture of the…

Off-Diagonal Orthogonal Regularization Off-Diagonal Orthogonal Regularization is a modified form of orthogonal regularization originally used in…

Towards Generating Diverse Audio Captions via Adversarial Training

2022-12-05 · Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D. Plumbley 외

Automated audio captioning is a cross-modal translation task for describing the content of audio clips with natural language sentences. This task has attracted increasing attention and substantial progress has been made …

Audio captioning Diversity Generative Adversarial Network

Diverse Audio Captioning via Adversarial Training

2021-10-13 · Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D. Plumbley 외

Audio captioning aims at generating natural language descriptions for audio clips automatically. Existing audio captioning models have shown promising improvement in recent years. However, these models are mostly trained…

Audio captioning Diversity Generative Adversarial Network Sentence

Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videos

2024-07-30 · Dhruv Verma, Debaditya Roy, Basura Fernando

Situation recognition refers to the ability of an agent to identify and understand various situations or contexts based on available information and sensory inputs. It involves the cognitive process of interpreting data …

Semantic Role Labeling Video Captioning

Pragmatic Inference with a CLIP Listener for Contrastive Captioning

2023-06-15 · Jiefu Ou, Benno Krojer, Daniel Fried

We propose a simple yet effective and robust method for contrastive captioning: generating discriminative captions that distinguish target images from very similar alternative distractor images. Our approach is built on …

CLIP4IDC: CLIP for Image Difference Captioning

2022-06-01 · Zixin Guo, Tzu-Jui Julius Wang, Jorma Laaksonen

Image Difference Captioning (IDC) aims at generating sentences to describe differences between two similar-looking images. Conventional approaches learn an IDC model with a pre-trained and usually frozen visual feature e…

Domain Adaptation Image Classification