Explaining CLIP through Co-Creative Drawings and Interaction
This paper analyses a visual archive of drawings produced by an interactive robotic art installation where audience members narrated their dreams into a system powered by CLIPdraw deep learning (DL) model that interpreted and transformed their dreams into images. The resulting archive of prompt-image pairs were examined and clustered based on concept representation accuracy. As a result of the analysis, the paper proposes four groupings for describing and explaining CLIP-generated results: clear concept, text-to-text as image, indeterminacy and confusion, and lost in translation. This article offers a glimpse into a collection of dreams interpreted, mediated and given form by Artificial Intelligence (AI), showcasing oftentimes unexpected, visually compelling or, indeed, the dream-like output of the system, with the emphasis on processes and results of translations between languages, sign-systems and various modules of the installation. In the end, the paper argues that proposed clusters support better understanding of the neural model.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation
Generating images that fit a given text description using machine learning has improved greatly with the release of technologies such as the CLIP image-text encoder model; however, current methods lack artistic control o…
Style TransferTranslationPencils to Pixels: A Systematic Study of Creative Drawings across Children, Adults and AI
Can we derive computational metrics to quantify visual creativity in drawings across intelligent agents, while accounting for inherent differences in technical skill and style? To answer this, we curate a novel dataset c…
DiversityCLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
This work presents CLIPDraw, an algorithm that synthesizes novel drawings based on natural language input. CLIPDraw does not require any training; rather a pre-trained CLIP language-image encoder is used as a metric for …
DeepMorph: A System for Hiding Bitstrings in Morphable Vector Drawings
We introduce DeepMorph, an information embedding technique for vector drawings. Provided a vector drawing, such as a Scalable Vector Graphics (SVG) file, our method embeds bitstrings in the image by perturbing the drawin…
Camera Pose EstimationDecoderobject-detectionObject Detection+2I wanna draw like you: Inter-and intra-individual differences in orang-utan drawings
Recently discovered, the oldest human abstract drawing is around 73,000 years. Although the origins of drawing behaviour have remained an enigma to this day, light may be shone on the subject through its study among our …