Multidimensional Evaluation for Text Style Transfer Using ChatGPT
We investigate the potential of ChatGPT as a multidimensional evaluator for the task of \emph{Text Style Transfer}, alongside, and in comparison to, existing automatic metrics as well as human judgements. We focus on a zero-shot setting, i.e. prompting ChatGPT with specific task instructions, and test its performance on three commonly-used dimensions of text style transfer evaluation: style strength, content preservation, and fluency. We perform a comprehensive correlation analysis for two transfer directions (and overall) at different levels. Compared to existing automatic metrics, ChatGPT achieves competitive correlations with human judgments. These preliminary results are expected to provide a first glimpse into the role of large language models in the multidimensional evaluation of stylized text generation.
Code (1)
Tasks
Style TransferText GenerationText Style TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training
In this work, we introduce the concept of complex text style transfer tasks, and constructed complex text datasets based on two widely applicable scenarios. Our dataset is the first large-scale data set of its kind, with…
AttributeContrastive LearningStyle TransferText Generation+1Evaluating Text Style Transfer Evaluation: Are There Any Reliable Metrics?
Text Style Transfer (TST) is the task of transforming a text to reflect a particular style while preserving its original content. Evaluating TST outputs is a multidimensional challenge, requiring the assessment of style …
Machine TranslationStyle TransferText Style TransferLMStyle Benchmark: Evaluating Text Style Transfer for Chatbots
Since the breakthrough of ChatGPT, large language models (LLMs) have garnered significant attention in the research community. With the development of LLMs, the question of text style transfer for conversational models h…
Style TransferText Style TransferSETTP: Style Extraction and Tunable Inference via Dual-level Transferable Prompt Learning
Text style transfer, an important research direction in natural language processing, aims to adapt the text to various preferences but often faces challenges with limited resources. In this work, we introduce a novel met…
Prompt LearningStyle TransferText Style TransferChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer
Large-scale language models, like ChatGPT, have garnered significant media attention and stunned the public with their remarkable capacity for generating coherent text from short natural language prompts. In this paper, …
SentenceStyle TransferText Summarization