SweCTRL-Mini: a data-transparent Transformer-based large language model for controllable text generation in Swedish
We present SweCTRL-Mini, a large Swedish language model that can be used for inference and fine-tuning on a single consumer-grade GPU. The model is based on the CTRL architecture by Keskar, McCann, Varshney, Xiong, and Socher (2019), which means that users of the SweCTRL-Mini model can control the genre of the generated text by inserting special tokens in the generation prompts. SweCTRL-Mini is trained on a subset of the Swedish part of the mC4 corpus and a set of Swedish novels. In this article, we provide (1) a detailed account of the utilized training data and text pre-processing steps, to the extent that it is possible to check whether a specific phrase/source was a part of the training data, and (2) an evaluation of the model on both discriminative tasks, using automatic evaluation methods, and generative tasks, using human referees. We also compare the generative capabilities of the model with those of GPT-3. SweCTRL-Mini is fully open and available for download.
Code (1)
Tasks
GPULanguage ModelingLanguage ModellingLarge Language ModelText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Segmenting Transparent Object in the Wild with Transformer
This work presents a new fine-grained transparent object segmentation dataset, termed Trans10K-v2, extending Trans10K-v1, the first large-scale transparent object segmentation dataset. Unlike Trans10K-v1 that only has tw…
DecoderObjectSegmentationSemantic Segmentation+1TransMatting: Tri-token Equipped Transformer Model for Image Matting
Image matting aims to predict alpha values of elaborate uncertainty areas of natural images, like hairs, smoke, and spider web. However, existing methods perform poorly when faced with highly transparent foreground objec…
Image MattingTransparent objectsTODE-Trans: Transparent Object Depth Estimation with Transformer
Transparent objects are widely used in industrial automation and daily life. However, robust visual recognition and perception of transparent objects have always been a major challenge. Currently, most commercial-grade d…
Depth EstimationObjectTransparent Object Depth EstimationTransparent objectsMechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI
Mechanistic interpretability improves the safety, reliability, and robustness of large AI models. This study examined individual attention heads in vision transformers (ViTs) fine tuned on distorted 2D spectrogram images…
DiffDecompose: Layer-Wise Decomposition of Alpha-Composited Images via Diffusion Transformers
Diffusion models have recently motivated great success in many generation tasks like object removal. Nevertheless, existing image decomposition methods struggle to disentangle semi-transparent or transparent layer occlus…
Flare Removal