COLLIE: Systematic Construction of Constrained Text Generation Tasks

2023-07-17 · Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, Karthik Narasimhan

Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. However, existing benchmarks for constrained generation usually focus on fixed constraint types (e.g.,generate a sentence containing certain words) that have proved to be easy for state-of-the-art models like GPT-4. We present COLLIE, a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g.,language understanding, logical reasoning, counting, semantic planning). We also develop tools for automatic extraction of task instances given a constraint structure and a raw text corpus. Using COLLIE, we compile the COLLIE-v1 dataset with 2080 instances comprising 13 constraint structures. We perform systematic experiments across five state-of-the-art instruction-tuned language models and analyze their performances to reveal shortcomings. COLLIE is designed to be extensible and lightweight, and we hope the community finds it useful to develop more complex constraints and evaluations in the future.

📄 PDF Abstract BibTeX arXiv:2307.08689

Code (1)

princeton-nlp/Collie 공식 구현

Tasks

Logical ReasoningSentenceText Generation

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Position-Wise Feed-Forward Layer 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Residual Connection 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

CoLLiE: Collaborative Training of Large Language Models in an Efficient Way

2023-12-01 · Kai Lv, Shuo Zhang, Tianle Gu, Shuhao Xing 외

Large language models (LLMs) are increasingly pivotal in a wide range of natural language processing tasks. Access to pre-trained models, courtesy of the open-source community, has made it possible to adapt these models …

GPU parameter-efficient fine-tuning

COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space

2026-05-31 · Yao Luan, Ni Mu, Hanfei Ge, Yiqin Yang 외 arxiv

Unsupervised skill discovery (USD) aims to learn diverse behaviors without reward functions, but often results in task-irrelevant or hazardous behaviors due to uniform exploration. Guided skill discovery (GSD) addresses …

Scalable Annotation of Fine-Grained Categories Without Experts

2017-09-07 · Timnit Gebru, Jonathan Krause, Jia Deng, Li Fei-Fei

We present a crowdsourcing workflow to collect image annotations for visually similar synthetic categories without requiring experts. In animals, there is a direct link between taxonomy and visual similarity: e.g. a coll…

CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings

2021-11-15 · Gabriel Skantze, Bram Willemsen

This paper presents CoLLIE: a simple, yet effective model for continual learning of how language is grounded in vision. Given a pre-trained multimodal embedding model, where language and images are projected in the same …

Continual Learning Few-Shot Learning

A Broad-Coverage Deep Semantic Lexicon for Verbs

2020-07-06 · LREC 2020 5 · James Allen, Hannah An, Ritwik Bose, Will de Beaumont 외

Progress on deep language understanding is inhibited by the lack of a broad coverage lexicon that connects linguistic behavior to ontological concepts and axioms. We have developed COLLIE-V, a deep lexical resource for v…