An Adaptive Deep Clustering Pipeline to Inform Text Labeling at Scale
Mining the latent intentions from large volumes of natural language inputs is a key step to help data analysts design and refine Intelligent Virtual Assistants (IVAs) for customer service and sales support. We created a flexible and scalable clustering pipeline within the Verint Intent Manager (VIM) that integrates the fine-tuning of language models, a high performing k-NN library and community detection techniques to help analysts quickly surface and organize relevant user intentions from conversational texts. The fine-tuning step is necessary because pre-trained language models cannot encode texts to efficiently surface particular clustering structures when the target texts are from an unseen domain or the clustering task is not topic detection. We describe the pipeline and demonstrate its performance and ability to scale on three real-world text mining tasks. As deployed in the VIM application, this clustering pipeline produces high quality results, improving the performance of data analysts and reducing the time it takes to surface intentions from customer service data, thereby reducing the time it takes to build and deploy IVAs in new domains.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringCommunity DetectionDeep ClusteringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Automatic Summarization of Online Debates
Debate summarization is one of the novel and challenging research areas in automatic text summarization which has been largely unexplored. In this paper, we develop a debate summarization pipeline to summarize key topics…
ClusteringText SummarizationReliable Pseudo-labeling via Optimal Transport with Attention for Short Text Clustering
Short text clustering has gained significant attention in the data mining community. However, the limited valuable information contained in short texts often leads to low-discriminative representations, increasing the di…
ClusteringContrastive LearningRepresentation LearningShort Text Clustering+1CSAL: Self-adaptive Labeling based Clustering Integrating Supervised Learning on Unlabeled Data
Supervised classification approaches can predict labels for unknown data because of the supervised training process. The success of classification is heavily dependent on the labeled training data. Differently, clusterin…
ClassificationClusteringGeneral ClassificationDual-level Adaptive Self-Labeling for Novel Class Discovery in Point Cloud Segmentation
We tackle the novel class discovery in point cloud segmentation, which discovers novel classes based on the semantic knowledge of seen classes. Existing work proposes an online point-wise clustering method with a simplif…
ClusteringNovel Class DiscoveryPoint Cloud SegmentationSegmentation+1MaxiMin Active Learning in Overparameterized Model Classes}
Generating labeled training datasets has become a major bottleneck in Machine Learning (ML) pipelines. Active ML aims to address this issue by designing learning algorithms that automatically and adaptively select the mo…
Active LearningBIG-bench Machine LearningBinary Classificationmodel