CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code
Recent works has widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account source code specifics. We propose subtokenziation that reduces average length by 17--40% without downstream performance drop, and show that a carefully chosen subtokenization may significantly improve quality by 0.5-2%, possibly with some length increase.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code
Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model arch…
Language ModelingLanguage ModellingLarge Language ModelHearing the Order: Investigating Position Bias in Large Audio-Language Models
Large audio-language models (LALMs) are often used in tasks that involve reasoning over ordered options. An open question is whether their predictions are influenced by the order of answer choices, which would indicate a…
Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions
Large Language Models (LLMs) have demonstrated remarkable capabilities in various NLP tasks. However, previous works have shown these models are sensitive towards prompt wording, and few-shot demonstrations and their ord…
Multiple-choiceSensitivityHigh-Frequency Options Trading | With Portfolio Optimization
This paper explores the effectiveness of high-frequency options trading strategies enhanced by advanced portfolio optimization techniques, investigating their ability to consistently generate positive returns compared to…
Portfolio OptimizationInvestigating Speech Recognition for Improving Predictive AAC
Making good letter or word predictions can help accelerate the communication of users of high-tech AAC devices. This is particularly important for real-time person-to-person conversations. We investigate whether per form…
Language ModelingLanguage Modellingspeech-recognitionSpeech Recognition