Showing Your Work Doesn't Always Work
In natural language processing, a recently popular line of work explores how to best report the experimental results of neural networks. One exemplar publication, titled "Show Your Work: Improved Reporting of Experimental Results," advocates for reporting the expected validation effectiveness of the best-tuned model, with respect to the computational budget. In the present work, we critically examine this paper. As far as statistical generalizability is concerned, we find unspoken pitfalls and caveats with this approach. We analytically show that their estimator is biased and uses error-prone assumptions. We find that the estimator favors negative errors and yields poor bootstrapped confidence intervals. We derive an unbiased alternative and bolster our claims with empirical evidence from statistical simulation. Our codebase is at http://github.com/castorini/meanmax.
Code (1)
Similar Papers 제목 키워드 기반
Domain Mismatch Doesn’t Always Prevent Cross-lingual Transfer Learning
Cross-lingual transfer learning without labeled target language data or parallel text has been surprisingly effective in zero-shot cross-lingual classification, question answering, unsupervised machine translation, etc. …
Bilingual Lexicon InductionCross-Lingual TransferMachine TranslationQuestion Answering+4Domain Mismatch Doesn't Always Prevent Cross-Lingual Transfer Learning
Cross-lingual transfer learning without labeled target language data or parallel text has been surprisingly effective in zero-shot cross-lingual classification, question answering, unsupervised machine translation, etc. …
Bilingual Lexicon InductionCross-Lingual TransferMachine TranslationQuestion Answering+4Interior Object Detection and Color Harmonization
Confused about renovating your space? Choosing the perfect color for your walls is always a challenging task. One does rounds of color consultation and several patch tests. This paper proposes an AI tool to pitch paint b…
Image SegmentationObjectobject-detectionObject Detection+2Domain-Specific New Words Detection in Chinese
With the explosive growth of Internet, more and more domain-specific environments appear, such as forums, blogs, MOOCs and etc. Domain-specific words appear in these areas and always play a critical role in the domain-sp…
Computerized enrollment system project report
The project entitled “COMPUTERIZED ENROLLMENT SYSTEM”. Every organization, whether big or small, has human resource challenges to overcome. Every organization has different employee management needs; therefore we design …
Management