MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point
In this paper, we introduce MCTensor, a library based on PyTorch for providing general-purpose and high-precision arithmetic for DL training. MCTensor is used in the same way as PyTorch Tensor: we implement multiple basic, matrix-level computation operators and NN modules for MCTensor with identical PyTorch interface. Our algorithms achieve high precision computation and also benefits from heavily-optimized PyTorch floating-point arithmetic. We evaluate MCTensor arithmetic against PyTorch native arithmetic for a series of tasks, where models using MCTensor in float16 would match or outperform the PyTorch model with float32 or float64 precision.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CuFP: An HLS Library for Customized Floating-Point Operators
High-Level Synthesis (HLS) tools have revolutionized FPGA application development by providing a more efficient and streamlined approach, significantly impacting digital design methodologies. Despite the capability of FP…
High-Level SynthesisCompressing deep neural networks on FPGAs to binary and ternary precision with HLS4ML
We present the implementation of binary and ternary neural networks in the hls4ml library, designed to automatically convert deep neural network models to digital circuits with FPGA firmware. Starting from benchmark mode…
Handwritten Digit RecognitionFINN-L: Library Extensions and Design Trade-off Analysis for Variable Precision LSTM Networks on FPGAs
It is well known that many types of artificial neural networks, including recurrent networks, can achieve a high classification accuracy even with low-precision weights and activations. The reduction in precision general…
Optical Character RecognitionOptical Character Recognition (OCR)QuantizationTernGEMM: GEneral Matrix Multiply Library with Ternary Weights for Fast DNN Inference
Efficient implementation of deep neural networks on CPU-based systems is very critical because applications proliferate to embedded and Internet of Things (IoT) systems. Many CPUs for personal computers and embedded syst…
CPUFBGEMM: Enabling High-Performance Low-Precision Deep Learning Inference
Deep learning models typically use single-precision (FP32) floating point data types for representing activations and weights, but a slew of recent research work has shown that computations with reduced-precision data ty…
Code GenerationDeep LearningQuantizationVocal Bursts Intensity Prediction