Please wait ...
0% Complete
فارسی
Home
/
شانزدهمین کنفرانس بین المللی فناوری اطلاعات و دانش
Design of low-latency Floating-Point units for Softmax Computation in Transformer-based Large Language Models
Authors :
Hoda Ghabeli
1
Amir Sabbagh Molahosseini
2
1- دانشکاه آزاد کرمان
2- دانشکاه آزاد کرمان
Keywords :
LLM،transformer،softmax،speculative،floating-point
Abstract :
Large Language Models (LLMs) have emerged as one of the most desirable and widely used interactive digital tools in the world in the last decade. Softmax is one of the key steps in LLMs where the output is a vector of probabilities for each token in the model dictionary. The softmax computations are time-consuming due to the large vocabulary size, which can significantly increase the exponential computations and normalization, impacting the overall speed of the model. Given the importance of accuracy and speed, some of the main operations and computations of softmax are performed on the floating-point units. Arithmetic speculative computations are considered when the result of the computations can be estimated from a path shorter than the critical path, with improved speedup. In this paper, speculative 32-bit floating-point computation is proposed by merging two formats, 32-bit and 16-bit, for softmax computations. Both the floating-point adder and the floating-point multiplier use this strategy. The proposed design, based on the input data of the softmax function, speculates that the 32-bit floating-point computations can be obtained by concatenating the result of 16-bit format and a part of the 32-bit format result, that gives correct results most of the time with less delay. If speculation is unsuccessful, the longer path from through the conventional 32-bit floating-point unit is activated at the cost of a slightly longer critical path. Experimental results show that speculative floating-point units lead to a reduction in delay with only marginal overhead in area and power consumption.
Papers List
List of archived papers
Analysing effect of news polarity on stock market prediction: a machine learning approach
Golshid Ranjbaran - Dr Mohammad-Shahram Moin - Dr Sasan H Alizadeh - Dr Abbas Koochari
بررسی تأثیر استقرار استاندارد COBIT در افزایش بهره وری سازمانها (مطالعه موردی: شعب نمایندگیهای همراه اول، ایرانسل، رایتل)
دکتر محمد ابراهیم سمیع - ساره رحمانیان محمد ابراهیم سمیع - ساره رحمانیان -
Data Analysis to Reduce Electrical Power Plants
Amirali Sahraei - Jamshid Shanbehzadeh
SBST challenges from the perspective of the test techniques
Sepideh Kashefi Gargari - Dr Mohammad Reza Keyvanpour
A Blockchain-Based Smart Contract Framework for Peer-to-Peer Energy Trading in Smart Grids
Hossein Shahinzadeh - Farshad Ebrahimi - S. Mohammadali Zanjani - Amirafshin Zamani - Saiedeh Mehrabani-Najafabadi - Gevork B. Gharehpetian
IoMT-Enabled Smart Healthcare: State-of-the-Art, Security and Future Directions
Shivam Tripathi - Vatsalkumar Makwana - Malaram Kumhar - Harshal Trivedi - Jitendra Bhatia - Sudeep Tanwar - Hossein Shahinzadeh
آسیب شناسی استقرار بلاکچین در صنعت بانکی کشور ایران
نیلوفر مرادحاصل
A Deep Neural Network-based Method for MmWave Time-varying Channel Estimation
Amirhossein Molazadeh - Zahra Maroufi - Mehrdad Ardebilipour
انتخاب ویژگی با استفاده از الگوریتم بهینه سازی ذرات مبتنی بر استراتژی خود تطبیقی دودویی جهت تشخیص بیماری
الهام صالحی - دکتر محمدرضا کرمی ملایی - دکتر حسام عمرانپور الهام صالحی - محمدرضا کرمی ملایی - حسام عمرانپور -
ارزیابی و برنامهریزی اجرای پیشنهادی هوش مصنوعی در صنعت پتروشیمی ایران
امین رضا انصاری - احد قائمی - سید مهدی کوچک کوثری
more
Samin Hamayesh - Version 45.0.1