0% Complete
English
صفحه اصلی
/
پانزدهمین کنفرانس بین المللی فناوری اطلاعات و دانش
Evaluating LLMs in Persian News Summarization
نویسندگان :
Arya VarastehNezhad
1
Reza Tavasoli
2
Mostafa Masumi
3
Seyed Soroush Majd
4
Mehrnoush Shamsfard
5
1- University of Tehran
2- University of South Carolina
3- Sharif University of Technology
4- shahid beheshti university
5- shahid beheshti university
کلمات کلیدی :
Text Summarization،Large Language Models،Persian News،LLM Evaluation،Natural Language Processing،Artificial Intelligence
چکیده :
This study evaluates the performance of eight Large Language Models (LLMs) in Persian news summarization: GPT-4o, Claude-3.5-Sonnet, Gemini-Pro-1.5, Llama-3.1-405B, Command-R, Mistral-Large-2, DeepSeek V2.5, and Gemma-2-9B. We assess these models across five news categories: Economy, International, Sports, Technology, and Social, using the pn_summary dataset. Our evaluation employs multiple metrics, including BERTScore and ROUGE, across two input conditions: article-only and article-with-title. Results show that Llama-3.1-405b performed best against reference summaries in the article-only setting, achieving the highest BERTScore F1 (50.60) and ROUGE-L (33.96) scores. Notably, including article titles helped models produce summaries which aligned more closely to the reference summary, increasing the average BERTScore F1 from 48.31 to 50.16 across most models. Moreover, when comparing generated summaries to original articles, Mistral-Large-2 led with a BERTScore F1 of 48.09. In category-specific analysis, Mistral-Large-2 consistently outperformed the reference summaries across all news categories, with the most significant improvement in the Economic category. This study provides valuable insights into the current capabilities of LLMs for Persian summarization, highlighting their potential and the impact of input structure on performance. Our findings contribute to the growing body of research on multilingual summarization and have practical implications for Persian language processing applications.
لیست مقالات
لیست مقالات بایگانی شده
A Model-Driven Approach for Automatic Generation of Android Tourism Applications
Sara Adib - Bahman Zamani
A New Routing Protocol in Internet of Vehicles Inspired of Spread Model of the Covid-19 Virus
Taha Yasin Rezapour - Esmaeil Zeinali - Reza Ebrahimi Atani - Mohammad Mehdi Gilanian Sadeghi
Effective Classifier for Predicting Churn in Payment Terminals Using RFM model and Deep Neural Network
Dr Mahila Dadfarnia - Ali Alemi Matinpour - Dr Monireh Abdoos
تشخیص ارتباط معنایی در استکاورفلو با رمزگذار جمله جهانی
مجید دلیری - جعفر حبیبی - عیسی انامرادنژاد
تخلیهبار محاسباتی ریزدانه تحرکآگاه در رایانش لبه برای اینترنت اشیاء
شکوفه نوروزی - دکتر زینب موحدی شکوفه نوروزی - زینب موحدی -
Mode Selection and Resource Allocation in D2D-Enabled MC-NOMA using Matching Theory
Alireza Gholamrezaee - Hamid Farrokhi - Javad Zeraatkar Moghaddam
جایگزینی دارو براساس پیشبینی یال روی گرافهای ناهمگون با بهرهگیری از جاسازی گراف ناهمگون
رسول سامانی - فهیمه شاهرخ شهرکی - دکتر ناصر قدیری رسول سامانی - فهیمه شاهرخ شهرکی - ناصر قدیری -
Silicon photonic microring resonators: A Novel optical router based on Negative-First routing algorithm
Negin Bagheri Renani - Elham Yaghoubi
Business Process Improvement Challenges: A Systematic Literature Review
Hanieh Kashfi - Fereidoon Shams Aliee
Using Deconvolutional Variational Autoencoder for Answer Selection in Community Question Answering
Golshan Afzali Boroujeni - Heshaam Faili
بیشتر
ثمین همایش، سامانه مدیریت کنفرانس ها و جشنواره ها - نگارش 41.3.1