0% Complete
English
صفحه اصلی
/
پانزدهمین کنفرانس بین المللی فناوری اطلاعات و دانش
Evaluating LLMs in Persian News Summarization
نویسندگان :
Arya VarastehNezhad
1
Reza Tavasoli
2
Mostafa Masumi
3
Seyed Soroush Majd
4
Mehrnoush Shamsfard
5
1- University of Tehran
2- University of South Carolina
3- Sharif University of Technology
4- shahid beheshti university
5- shahid beheshti university
کلمات کلیدی :
Text Summarization،Large Language Models،Persian News،LLM Evaluation،Natural Language Processing،Artificial Intelligence
چکیده :
This study evaluates the performance of eight Large Language Models (LLMs) in Persian news summarization: GPT-4o, Claude-3.5-Sonnet, Gemini-Pro-1.5, Llama-3.1-405B, Command-R, Mistral-Large-2, DeepSeek V2.5, and Gemma-2-9B. We assess these models across five news categories: Economy, International, Sports, Technology, and Social, using the pn_summary dataset. Our evaluation employs multiple metrics, including BERTScore and ROUGE, across two input conditions: article-only and article-with-title. Results show that Llama-3.1-405b performed best against reference summaries in the article-only setting, achieving the highest BERTScore F1 (50.60) and ROUGE-L (33.96) scores. Notably, including article titles helped models produce summaries which aligned more closely to the reference summary, increasing the average BERTScore F1 from 48.31 to 50.16 across most models. Moreover, when comparing generated summaries to original articles, Mistral-Large-2 led with a BERTScore F1 of 48.09. In category-specific analysis, Mistral-Large-2 consistently outperformed the reference summaries across all news categories, with the most significant improvement in the Economic category. This study provides valuable insights into the current capabilities of LLMs for Persian summarization, highlighting their potential and the impact of input structure on performance. Our findings contribute to the growing body of research on multilingual summarization and have practical implications for Persian language processing applications.
لیست مقالات
لیست مقالات بایگانی شده
جایگزینی دارو براساس پیشبینی یال روی گرافهای ناهمگون با بهرهگیری از جاسازی گراف ناهمگون
رسول سامانی - فهیمه شاهرخ شهرکی - دکتر ناصر قدیری رسول سامانی - فهیمه شاهرخ شهرکی - ناصر قدیری -
A clonal selection mechanism for load balancing in the cloud computing system
Melika Mosayyebi - Reza Azmi
A Novel Decentralized Privacy Preserving Federated Learning Model for Healthcare Applications
Saba Ameri - Reza Ebrahimi Atani
An ESB-based Architecture for Authentication as a Service Through Enterprise Application Integration
Masoumeh Hashemi - Mehdi Sakhaei-nia - Morteza Yousef Sanati
تحلیل سازههای موثر بر پذیرش فناوری بلاکچین و استفاده از آن در صنعت بیمه ایران با استفاده از تکنیک معادلات ساختاری (مطالعه موردی: شرکت کارگزاری رسمی بیمه زندگی خوب)
احسان هنری - آفرین اخوان
Statistical Disorder Parameters Computing For Hyperspectral Image Anomaly Detection
Dr Maryam Imani
ML-based Optical Fibre Fault Detection in Smart Surveillance and Traffic Systems
Rushil Patel - Sana Narmawala - Nikunjkumar Mahida - Rajesh Gupta - Sudeep Tanwar - Hossein Shahinzadeh
Combinatorial Auction Based on Social Choice in the Internet of Things
Maede Esmaeili - Faria Nassiri-Mofakham - Fatemeh Hassanvand
استخراج ویژگی مجموعه دادههای پزشکی دارای ابعاد بالا با استفاده از برنامه نویسی ژنتیک چند منظوره
سحر فقیهی راد - دکتر سیده نفیسه آل محمد سحر فقیهی راد - سیده نفیسه آل محمد -
Distributed Learning Automata-based Algorithm for Finding K-Clique in Complex Social Networks
Mohammad Mehdi Daliri Khomami - Alireza Rezvanian - Ali Mohammad Saghiri - Mohammad Reza Meybodi
بیشتر
ثمین همایش، سامانه مدیریت کنفرانس ها و جشنواره ها - نگارش 42.5.2