0% Complete
فارسی
Home
/
پانزدهمین کنفرانس بین المللی فناوری اطلاعات و دانش
Evaluating LLMs in Persian News Summarization
Authors :
Arya VarastehNezhad
1
Reza Tavasoli
2
Mostafa Masumi
3
Seyed Soroush Majd
4
Mehrnoush Shamsfard
5
1- University of Tehran
2- University of South Carolina
3- Sharif University of Technology
4- shahid beheshti university
5- shahid beheshti university
Keywords :
Text Summarization،Large Language Models،Persian News،LLM Evaluation،Natural Language Processing،Artificial Intelligence
Abstract :
This study evaluates the performance of eight Large Language Models (LLMs) in Persian news summarization: GPT-4o, Claude-3.5-Sonnet, Gemini-Pro-1.5, Llama-3.1-405B, Command-R, Mistral-Large-2, DeepSeek V2.5, and Gemma-2-9B. We assess these models across five news categories: Economy, International, Sports, Technology, and Social, using the pn_summary dataset. Our evaluation employs multiple metrics, including BERTScore and ROUGE, across two input conditions: article-only and article-with-title. Results show that Llama-3.1-405b performed best against reference summaries in the article-only setting, achieving the highest BERTScore F1 (50.60) and ROUGE-L (33.96) scores. Notably, including article titles helped models produce summaries which aligned more closely to the reference summary, increasing the average BERTScore F1 from 48.31 to 50.16 across most models. Moreover, when comparing generated summaries to original articles, Mistral-Large-2 led with a BERTScore F1 of 48.09. In category-specific analysis, Mistral-Large-2 consistently outperformed the reference summaries across all news categories, with the most significant improvement in the Economic category. This study provides valuable insights into the current capabilities of LLMs for Persian summarization, highlighting their potential and the impact of input structure on performance. Our findings contribute to the growing body of research on multilingual summarization and have practical implications for Persian language processing applications.
Papers List
List of archived papers
Sparse Beamforming Design for Non-Coherent UD-CRAN with mm-Wave Fronthaul Links
Alireza M. Hosseini - Dr Abbas Mohammadi
A Swarm Intelligence Approach to Design Optimal Repeaters in Multilayer Graphene Nanoribbon Interconnects
Majid Sanaeepur - Maryam Momeni
Presentation of a New Decoder Based on Quantum Cellular Automata Technology Along with an Analysis of Energy Consumption
- - -
پیش بینی بیماری قلبی با استفاده از روش تحلیل شبکه ای
هدیه مشتاقی محمدزاده - فاطمه باقری
A Novel Service Deployment Policy in Fog Computing Considering The Degree of Availability and Fog Landscape Utilization Using Multiobjective Evolutionary Algorithms
Maryam Eslami - Dr Mehdi Sakhaei-nia
کشف برخط تقلب پیشنهاد ساختگی (Bid-Shielding) در مناقصه و مزایدههای الکترونیکی هلندی با رویکرد تحلیل شبکه اجتماعی
فاطمه الثلایا - دکتر سید علیرضا هاشمی گلپایگانی فاطمه الثلایا - سید علیرضا هاشمی گلپایگانی -
Optimal selection of seed nodes by reducing the influence of common nodes in the influence maximization problem
Farzaneh Kazemzadeh - Ali Asghar Safaei - Mitra Mirzarezaee
یک رویکرد سریع تحلیل و شناسایی آسیب پذیری Next-Intent در برنامه های کاربردی اندروید
زهرا کلوندی - دکتر مهدی سخائی نیا زهرا کلوندی - مهدی سخائی نیا -
پیاده سازی موازی یک طرح (t,n)-تسهیم چند تصویر با استفاده از GPU
سعیده کبیری راد
COVID-19 Image Retrieval Using Siamese Deep Neural Network and Hashing Technique
Farsad Zamani Boroujeni - Doryaneh Hossein Afshari - Fatemeh Mahmoodi
more
Samin Hamayesh - Version 41.3.1