0% Complete
English
صفحه اصلی
/
پانزدهمین کنفرانس بین المللی فناوری اطلاعات و دانش
Evaluating LLMs in Persian News Summarization
نویسندگان :
Arya VarastehNezhad
1
Reza Tavasoli
2
Mostafa Masumi
3
Seyed Soroush Majd
4
Mehrnoush Shamsfard
5
1- University of Tehran
2- University of South Carolina
3- Sharif University of Technology
4- shahid beheshti university
5- shahid beheshti university
کلمات کلیدی :
Text Summarization،Large Language Models،Persian News،LLM Evaluation،Natural Language Processing،Artificial Intelligence
چکیده :
This study evaluates the performance of eight Large Language Models (LLMs) in Persian news summarization: GPT-4o, Claude-3.5-Sonnet, Gemini-Pro-1.5, Llama-3.1-405B, Command-R, Mistral-Large-2, DeepSeek V2.5, and Gemma-2-9B. We assess these models across five news categories: Economy, International, Sports, Technology, and Social, using the pn_summary dataset. Our evaluation employs multiple metrics, including BERTScore and ROUGE, across two input conditions: article-only and article-with-title. Results show that Llama-3.1-405b performed best against reference summaries in the article-only setting, achieving the highest BERTScore F1 (50.60) and ROUGE-L (33.96) scores. Notably, including article titles helped models produce summaries which aligned more closely to the reference summary, increasing the average BERTScore F1 from 48.31 to 50.16 across most models. Moreover, when comparing generated summaries to original articles, Mistral-Large-2 led with a BERTScore F1 of 48.09. In category-specific analysis, Mistral-Large-2 consistently outperformed the reference summaries across all news categories, with the most significant improvement in the Economic category. This study provides valuable insights into the current capabilities of LLMs for Persian summarization, highlighting their potential and the impact of input structure on performance. Our findings contribute to the growing body of research on multilingual summarization and have practical implications for Persian language processing applications.
لیست مقالات
لیست مقالات بایگانی شده
Enhancing Software Effort Estimation with an Integrated Approach of Particle Swarm Optimization and Genetic Algorithms in Analogy-based Method
Ehsan Nasr - Keyvan Mohebbi
Combinatorial Auction Based on Social Choice in the Internet of Things
Maede Esmaeili - Faria Nassiri-Mofakham - Fatemeh Hassanvand
ISAAF: بهبود چارچوب مجوز خودتطبیق SAAF با استفاده از پیادهسازی مبتنی بر عامل و مفهوم I-Shairing
الهام معین الدینی - دکتر منیره عبدوس - دکتر اسلام ناظمی
Embedded speech encoder for low-resource languages
Alireza A.Tabatabaei - Pouria Sameti - Ali Bohlooli
Using Deconvolutional Variational Autoencoder for Answer Selection in Community Question Answering
Golshan Afzali Boroujeni - Heshaam Faili
Face Recognition Based on Local Statistical Features and Artificial Neural Network
Mehdi Moghimi - Dr Hadi Grailu
شبکههای نرمافزار محور در کلان داده: مطالعهی راهکارهای امنیتی و چالشها
احسان سلیمانی دهکردی - محمدرضا ملاخلیلی میبدی
سیستم توصیه گر برای خرید لوازم آرایشی و بهداشتی مبتنی بر الگوریتم جنگل تصادفی
فاطمه رمضانی خوزستانی - مجید رفیعی
A Survey on Utilizing Reinforcement Learning in Wireless Sensor Networks Routing Protocols
Ali Forghani Elah Abadi - Seyedeh Elham Asghari - Sepideh Sharifani - Seyyed Amir Asghari - Mohammadreza Binesh Marvasti
Towards Provable Privacy Protection in IoT-Health Applications
Samane Sobuti - دکتر سیاوش خرسندی
بیشتر
ثمین همایش، سامانه مدیریت کنفرانس ها و جشنواره ها - نگارش 41.3.1