0% Complete
English
صفحه اصلی
/
پانزدهمین کنفرانس بین المللی فناوری اطلاعات و دانش
Evaluating LLMs in Persian News Summarization
نویسندگان :
Arya VarastehNezhad
1
Reza Tavasoli
2
Mostafa Masumi
3
Seyed Soroush Majd
4
Mehrnoush Shamsfard
5
1- University of Tehran
2- University of South Carolina
3- Sharif University of Technology
4- shahid beheshti university
5- shahid beheshti university
کلمات کلیدی :
Text Summarization،Large Language Models،Persian News،LLM Evaluation،Natural Language Processing،Artificial Intelligence
چکیده :
This study evaluates the performance of eight Large Language Models (LLMs) in Persian news summarization: GPT-4o, Claude-3.5-Sonnet, Gemini-Pro-1.5, Llama-3.1-405B, Command-R, Mistral-Large-2, DeepSeek V2.5, and Gemma-2-9B. We assess these models across five news categories: Economy, International, Sports, Technology, and Social, using the pn_summary dataset. Our evaluation employs multiple metrics, including BERTScore and ROUGE, across two input conditions: article-only and article-with-title. Results show that Llama-3.1-405b performed best against reference summaries in the article-only setting, achieving the highest BERTScore F1 (50.60) and ROUGE-L (33.96) scores. Notably, including article titles helped models produce summaries which aligned more closely to the reference summary, increasing the average BERTScore F1 from 48.31 to 50.16 across most models. Moreover, when comparing generated summaries to original articles, Mistral-Large-2 led with a BERTScore F1 of 48.09. In category-specific analysis, Mistral-Large-2 consistently outperformed the reference summaries across all news categories, with the most significant improvement in the Economic category. This study provides valuable insights into the current capabilities of LLMs for Persian summarization, highlighting their potential and the impact of input structure on performance. Our findings contribute to the growing body of research on multilingual summarization and have practical implications for Persian language processing applications.
لیست مقالات
لیست مقالات بایگانی شده
تشخیص مراحل خواب با کمک جنگل تصادفی و ویژگی های فرکانسی استخراج شده از سیگنال های EEG و EOG
سیدعلی حسینی
Inner and Outer Bearing Fault Diagnosis of electrical Motors Using a Proposed Algorithm and Vibration Signals
Vahid Safari Dehnavi - Masoud Shafiee
AI-based Secure Intrusion Detection Framework for Digital Twin-enabled Critical Infrastructure
Tanisha Patel - Nilesh Kumar Jadav - Tejal Rathod - Sudeep Tanwar - Deepak Garg - Hossein Shahinzadeh
Adaptive Stopping Criteria-based A-RANSAC algorithm in Copy Move Image Forgery detection
ZAHRA HOSEINNEJAD - Dr MEHDI NASRI
ParsEL 1.0: Unsupervised Entity Linking in Persian Social Media Texts
Majid Asgari-bidhendi - Farzane Fakhrian - Dr Behrouz Minaei-bidgoli
DynamicEvoStream : خوشه بندی پویای جریان داده تکاملی در زمانهای بیکاری
زهرا عمیقی - مرتضی یوسف صنعتی - میرحسین دزفولیان
Aspect-Based Sentiment Analysis of After-Sales Service Quality: A Case Study of Snowa and Competitors Using Digikala Reviews
Safiyeh Samadanian - Marjan Kaedi
ParaKavosh: A Parallel Algorithm for Finding Biological Network Motifs
Dr Zahra Razaghi Moghadam Kashani - Dr Ali Masoudi-nejad - Dr Abbas Nowzari-dalini
بکارگیری الگوریتم بهینه سازی فاخته و منطق فازی به منظور بهبود زمانبندی وظایف در محیط محاسبات مه
فاطمه دوامی - حمید جلیلوند - فاطمه نجفی
A Community-Based Method for Identifying Influential Nodes using Network Embedding
Nargess Vafaei - Dr Mohammad Reza Keyvanpour
بیشتر
ثمین همایش، سامانه مدیریت کنفرانس ها و جشنواره ها - نگارش 42.5.2