لطفا منتظر بمانید ...
0% Complete
English
صفحه اصلی
/
پانزدهمین کنفرانس بین المللی فناوری اطلاعات و دانش
Evaluating LLMs in Persian News Summarization
نویسندگان :
Arya VarastehNezhad
1
Reza Tavasoli
2
Mostafa Masumi
3
Seyed Soroush Majd
4
Mehrnoush Shamsfard
5
1- University of Tehran
2- University of South Carolina
3- Sharif University of Technology
4- shahid beheshti university
5- shahid beheshti university
کلمات کلیدی :
Text Summarization،Large Language Models،Persian News،LLM Evaluation،Natural Language Processing،Artificial Intelligence
چکیده :
This study evaluates the performance of eight Large Language Models (LLMs) in Persian news summarization: GPT-4o, Claude-3.5-Sonnet, Gemini-Pro-1.5, Llama-3.1-405B, Command-R, Mistral-Large-2, DeepSeek V2.5, and Gemma-2-9B. We assess these models across five news categories: Economy, International, Sports, Technology, and Social, using the pn_summary dataset. Our evaluation employs multiple metrics, including BERTScore and ROUGE, across two input conditions: article-only and article-with-title. Results show that Llama-3.1-405b performed best against reference summaries in the article-only setting, achieving the highest BERTScore F1 (50.60) and ROUGE-L (33.96) scores. Notably, including article titles helped models produce summaries which aligned more closely to the reference summary, increasing the average BERTScore F1 from 48.31 to 50.16 across most models. Moreover, when comparing generated summaries to original articles, Mistral-Large-2 led with a BERTScore F1 of 48.09. In category-specific analysis, Mistral-Large-2 consistently outperformed the reference summaries across all news categories, with the most significant improvement in the Economic category. This study provides valuable insights into the current capabilities of LLMs for Persian summarization, highlighting their potential and the impact of input structure on performance. Our findings contribute to the growing body of research on multilingual summarization and have practical implications for Persian language processing applications.
لیست مقالات
لیست مقالات بایگانی شده
Enhancing Supervised Learning in Speech Emotion Recognition through Unsupervised Representations
Niloufar Faridani - Amirali Soltani Tehrani - Ramin Toosi
AI-Powered Beauty Insights: Sentiment Analysis in a Low-Resource Language
Sajedeh Talebi - Neda Abdolvand - Fatemeh Mahdian
KGLM-QA: A Novel Approach for Knowledge Graph-Enhanced Large Language Models for Question Answering
Alireza Akhavan safaei - Pegah Saboori - Reza Ramezani - Mohammadali Nematbakhsh
Conceptual Intelligent Model for Visual Question Answering using Attention Mechanism and Relational Reasoning
ٍElham Alighardash - Dr Hassan Khotanlou - Vahid Pour Amin
Establishing security using cryptography and biometric authentication to counter cyber-attacks
Mohammed ADIL AKABR - Mehdi Hamidkhani - Mostafa Sadeghi
Classification of mental states of human concentration based on EEG signal
Mehran Safari Dehnavi - Vahid Safari Dehnavi - Dr Masoud Shafiee
Mode Selection and Resource Allocation in D2D-Enabled MC-NOMA using Matching Theory
Alireza Gholamrezaee - Hamid Farrokhi - Javad Zeraatkar Moghaddam
Benchmarking Embedding Models for Persian-Language Semantic Information Retrieval
Mahmood Kalantari - Mehdi Feghhi - Nasser Mozayani
Binary water stream algorithm: a new meta-heuristic optimization technique
Faezeh Rahimi Sebdani - Mehdi Nasri
Targeted Vaccination for COVID-19 Using Mobile Communication Networks
Mohammadmohsen Jadidi - Pegah Moslemi - Saeed Jamshidiha - Iman Masroori - Abbas Mohammadi - Vahid Pourahmadi
بیشتر
ثمین همایش، سامانه مدیریت همایش ها و جشنواره ها - نگارش 45.0.1