Benchmarking Arabic LLMs: Dialect Coverage in Frontier Models
Deploying AI in the Arab world requires moving past Modern Standard Arabic. Here is how GPT-4o, Claude 3.5, and Gemini 1.5 handle real-world regional dialects.

Modern Standard Arabic (MSA) is the lingua franca of formal Arab bureaucracy, news broadcasting, and academic publishing. It is also the language on which virtually all commercial large language models are trained when vendors claim Arabic capability. Yet, none of the 400 million native speakers across the Middle East and North Africa use Modern Standard Arabic in their daily consumer interactions, customer support tickets, internal enterprise chats, or commercial voice communications.
For enterprise leaders deploying generative AI across the Gulf, Levant, Egypt, or North Africa, this linguistic divide presents an immediate operational risk. A customer service copilot or internal retrieval system that parses only MSA fails on contact with real customer intent.
To establish an objective baseline for enterprise procurement, Masar evaluated the leading frontier models—OpenAI's GPT-4o, Anthropic's Claude 3.5 Sonnet, and Google's Gemini 1.5 Pro—across dialectal comprehension, code-switching, and task completion.
The Dialect Spectrum: Regional Nuances
Arabic is not a single language with minor accents; it is a diglossic continuum. We benchmarked models across four core linguistic groupings:
- Gulf Arabic (Khaleeji): Predominant in Saudi Arabia, the UAE, Qatar, and Kuwait. Characterized by specific lexical markers and heavy intermixing with English business terminology in enterprise settings.
- Egyptian Arabic (Masri): The most widely understood dialect regionally due to media export, featuring distinct verbal syntax and negation patterns.
- Levantine Arabic (Shami): Common across Jordan, Lebanon, Syria, and Palestine, with substantial loanwords and distinct phonetic variations.
- North African (Maghrebi / Darija): The most linguistically distinct, integrating French and Berber syntactical structures, presenting severe challenges for standard Arabic tokenizers.
The benchmark for Arabic AI cannot be measured by syntactic accuracy in MSA. The real metric is whether a model understands commercial intent expressed in regional colloquialisms without requiring the user to adapt their natural speech.
Model-by-Model Analysis
OpenAI: GPT-4o
GPT-4o demonstrates the highest baseline speed and lowest tokenization overhead across Arabic scripts due to OpenAI's revised vocabulary tokenizer. In standard MSA and Egyptian Arabic, GPT-4o provides reliable classification, sentiment extraction, and summarization.
When evaluated on Gulf Arabic, GPT-4o accurately maps colloquial phrasing into formal actions, making it well-suited for GCC-focused retail and banking workflows. However, it displays a tendency to normalize output back into formal MSA unless explicitly instructed to retain dialectal tone. On Maghrebi Darija, performance drops sharply; semantic understanding remains acceptable, but generation frequently collapses into awkward French-Arabic blends.
Anthropic: Claude 3.5 Sonnet
Claude 3.5 Sonnet delivers the strongest performance in nuanced reasoning, complex context retention, and strict adherence to enterprise instructions in Arabic environments.
Where Claude distinguishes itself is in dialectal comprehension during code-switching—the common practice in Dubai, Riyadh, and Beirut of alternating between Arabic and English within the same sentence. Claude accurately untangles mixed-script prompts that destabilize other models. Its grasp of Levantine nuance is notably superior to competitors, capturing underlying sentiment and colloquial sarcasm that automated sentiment analysis typically misreads. Its primary trade-off remains tokenization efficiency; Arabic text consumes significantly more tokens per word in Claude than in GPT-4o, impacting operational inference costs at scale.
Google: Gemini 1.5 Pro
Gemini 1.5 Pro benefits from Google's extensive multilingual indexing history. Its massive context window allows enterprises to feed large volumes of local legal documentation, call center transcripts, and vernacular knowledge bases directly into the prompt.
Gemini shows surprising strength in Maghrebi dialects, where both GPT-4o and Claude stumble. Its retrieval accuracy against raw, non-standardized Arabic text is robust. However, Gemini's generative output in colloquial Arabic often sounds mechanical. While it understands dialectal inputs with high fidelity, its native dialect generation lacks the conversational fluency required for direct consumer-facing interfaces without fine-tuning.
Key Technical Takeaways for Architecture Design
Evaluating dialectal coverage reveals three structural realities that enterprise architects must account for:
1. Tokenizer Cost Variance
Arabic tokenization remains inefficient across all major providers compared to Latin scripts. A query in Gulf Arabic can consume between 1.8 to 2.6 times the number of tokens required for an equivalent English query. GPT-4o mitigates this better than Claude 3.5 Sonnet, meaning high-volume transactional systems must balance Claude's reasoning edge against GPT-4o's unit economics.
2. The Arabizi Blind Spot
In everyday digital communication, millions of users across the Levant and North Africa write Arabic using the Latin alphabet and numbers (Arabizi or Franco-Arabic). Frontier models handle common Arabizi expressions adequately in isolated zero-shot prompts, but accuracy degrades rapidly in complex multi-turn dialogues. Dedicated preprocessing layers remain mandatory for consumer applications.
3. Orchestration Over Monolithic Selection
No single frontier model leads across every dialect and evaluation vector. High-performing enterprise architectures in the MENA region increasingly rely on routing layers: using lightweight classification to detect dialect, routing comprehension to Claude 3.5 Sonnet or Gemini 1.5 Pro, and utilizing fine-tuned open-weight models or GPT-4o for generation and cost-efficient task execution.
Strategic Recommendations
Organizations scaling AI operations in the Arab world must stop relying on global benchmarks that evaluate Arabic solely through academic corpora. Enterprise testing protocols should incorporate real-world customer transcripts, dialectal voice-to-text outputs, and regional business idioms. Selecting an AI partner who understands the nuance between Gulf commercial colloquialism and Levantine colloquialism is not a localization detail; it is the difference between an AI implementation that drives adoption and one that produces customer friction.