Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva-3B, a large language model trained from scratch primarily on Italian data, scored just 4.9% on the INVALSI Italian school-exam benchmark, despite significant investment and infrastructure. This performance raises questions about the effectiveness of native-language training at current scale levels and highlights ongoing debates about European sovereign AI strategies.

Minerva-3B was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, with support from Italy’s national supercomputing consortium CINECA and funded through Italy’s PNRR AI strategy. It was trained on approximately 2.5 trillion tokens, with around 50% Italian content, making it one of the largest efforts to build a European sovereign LLM from scratch.

Despite these efforts, Minerva-3B’s performance on the INVALSI Italian exam benchmark was markedly poor—scoring only 4.9%, which is near chance level. Researchers noted that while dataset composition and size are important, the overall scale of parameters and data may still be insufficient for complex language tasks. This contrasts with earlier expectations that large native-language models would inherently perform well on academic assessments.

Comparatively, the European project AMÁLIA, which extended multilingual models with a small portion of European Portuguese data, has not published its weights but is part of a broader debate on whether continuation training or training from scratch better addresses language-specific needs. Minerva’s results suggest that even large-scale native-language training may not guarantee high performance in complex, academic language tasks.

Implications for European Sovereign AI Investments

The results from Minerva-3B challenge assumptions that large-scale native-language training alone can produce models capable of handling complex academic and language-specific tasks. Despite Italy’s substantial investment and infrastructure, the model’s poor performance indicates that current parameter scales and training approaches may be insufficient. This raises important questions for European AI policy: how much native-language data and what scale of investment are truly needed to develop effective country-specific models? The findings suggest that European sovereign AI efforts may need to reconsider their scale and methodology to achieve meaningful language understanding and knowledge depth, especially for high-stakes applications like education.

European Sovereign LLM Strategies and the Minerva Case

Italy’s Minerva project represents a deliberate choice to build a large, native-language LLM from scratch, leveraging extensive computational resources and a significant dataset. This approach contrasts with models like Portugal’s AMÁLIA, which employs continuation training on multilingual foundations with smaller European-specific datasets. The debate centers on whether starting from scratch or extending multilingual models is more effective for capturing country-specific knowledge.

Prior to Minerva, European efforts have largely focused on multilingual models or smaller native-language adaptations, with mixed results. The Italian project’s large-scale investment aimed to demonstrate that native-language models could reach high performance, but the recent benchmark results complicate this narrative, suggesting that scale alone may not suffice for complex language understanding.

“The Italian approach demonstrates that large-scale native-language training is feasible but not necessarily effective at producing models capable of academic-level performance.”

— Thorsten Meyer

Unresolved Questions About Native-Language Model Scaling

It remains unclear whether increasing the scale of parameters and data will eventually lead to high performance in complex language tasks or if fundamentally different approaches are needed. The exact threshold of data and parameter scale required for effective country-specific models is still unknown, as is how to optimize training methodologies for these objectives.

Next Steps for European Sovereign AI Development

The Minerva team continues to refine their models, with ongoing research into different training regimes and dataset compositions. Future evaluations will likely explore larger models and alternative training strategies to better understand the scale-performance relationship. Policymakers and researchers will need to consider these findings when designing future investments, potentially emphasizing not just scale but also training methodology and data quality.

Key Questions

Why did Minerva-3B perform so poorly on the Italian exam benchmark?

Despite large-scale native-language training, the model’s limited performance suggests that current parameter scales and dataset size may still be insufficient for complex academic tasks.

Does this mean European efforts to build sovereign models are failing?

Not necessarily; it indicates that scale alone may not be enough, and that different approaches or larger investments might be needed to achieve desired performance levels.

What are the implications for future European AI policy?

Policymakers may need to reconsider the scale and methodology of native-language model investments, emphasizing the importance of data quality, parameter scale, and training approaches to meet complex language understanding goals.

Will increasing model size solve the performance issues?

It is uncertain; while larger models may improve results, the current findings suggest that scale alone might not be sufficient without methodological improvements.

Source: ThorstenMeyerAI.com