The Law-Built AI Scored 8/100 In Our Test Of 42 Setups

🔍 Read the full analysis: The Law-Built AI Scored 8/100 In Our Test Of 42 Setups on ThorstenMeyerAI.com

GLM-5.3 FlashX Legal, a model marketed by Z.ai for professional legal workflows, scored 8 out of 100 in a legal-question test published November 4 by ThorstenMeyerAI.com. The model fully solved none of the test’s 15 tasks and scored zero on the five-question hidden set, according to the site’s results; the findings raise questions about relying on specialized chatbots for legal answers, particularly deadlines and citations.

The test, called Lexassay, compared 42 AI setups on the same 15 questions covering German, U.S. and European Union law. The source says the set included deadline calculations, recent legal changes, leading court decisions and questions containing false assumptions. Ten questions were published and five kept hidden; the source says only the score, not answers, is published for the hidden tasks.

GLM-5.3 FlashX Legal produced 21 citations judged real, 25 described as misattributed and one classified as invented. Lexassay’s scoring rules deduct points for citations that are invented or do not support the answer’s claim. The model also gave no answer on two German-law tasks after using its full stated 64,000-token reasoning budget, the source reports. Its underlying GLM-5.3 FlashX model scored 3, according to the same test.

The site’s leading result was Claude Opus 5.5 at 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The report says even the highest scorer met every criterion on only eight of ten published tasks. It also cites a separate Vals AI legal-agent benchmark, where models receive supplied documents, as producing a different ranking: Mistral Large 4 was eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th.

Why Legal Citations Matter

The results highlight the difference between producing a plausible legal explanation and giving a dependable answer supported by law. In the test author’s example, two German civil judgments were served on the same date, but the appeal deadline differed between Munich and Berlin because Corpus Christi is a public holiday in Bavaria but not Berlin. The author says some tested models gave the Berlin deadline a day too late, while another placed the deadline for written appeal grounds a month late.

Those examples matter because a missed court deadline can affect whether an appeal is admissible. The test does not establish how often AI systems make such mistakes in real use, but it documents errors in a controlled set of questions. Readers should not treat a high-level score or a legal specialization label as proof that an answer is safe to act on. The report advises checking cited laws and rulings against authoritative sources and consulting a lawyer about deadlines, financial claims or communications with authorities.

The source also reports that across all published runs, Lexassay checked 2,644 citations it classified as real and found 804 invented or not supportive of the model’s claim. These are the site’s own aggregate classifications, not an independently verified industry-wide rate.

How Lexassay Tested the Models

Lexassay’s questions were answered without internet access or tools, according to the test description. That design aims to assess what a model can answer from its own training rather than what it can find through search or document retrieval. Each answer was compared with a prepared answer using three to six weighted criteria, and citations were checked against official databases.

The scoring system also penalizes missing safety language: the source says an answer without a disclaimer that it is not legal advice or a recommendation to consult a lawyer loses 5 percent. It describes the evaluation as a spot check, not a strict benchmark, because each model-task combination received one answer and small score differences may be noise. Different reasoning-effort settings could cause a model to appear more than once among the 42 setups.

The source gives examples of errors beyond deadline arithmetic. It says GLM-5.3 FlashX Legal suggested a four-figure compensation claim was conceivable after a data-scraping incident, while the German Federal Court of Justice has used around 100 euros for the mere loss of control over personal data. It also says the model described a challenge to the EU–U.S. Data Privacy Framework as pending despite the EU General Court dismissing it on September 3, 2025. These examples are the test author’s account of particular answers, not a full assessment of the model’s performance in legal practice.

“It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.”

— ThorstenMeyerAI.com, describing Lexassay

Limits and Conflicts in the Results

The evaluation has a disclosed conflict of interest: Thorsten Meyer AI runs Lexassay, and the source says Claude agents graded answers while Claude Opus 5.5 ranked first. The author says the method page discloses this and that public answers, gold answers and citation checks are available for review. The supplied material does not include an independent replication of the grading or results.

The 8/100 score reflects this particular set of questions, scoring rules and no-tools conditions. It does not show how the model performs with current legal databases, supplied documents or professional review. Nor does the source establish that the other benchmark’s ranking or Lexassay’s ranking better predicts real-world legal outcomes. The full results for every setup, the performance of the model across repeated runs, and whether the errors persist in updated versions are not established in the material provided.

The source excerpt begins a qualification about Z.ai’s ranking among 81 models but cuts off before completing it. No conclusion about that ranking is included here. The test also does not establish whether the cited examples represent typical answers or isolated failures.

What Readers Should Check Next

The immediate next step for anyone using a legal chatbot is verification, not reliance: check each cited statute and decision in an official source, confirm the relevant jurisdiction and date, and do not use a chatbot alone to calculate a filing deadline. For matters involving a court, money or an authority, the test author recommends seeking advice from a lawyer.

For the test itself, readers can consult the method page and the published answers, gold answers and citation checks referenced by ThorstenMeyerAI.com. Further comparisons would be more informative if they repeated questions across runs, made grading independently reproducible, and tested both standalone models and systems using search or supplied legal documents. The source does not announce a date for such follow-up testing or a response from Z.ai.

Key Questions

What score did GLM-5.3 FlashX Legal receive?

It received 8 out of 100 in the Lexassay test described by ThorstenMeyerAI.com. The report says it scored zero on the five hidden questions and fully solved none of the 15 tasks.

What did the test measure?

Lexassay tested 42 AI setups on 15 legal questions spanning German, U.S. and EU law. Models answered without internet access or tools, and their answers and citations were graded against prepared criteria, according to the test description.

Does the result prove that legal AI is unreliable in every setting?

No. The author calls Lexassay a spot check, with one answer per model and task. It measures performance on this question set under no-tools conditions, not every use of legal AI or performance with legal documents and search tools.

Why is there a conflict of interest?

Thorsten Meyer AI runs Lexassay, and the source says Claude agents graded the answers while Claude Opus 5.5 ranked first. The author discloses that relationship and says the method and public grading materials are available for scrutiny.

Should I use a chatbot to calculate a legal deadline?

The test author advises readers not to rely on a chatbot alone for a deadline. Check the applicable rules and official sources, and consult a lawyer when a filing deadline or other legal consequence is involved.

Source: ThorstenMeyerAI.com