🔍 Read the full analysis: The Law Model Scored 8/100: Results From 42 AI Setups on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Lexassay, a legal AI test run by Thorsten Meyer AI, scored 42 AI setups on 15 questions covering German, US and EU law. GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the tasks; the test operator disclosed that Claude agents graded the answers and that Claude ranked first.
GLM-5.3 FlashX Legal scored 8 out of 100 in a legal AI test covering 42 setups and 15 questions, according to results published November 4 by Thorsten Meyer AI. The model, which its maker Z.ai describes as optimized for professional legal workflows, fully solved none of the tasks; it provided no answer on two.
The test, called Lexassay, asks models the same questions about German, US and EU law. They cover deadlines, changes to law, court decisions and questions based on false premises. Models answer without internet access or tools, and answers are graded against written criteria. The operator says each task has three to six weighted criteria and every citation is checked against official databases.
Across the published runs, the citation review counted 2,644 citations as real and found 804 invented or misattributed. A misattributed citation refers to a real law or ruling that does not support the claim made; an invented citation does not exist. Under Lexassay’s rules, either type reduces a task’s score. The operator also deducts points when an answer omits a legal-advice disclaimer or referral to a lawyer.
Among the reported leading systems, Claude Opus 5.5 scored 79, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leader met every criterion on only eight of the ten publicly available questions, Meyer reported. Mistral Large 4 scored 10; the test account says it got about half the substance right but lost many points over citation errors.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Why Legal Citations Matter
The results illustrate a practical risk for people using chatbots to interpret legal issues: a fluent answer can still misstate the law, cite a decision inaccurately or miss a deadline. In the German civil-procedure example described by Meyer, two appeals with judgments served on the same date had different filing deadlines because Corpus Christi was a public holiday in Bavaria but not Berlin. The example shows how location and calendar details can change an answer that may appear straightforward.
Meyer says an appeal filed late in German civil procedure can be rejected as inadmissible. That consequence makes unsupported confidence particularly risky for users handling court deadlines, claims or official correspondence. The test does not show that every legal AI answer is wrong, nor does it establish how a model would perform with legal research tools. It does show that, under this test’s conditions, citation accuracy and incomplete answers materially affected scores.
The findings are a warning about reliance, not a substitute for an assessment by a lawyer. The test operator recommends treating AI as a starting point and checking cited statutes and rulings against the original sources. Legal rules and deadlines can depend on jurisdiction and facts, and errors can carry financial or procedural consequences.
Products Worth Considering
Ledger Nano Gen5 Crypto Wallet, Black
LG UltraGear 32GX850A-B 32-inch 4K UHD Glossy OLED Gaming Monitor
Trezor Safe 7 Crypto Hardware Wallet
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay includes 10 published and five hidden tasks. The published answers, gold answers and citation checks are made available, while only scores are released for the hidden questions. Meyer says the hidden set is intended to make it harder for future systems to learn the test answers from material posted online.
Different effort settings are tested separately, so one underlying model can appear in more than one setup. The test is a spot check rather than a formal benchmark, according to its operator: each model answers each task once, meaning small score differences may reflect variation rather than a stable ranking. The answers are graded blind by several Claude agents, but Meyer discloses that his organization runs Lexassay and that a Claude model ranked first.
Another assessment gives a different ordering. Meyer cites Vals AI’s legal-agent benchmark, where models work with supplied documents: Mistral Large 4 ranked eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. The differing results reflect different test conditions and should not be read as a direct comparison of general legal ability.
“optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution”
— Z.ai, in developer documentation as quoted by Thorsten Meyer AI
Limits of the Published Scores
The available material does not establish how well the tested models perform across the full range of legal work, with web searches, legal databases, supplied documents or human review. Lexassay uses one answer per model and task, and its operator warns that small score differences may be noise. The result is therefore not a measure of every product configuration or a guarantee of how a system will handle an individual case.
The test also has a disclosed conflict: Thorsten Meyer AI runs Lexassay, and Claude agents graded answers while a Claude model ranked first. Meyer says the method and public answers are available for scrutiny, but the supplied material does not describe an independent replication of the scores. The hidden questions also cannot be checked by readers from the published answers alone.
The source material gives a score of 8 for GLM-5.3 FlashX Legal and says its hidden-set score was zero, but the results should not be taken as a definitive ranking of all legal AI systems. The exact performance users would see with different prompts, tools or updated model versions remains unclear.
What Readers Should Check
The next step for readers is not to treat a test score as legal advice. For any AI-generated legal answer, check each cited law and court ruling against an authoritative source, and verify that the rule applies to the relevant jurisdiction and date. Do not rely on a chatbot alone for a filing deadline; consult a qualified lawyer when a deadline, financial claim or communication with an authority is involved.
Meyer says Lexassay publishes its method, public answers, gold answers and citation checks. Further scrutiny of those materials, along with future test results, may clarify whether the reported gaps persist across new models and repeated runs. The source does not give a date for another assessment or report a response from Z.ai to the findings.
The article describes an evaluation of AI systems, not legal advice. The scores apply to the stated questions and test setup; readers should not use them to decide a legal claim or miss a deadline.
Key Questions
What did GLM-5.3 FlashX Legal score?
It scored 8 out of 100 on Lexassay, according to the results published by Thorsten Meyer AI. The report says it fully solved none of the 15 tasks and gave no answer on two.
What did the test measure?
Lexassay asked AI setups 15 questions on German, US and EU law, including deadlines, legal changes, court decisions and false-premise questions. Models answered without internet access or tools, and graders evaluated answers against written criteria and checked citations.
Can the scores be treated as a definitive ranking?
No. The operator calls Lexassay a spot check and says each model answered each task once, so small differences may be noise. Another cited benchmark produced a different ordering under different conditions.
What conflict of interest did the test disclose?
Thorsten Meyer AI runs Lexassay, and Claude agents graded the answers; a Claude model ranked first. Meyer disclosed this and said the method, public answers, gold answers and citation checks are published. The supplied material does not report an independent replication.
Should I use a chatbot to calculate a legal deadline?
The test operator says not to rely on a chatbot alone for a deadline. Check the applicable rules and dates with authoritative sources, and ask a lawyer when a deadline or other legal consequence is involved.
Source: ThorstenMeyerAI.com
