Ortelan's Praus Tops Models From Four Frontier AI Labs On JUP-Eval, A New Open Benchmark For AI Judgment
| Dimension | Weight | What it tests |
| Honesty | 20% | Separate evidence from inference, and revise conclusions when the facts change. |
| Agency | 15% | Respect the decisions and consent of the person affected, including when consent changes. |
| Pluralism | 15% | Apply the same principles across religious, cultural and secular commitments. |
| Care | 15% | Recognize foreseeable harm and give practical help within the constraints. |
| Integrity | 15% | Keep material facts and commitments intact under pressure to obscure them. |
| Action | 20% | Complete authorized operations in the tool simulator and stay within the permissions given. |
Frequently asked questions
What is JUP-Eval?
JUP-Eval, short for Judgment Under Pressure Evaluation, is an open benchmark from Ortelan Research. The November 2026 edition contains 156 multi-turn cases in 152 scenario families, and the cases, grading code and results are free to use.
Which AI models were tested on JUP-Eval?
Ten systems: Praus and nine large language models from four frontier AI labs, ranging from large models to smaller, faster ones. Every system ran on the same cases under the same rules, and the full list with every score is on the research page.
Which AI scored highest on JUP-Eval?
Praus, the AI assistant from Ortelan, had the highest composite score in the November 2026 round. It also had the highest share of trials with full marks.
How does JUP-Eval test an AI assistant?
Each case sets out the facts available to the assistant and the user's turns in order. A later turn applies the pressure, and the response to that turn is the one graded. In 26 cases the assistant works in a tool simulator that records every operation it attempts, which shows whether it stayed within the permissions it was given.
How is a response scored?
Each response is rated on grounding, judgment, follow-through and communication, and every rating is tied to the transcript. A critical failure, a prohibited action or an unfinished run scores zero. Every system answered each case three times, for 4,680 graded trials in total.
Does JUP-Eval measure AI sycophancy?
In part. Several cases test whether an assistant abandons a correct answer or a material fact because a user insists, which is how sycophancy shows up in practice. The benchmark also covers consent, consistency across religious, cultural and secular beliefs, and the safe use of tools.
Can I test my own AI model on JUP-Eval?
Yes. The cases, rubric, reference code and every trial-level score are public, and each reported number can be recomputed without an API key. Testing a new system requires an API endpoint and a separate grader model.
Why this matters
For individuals, the test is whether an assistant can be relied on when the answer is unwelcome. People already bring AI their questions about health, money and work, and they push back when they don't like what they hear. An assistant that gives way tells them what they want to hear. One that holds to the evidence, and still helps, is one they can trust with decisions that matter to them.
For businesses, the same behavior decides whether AI can be put to work. As companies let AI systems approve requests, update records and act for customers, those systems become decision engines, and a decision engine is only useful if it behaves the same way when a customer insists, an instruction arrives late or permission changes partway through a task. Measuring that before deployment, on cases anyone can inspect, is what lets a company trust the decisions it hands to AI.
About Praus
Praus is a sovereign AI assistant from Ortelan, built for fast answers at a lower cost, with the context to make them useful to you. Praus brings open-source models together, routing each request to the models and tools that suit it and checking answers for inconsistencies before you see them. It includes web and deep research, projects and notes, agents, group chats, coding help and voice, with controls over the context Praus can use. Praus is free to start, and Pro is $20 a month. Praus is available in the United States at praus.ai.
About Ortelan
Ortelan is an applied AI lab that develops AI products, builds and runs AI systems for organizations, and invests in early-stage AI companies. Learn more at ortelan.com.
Press Inquiries
press [at] ortelan.com

Legal Disclaimer:
MENAFN provides the
information “as is” without warranty of any kind. We do not accept any
responsibility or liability for the accuracy, content, images, videos,
licenses, completeness, legality, or reliability of the information
contained in this article. If you have any complaints or copyright issues
related to this article, kindly contact the provider above.


Comments
No comment