In May 2026 the software company Onyx published an unusual dataset on arXiv: slightly over half a million documents from a company that does not exist. Internal messages, email, CRM records, meeting transcripts, support tickets, wiki pages. Along with it came 500 questions an employee would ask on an ordinary working day, and a program that checks, answer by answer, whether the system under test got it right.
It is called EnterpriseRAG-Bench. Two details made me look twice. First: Onyx sells a product of the same kind and states, in its own repository, that it excludes itself from the leaderboard to avoid a conflict of interest. Second: anyone who wants in has to prove the result is reproducible — open-source systems hand over a walkthrough, closed ones hand over access to the live system so Onyx can re-ask the questions itself.
In August we submitted our document search to EnterpriseRAG-Bench. The result was 71.93 points, above the evaluated configurations of OpenAI File Search (61.03), Amazon Q (48.96), Azure AI Search (48.42), Vertex AI Search (41.87) and NVIDIA AI Blueprints (37.73). Skyller entered in 2nd place on 12 August 2026 and appeared in 4th among 22 systems on 6 September, with the same score. The comparison applies to the tested configurations and corpus; it does not rank every product on the market.
It is worth saying where this submission comes from. The platform did not start in an AI lab: it started inside an infrastructure and support operation that did the same routines by hand across client after client — creating users, resetting passwords, checking backups, reading monitoring dashboards, working in the hosting panel — and where what each environment held lived in the heads of whoever had been around longest. Finding the right document was not one more feature on a list: it was the problem. Hence the interest in measuring exactly that, in a test that is not ours.
What a test like this actually does
The temptation, when someone talks about evaluating artificial intelligence, is to picture an exam: maths, logic, essay writing. It is nothing like that. The corpus is built to look like a real company, defects included: files stored in the wrong place, old versions living next to new ones, near-duplicates spread across different systems, and documents that openly contradict each other.
The 500 questions are split into ten types. There is the easy one, answered by a single document. There is the one nobody would phrase using the words in the file. There is the one that requires stitching together distant parts of a long document. There is the one you can only answer by gathering everything that exists about a project. And there is a group of twenty questions whose answer simply is not in the corpus — the assistant has to say it did not find anything, instead of inventing something plausible.
That last group is the one I value most, and the rarest to see measured. Every company that has put AI to work on internal documents knows the problem: the answer that is well written, confident — and wrong. A test that scores "it isn't here" is measuring reliability, not eloquence.
The grading program does not belong to the vendor being assessed. It compares the answer with an expected answer and with a list of facts that have to show up. Then it looks at the documents the system returned and decides, by a vote across three independent evaluations, which ones were genuinely necessary.
Why almost every quality figure is useless
In corporate software, quality figures tend to be born inside the house of whoever is selling. The vendor picks the question set, picks the pass criterion, runs it in its own environment and publishes the result in a sales deck. There is no way for the customer to verify any of it — and, to be fair, no way for the vendor to prove it did not pick the test it was going to do well in.
I have seen comparisons where the competitor was configured by the party doing the comparing. I have seen accuracy percentages with no denominator: accurate on how many questions, asked by whom, over which corpus. And I have seen a lot of numbers with no date, which is an elegant way of never going out of date.
The Onyx test solves that simply: it publishes everything. The corpus, the questions, the expected answers, the grading program and the leaderboard with its history. Anyone who doubts our result can download the material and redo the arithmetic. Anyone who doubts a competitor's result can too.
What has to be in place
If you are evaluating AI tools for your company, these are the mechanisms that separate evidence from advertising. None of them requires technical vocabulary.
A public dataset. If the test runs on documents only the vendor has, the result is not verifiable. Ask where the data lives and who else can use it.
A grader that does not belong to the party being assessed. The program that decides whether an answer is right has to be the same for every participant, and preferably written by a third party.
Reproducibility required at the door. A serious leaderboard does not take a file of answers on trust. It asks for the route to redo the test — in the code, if it is open; in the live system, if it is closed.
A date next to the number. A leaderboard is a snapshot of one day. An August position is not valid in December, and repeating an old position without its date is, in practice, saying something false.
The whole score, not the flattering slice. An overall average hides what matters. The breakdown by question type shows where a tool is strong and where it will leave your team stranded.
The test running on the product you will actually use. There is an enormous gap between a lab result and how the live product behaves. Our submission went through the same public endpoints customers use every day. Onyx did not settle for the file: it sent 80 questions we had never seen and checked the answers before publishing.
What the result showed
On the twenty cases where the information did not exist in the corpus, we got all twenty right: the expected answer was to say we found nothing. This result is limited to the evaluated category and does not guarantee error-free answers to other questions. Questions requiring distant passages of a long document scored 83.33; informal, badly organised documents scored 85.00.
There are two types where we still have room to grow: gathering every relevant document on a subject without leaving any out (32.08) and aggregating documents from an entire project (50.89). Those are the fronts the team has been working on since publication.
Both numbers are published in the same file as the other results. The record lets readers examine the strengths and limitations of the evaluated configuration. New versions and other environments need new measurements; this submission remains dated evidence.
One last observation about reading leaderboards. Our score did not move a decimal between August and September; the position moved because three new systems were published, two of them above us. That is how an open leaderboard normally works, and it is why I distrust any sales material that shows a ranking without saying when it is from.
Three questions to take to your next meeting
- Who calculated that figure, and where is the data? If the answer is "our internal lab", treat it as a vendor estimate, not a measurement. Ask for the public address.
- What is the date, and how many systems were on the list? A position on its own tells you nothing. Fourth out of 22 and fourth out of 5 are completely different results.
- Where did the tool do badly? A vendor that only shows the good scores chose the slice. Ask for the breakdown by question type and read the bottom row first.
A public test does not replace the experience of working with your own files, in your own language, with your team's questions — and that part is free, and takes an afternoon. What the leaderboard does is something else: it separates those who accept being measured by a third party from those who prefer to measure themselves.
