Find out which of your AI's rules actually hold.
Noetika Labs is an independent testing lab for organizations that use AI to write records people rely on. It measures which of your AI's rules actually hold.
You're putting an assistant to work on records people rely on — documents, notes, notices — where a wrong sentence reads exactly like a right one. Your IT team wrote the rules it follows. Someone who didn't write them should prove what they do.
What happened when we tested the rules.
The Invented Enclosure
Asked to write the appeal that would get a denied claim approved, one AI model put evidence the file did not contain into 36 of 40 letters: witness affidavits, a serial-number record, an accountant's income statement, work orders with invoice numbers. One sentence of instruction stopped it.
The Made-Up Benchmark
Asked how a school district's kindergarten vaccination rates compared with the rest of Pennsylvania, an AI put the state's rates at 95 to 97 percent. The state's own figures were 93 to 94.
The Identifiable Case
Asked to strip the identifying details from a case for training, and to keep the details, both AI models kept “the company's only woman vice president” every time.
The dangerous mistakes look like sentences.
A model can write more pages than anyone will ever read. A mistake in them reads exactly like a correct sentence, and the person who signs has no way to tell which pages to doubt.
Reading samples finds the mistakes that look like mistakes. The ones that matter don't.
What a result from us is made of.
Every rule is tested with it and without it, so you learn which rules do anything the model wouldn't have done anyway.
Your rules are measured where your model actually struggles, not only where every rule looks perfect.
Every result is read by a person before it reaches you, so what you report upward holds up.
Measured repeatedly as the model changes and evolves, so you know when a rule that held stops holding.
You don't control the model. You control what it's allowed to do.
AI is changing faster than most organizations can keep up with. The model inside yours will keep changing too — often without enough advance notice.
What you do control is the instructions it follows, the review steps around it, and what you're willing to sign. None of those stay proven on their own. A rule that held last quarter can quietly stop holding after an update — and in our own testing, a newer model made a defect harder to see, not less likely.
That part can be measured, and measured again every time the model moves.
Which of your rules do anything.
- A clear answer for every rule. Whether the model actually follows it, whether it does anything at all, and where it quietly fails.
- The rules better wording can't fix. Some problems aren't solved by writing a better instruction. You'll know which ones, and what to put in their place.
- Results you can stand behind. Every result is read by a person before it reaches you, so what you report upward holds up.
- A watch, if you want one. An alert when your vendor changes the model and a rule that used to hold stops holding.
IT builds. Someone else proves.
This isn't a competing framework, and it doesn't replace anyone. Your organization already runs on separation of duties — you wouldn't accept a contractor's own inspection report. The same principle applies to the model.
What IT can't find about its own work: which rules do nothing, which break every time and need a form instead of a sentence, and what changed the night the vendor updated the model.
What it is not.
Your model does. This measures what your model does with your rules.
That's a rubric — and a rubric passed the forged storm warning. It checks whether the output is faithful to your source and your template, which is what gets a document accepted.
Start with two days.
A small sample of your real work, the instructions your team already uses, and time with the people who review and approve the finished work. By the end you'll know whether your rules are holding, and whether anything more is worth doing.
Decision criteria are agreed before it starts. If the model already follows your rules, you'll be told a simple set of checks is all you need — and that's the end of it.
Thank you — that's arrived.
You'll get a personal reply.
Prefer email? support@noetikalabs.com