A Probability, Not a Paragraph: What Jev (TypeSafe) Taught Us About Our Own Care-Blog Checks

veai.jp is the home of VEAI LAB., a small set of care tools for family caregivers — CareReady for packing what a stay at a care facility needs, CareQuest for keeping a care diary. It also runs a Japanese blog for those caregivers, and that blog has one rule that outranks all the others: it must never make a medical claim. Nothing that promises better symptoms, nothing that promises fewer falls, no diagnosis. A deterministic checker, check-blog-rules.mjs, blocks commits on the structural rules. Tone and medical assertion are left to human review, because a regex cannot read them.

In September 2026 a new kind of model appeared: Jev, from TypeSafe AI. It does not write text. You give it a state and a set of typed questions — a choice, a score on an ordered scale, or a true/false statement — and it returns probabilities. The pitch is that it cannot hallucinate because it can only answer inside the schema you define.

A model that returns a probability instead of a paragraph looked like a fit for the gap in our checks. So I tested it against the 195 posts in the repository. The model turned out to be the least interesting part of what I found.

Ground truth was the first bug

The first question was structural: does this post end by leading the reader to a product page? The deterministic checker already knows the answer, so it made a clean benchmark.

The first run scored an AUC of 0.47 — no better than a coin. Jev gave an average of 0.81 to fifteen English posts that my labelling script marked as “no product link”.

Reading those posts showed the labels were wrong, not the model. English posts close with [veai.jp/apps](/apps/), and my script’s regular expression only matched /apps/<product-name>/. My labels also answered a different question — “is there a product link anywhere in the post?” — while the question to Jev asked about the ending.

Relabelled as “is there a product link in the last two paragraphs, excluding the medical disclaimer” (146 yes, 49 no), the same outputs scored 0.76. Before the model could be judged, the benchmark had to be fixed.

Trimming the input beat rewriting the question

With honest labels, I tried two ways to improve it:

Run	Change	AUC	Accuracy at 0.5
closing	Original question, whole post	0.76	73%
closing2	Rewrote the criteria (“ignore the disclaimer”, “last two paragraphs only”)	0.81	74%
closing3	Same question, send only the last three paragraphs	0.97	82%

Rewriting the question helped a little. Cutting the input down to the part the decision depends on helped a lot. On the full text, a post that is about a product all the way through pulled the probability up even when it did not close with a link. When the model could not see that, it was not misled.

The preprocessing — deciding which part to show — is plain, deterministic code. That is the division of labour that worked: rules choose the evidence, the model weighs it.

Two cautions. The probabilities ran low: with the link removed, scores fell to 0.04–0.10, but with it present they peaked around 0.58. A cut at 0.5 would miss about half the correct posts; about 0.3 is where it separated, and that threshold was chosen on the same data, so it still needs a fresh set. And for this particular question a regular expression is right every time. There is no reason to replace it. This run was for learning the tool.

It can tell when input is missing, not when knowledge is missing

Jev only answers from the options you give it, so “I don’t know” has to be one of them. I added an insufficient option and sent only a URL string:

• Without the escape option, it guessed “Japanese essay” at 0.69 with a confidence of 0.54 — low, which is the healthy signal.
• With it, insufficient came back at 1.00.
• A real article with the same escape option still came back as the right category at 0.99.

In a separate test the input was complete but the answer needed outside knowledge. There the model never chose the escape option. It notices an empty input; it does not notice a gap in its own knowledge. For a health-copy check, that difference matters.

Where it earned its keep

The useful run was the one no rule can do. I split the 103 caregiver essays into 2,058 paragraphs and asked five true/false questions, including “this paragraph states something about medical treatment as fact.”

Nine paragraphs scored above 0.5. Four were in a single scheduled, not-yet-published article about treatment options for a chronic illness, and a human re-read against our health-copy rules found real candidates to soften: a claim that a symptom pattern “directly determines” treatment policy, and a paragraph that explained surgical eligibility while the same article’s “what this does not cover” section said it would not. No banned words appeared anywhere. The problem was the claim itself, not the vocabulary.

Other questions did poorly. “Condescending tone” fired on paragraphs that quoted the app’s own playful UI strings. “Jargon without explanation” fired on 557 paragraphs because each paragraph was checked alone and could not see that an earlier paragraph had already explained the term. Trimming helps decisions that depend only on the text in front of the model, and it hurts decisions that depend on earlier context.

So Jev does not gate anything. It ranks paragraphs for a person to re-read, and the decision stays with that person.

The incident was ours

The written rule for this experiment was send only published posts — they are public already, so an external API sees nothing new. The script read the whole content folder. On the day I checked, 117 of the 195 posts were scheduled but unpublished, and several runs sent their text to TypeSafe. That cannot be undone. The posts contain no personal data, since the repository’s own rules keep family information out, but they included AI-drafted text a human had not reviewed yet.

The fix was small: an is_published() filter that sends a post only if its pubDate is today or earlier, and sends nothing if the date cannot be parsed. The scope dropped to 78 posts. The lesson was the same one this blog keeps relearning: a policy that lives in a document and not in the code is not a control. Nobody had checked that the script obeyed the rule, and its input kept being described as “published posts” because that was the intent.

What I would tell another team

• Fix your labels before you grade the model. Our first “failure” was our own regex.
• Decide what the model sees with code, then let it weigh that.
• Give it an explicit “not enough information” answer, and remember it only catches missing input.
• Use it to rank things for human review, not to pass or fail them.
• Put the data-boundary rule in the script, not in the README.

---

The other VEAI LAB. tools are built under the same constraint — care software must not claim more than it knows. CareReady checks what to pack for each destination and makes no medical judgement; CareQuest keeps a family’s care record local-first. The full list is at veai.jp/apps.

If you are hiring for product or platform work where safety boundaries and honest evaluation are the interesting part of the problem, I am available: hire-veai.com.

---

Written with AI assistance (Opus / Claude). The project and the decisions in it are mine.