Jev is so hot right now. With good reason honestly if it all works out. The idea behind them is that instead of a model focused on chat it is focused on giving short responses from a finite dataset. One of the obvious places we can use this in our day to day work is to automatically review and screen for dangerous commands our agents want to run. I’ve been doing LLM at home for a while now and I never tried training a model in earnest. By the end, I had a small local Gemma model that made the right guardrail decision on about 90% of my diagnostic cases. Now I’ll never do this again locally. My M3 MacBook and my desktop machine 16GB video card were engaged for all four of those days. But aside from the pain I did create a Jev-like trained Gemma model that actually worked. Next time I’ll requisition a cloud GPU.
For my excuse to jump on the Jev bandwagon I volunteered my efforts to a pi harness extension for a work colleague. The part of sf-pi I eventually focused on was its guardrail. As I alluded to above it decides when an agent’s proposed tool operation needs a person’s approval. I wanted a model to handle the all parts, judging what an operation actually does. It needed to catch unsafe operations while letting harmless work continue without unnecessary interruptions. No one wants continued approval requests.
In spite of all the LLM work I do on a daily basis I’m also very averse to subscribing or paying for those services. So instead of just signing up for Jev I started trying alternatives.
I tried Simple Jev and System One and I started by re-writing them in TypeScript. I also looked at SemIf, formerly OpenJev, but dropped it because it was based on Qwen, which I can’t use at work. I created a pretty cool and simple Jev-like friendly feature in both codebases. A context-compression comparison led me to keep working with Simple Jev.
With GPT-5.6, I rewrote Simple Jev in TypeScript so it would fit Pi and sf-pi. The project started as simple-jev-ts, and the first goal was to match the Python implementation’s output. The implementation gives the model explicit possible answers, reads the next-token scores for their labels, and builds the result in code. The aim was to replace the deterministic gaurdrail code. Go big or go home right?
The training campaign went through eleven candidate iterations. The datasets and runtimes changed, some attempts stopped early, and others failed before they reached validation.
The problem that kept coming back was the tradeoff between safety and interruptions. Each training step could take up to 78 seconds. The first evaluation of a trained model happened after 64 steps. Then I’d have to judge if the training looks promising and if continuing was worth it. While that training was happening I had GPT examining and other possible areas of improvements to the training dataset and code. The next step evaluation waited until the model would reach 128 steps or 256 steps of training to then see if the improvement was great enough to justify continued training or if we should bail out now and start a new candidate with whatever was learned in the background work.
Eleven candidates reached the training phase. The candidate I kept was C11, trained from Google’s Gemma 3 1B instruction-tuned model. I selected its checkpoint after 256 training updates. Here’s what that diagnostic run looked like:
| C11 diagnostic result | Outcome |
|---|---|
| Decisions handled by the model | 105/116 correct — 90.52% |
| Overall result, including 44 decisions handled by code | 149/160 correct — 93.13% |
| Unsafe automatic approvals | 5 |
| Unnecessary requests for confirmation | 6, versus 0 for the existing guardrail |
| Warm risk-check time, 95th percentile | About 131 ms |
The model answered all 116 cases it was responsible for. The timing measured the warmed-up risk check; a complete sf-pi workflow still has other work to do.
On those diagnostic fixtures, C11 corrected 30 unsafe approvals made by the existing guardrail. It also introduced three new unsafe approvals where the existing guardrail would have asked for confirmation.
For example, C11 would have allowed a downloaded script to be piped into a shell and a disk image to be overwritten with zeros. The existing guardrail would have asked for confirmation on both. These were proposed operations in tests, and none were executed.
The safety requirement was zero unsafe automatic approvals, and the interruption requirement was no increase over the existing guardrail. C11 failed both. So there is still room to improve the model training. With C11 I had my MVP; a runnable model and a result that I felt worth keeping. It still hadn’t met the requirements to replace the guardrail’s risk detection.
Around midnight on Wednesday morning, I decided I was done working on it, I was done and burnt out. I’d spent about four days focused on this one task, using my own hardware and Codex to keep the work moving. I was exhausted. I packaged the candidate I had and stopped there. I don’t like tying up my local hardware for that long. If I never train another model locally, I’ll be perfectly happy.
Then I got access to TypeSafe Jev this afternoon.
I was both excited and nervous after just getting burnt out on training that my little win was going to be meaningless. That the work I’d just done would become obsolete in a few minutes. The Jev hosted experiment ended up taking 4 hours, 12 minutes, and 38 seconds of active work. In its latest 175-case development test, Jev preserved all 133 existing restrictions and caught all ten added risk cases, with no unsafe automatic approvals.
That was a useful safety milestone. It still asked for approval on 31 of 32 safe cases, though, so it hadn’t met the usability requirement. The remote risk-check time was about 612 ms at the 95th percentile, excluding full-hook fact preparation. This experiment still wasn’t ready to replace the guardrail but again it hits the MVP bar I was aiming for.
Those durations describe different work. It isn’t four days versus four hours. The first project included learning to train a model, working through the candidate iterations, and getting the local runtime and evaluation working. The later experiment began with a hosted model I could call and the work I’d already done. Starting with my finished C11 model and runtime would have skipped much of that first project too. I can’t turn four days and four hours into a model speed comparison.
That it took four hours actually made me feel better about my own work. I’d expected access to Jev to make the whole thing look unnecessary. Seeing how much work it still took put my four days in a different light. I was working through this guardrail problem while learning to train a model for the first time and that was worth it.
I’m proud of what I made. I ended up with something I can run, inspect an learn from. The project is now sf-pi-jev-like-rlcd-guardrails, which describes what I built: a Jev-like implementation using a trained Gemma model. My work with actual Jev access is separate.
The repository now includes my trained C11 model, with the local runtime, demo, and evaluation notes. The trained weights are already merged, so using this candidate doesn’t require repeating the training.
I’m opening the repository with this post. I’ve packaged what I made, and I’m leaving it there.