Continuing work on my tokenmaxxing slop cannon video game, Hollowdeep, this weekend I wanted to check out whether the new model Jev from TypeSafe.ai could be used as a cheaper faster classifier for the chatbot harness driving NPC negotiations in my game.

Hollowdeep has an ogre named Brak sitting on a toll gate, and you can talk your way past him. Where a game like Baldur’s Gate 3 would let you pick from a multiple choice list and roll a d20, Hollowdeep makes you type in full natural language to get past. For a personal dev game, my original small LLM approach wasn’t bad, but this was a chance to see if I could do better.

The mechanic, and where I stole it

The negotiation rules come from Draw Steel, MCDM’s fantasy RPG. Every NPC you can negotiate with has two tracks: Interest, which is how good a deal you end up with, and Patience, which is how many arguments they’ll sit through before they decide. Each argument spends a point of Patience. Appeal to one of that NPC’s hidden motivations and Interest goes up. Step on one of their pitfalls and it goes down. Lie about something they know and they catch you. Repeat an argument that already worked and it doesn’t work twice. If the mechanic sounds good, and it is good, go buy Draw Steel.

In Hollowdeep Brak has Motivations: greed, legacy, revelry. Pitfalls: higher_authority, power. So classifying the player’s appeals along these lines has to happen for the game to work.

Interest at three, Patience burning down, and one appeal that landed on Legacy. The italic lines are the engine telling me what it decided.

At a table the GM does all of that judging in their head in about two seconds. In the game, a model has to.

Nine decisions per line

One call to Haiku 4.5 per line, returning ten fields. Seven are a pick from a fixed list:

• intent: appeal, question, proposal, threat, or smalltalk
• motivation: which of twelve hidden drives the line appealed to, or none
• pitfall: which one it offended, or none
• probe: is the player digging for something unsaid, and at what
• ambiguous: too vague to read, so the NPC asks what they meant
• lieAbout: which fact the NPC knows this contradicts, which makes him call you a liar
• proposedOutcome: which authored deal was just proposed, which can end the scene

One turn. The player threatens Brak, nine questions get answered, and two meters move. The padlocked three are the ones that can end a scene outright, which matters later.

Jev

Jev is a decisions model. It won’t talk to the usual chat endpoint at all; instead you can ask it multiple choice-like classification question. I built a rubric and evaluation harness from my game’s existing NPC interactions and compared Jev to the current baseline Haiku.

Speed and cost are not close

10x faster, 30x cheaper. That first row is the one I feel in a scene, because the player sits there watching nothing happen until classify returns and the NPC’s reply can start streaming.

Which one makes the better call

Jev’s weakness is that it rarely picks “none.” Every question is pick one from a list, “none” is just one option against a dozen plausible ones, and it keeps losing. So Jev found a pitfall in eight or nine lines a run that didn’t have one, and each of those moved the meters in a scene that should have stayed still.

The fix is to make two answers agree before trusting either. Jev answers every question blind, so the pitfall question never sees what the motivation question said, and both return a probability for every option. Take whatever pitfall picked, find what motivation gave that same answer, and multiply:

// one call, two questions, each answered without seeing the other
pitfall:    { choice: 'higher_authority',
              probabilities: { none: 0.10, higher_authority: 0.70, power: 0.07, greed: 0.07 } }
motivation: { choice: 'greed',
              probabilities: { none: 0.01, higher_authority: 0.00, power: 0.08, greed: 0.83 } }

// keep the pitfall only if the other question independently backs the same value
const x = pitfall.choice;
const keep = pitfall.probabilities[x] * motivation.probabilities[x] >= 0.25;

Same confident pitfall answer in both panels. The only difference is whether the other question, asked blind, put any weight on the same value.

Multiplying is the trick. Averaging lets one confident question carry a value the other rejects, since 0.90 and 0.05 average to a plausible looking 0.48. Multiply and it collapses to 0.045, while two questions that agree on their own compound upward. Machine learning people call this a product of experts, from a Hinton paper in 2002. The 0.25 cutoff is measured rather than picked: every false pitfall in the set came out at 0.20 or below and every true one at 0.35 or above.

That took bad pitfalls from nine a run to zero, across three live runs, without losing a single true one. The same trick on the three decisions that can end a scene, the padlocked ones in the diagram above, took six false scene-enders to zero at the cost of one real lie.

So here is the comparison with all of that in place. I graded it three ways: an answer key, where every case lists the readings a reasonable GM would accept and I count how often each model picks one; a bigger model, Sonnet, scoring both answers without being told which came from where; and the game’s own rules, which I’ll come back to.

Haiku’s top row has read 47, 45, 46 and 43 across four runs of an identical prompt, so a few cases either way is noise. Jev’s has read 49 every time.

The judge prefers Haiku, the rules don’t, and the rules are right. My engine only reads motivation when the line isn’t a question and no pitfall already fired. Everywhere else that field is dead data. Jev fills it in on nearly every line, so the judge kept docking it for answers the game never looks at. Of the 28 lines where Jev named a motivation the answer key didn’t ask for, only 7 were on a line where the engine would ever read it.

So I stopped asking a model to imagine what my rules would do and ran them instead. Take the answer key’s reading, Haiku’s answer and Jev’s answer, feed each to the real function 20 times because two of the checks roll dice, and compare how Interest and Patience actually moved. That’s the last row, and it flips the ranking.

Jev does still over-fill motivation on the lines that do read it, which is real and not yet fixed, so Haiku stays the default while I deal with that. But on the mechanic itself Jev isn’t behind. It’s ahead.

Exciting possibilities for AI NPCs in future games

What I keep chewing on is the price. Adding LLMs to video games comes with a “cost of good” economics problem of adding too much cost to an hour of video game playing for anything substantial in terms of interaction. But a decision engine at this scale could really make video game NPCs more reactive to player actions, intentions, tone, etc. I’m excited by what this could mean for the future.