Onur Solmaz

Introducing *Switch*: a *harness framework* for *decision models* like Jev

I looked for this the other day and was surprised to not find anything like it

You obviously cannot use a decision model like an LLM in a coding agent harness like @pidotdev, because Jev cannot generate tool calls

But what if there was a framework that "generated the tool calls" for it programmatically, and Jev only had to pick out of that finite set of choices

That is what switch does. You pick a "domain" to implement the decision model in, and it provides a core API to define your state, and the functions that will derive decision model questions out of that state. Chess is a domain, as well as being a rubric judge for evals

It basically standardizes what I imagine a lot of people are already vibe-coding, to an abstract interface: calling a decision-model API through multiple steps, i.e. an agent loop over finite choices

Because of the finiteness of the output of a decision model, you cannot have a general pure decision model harness. Instead, you need to implement specific ones that generate the choices programmatically (or have an LLM generate the choices, but I find the former more interesting)

A primary use case I have for this is use as a judge in an eval. For example, At the end of a task run that involves a bunch of documents, most people currently follow this approach: extract text out of PDF/docx/md files, and dump them right into the context

But what if the visual formatting of the output is a judging criterion? Then you have to dump an image of every page in the input state. But what if you have 200 pages, and your question does not need looking at all of them? What if Jev could navigate through the haystack and find the info it needs?

I have noticed that Jev is not so good at chess nor being a Decision-Model-as-a-Judge. It sucks at chess, and it does not perform well enough to replace GPT 6.1 Sol as a judge in the domain I am working for

But switch is not only for Jev! It is for any decision model including the ones supported by the pi-ai package

So once you can generate enough data with a better model, you should be able to pick an open source one and fine-tune it on your use case to perform better! And the model should be trained in a harness like switch's to be especially performant and cost-efficient

I wanted to get this out because I find the idea very exciting, at least for benchmarking. Jev is currently not there, but decision models obviously will be there at some point, and we can take them there! There are also a lot of potential gains from picking a specific domain, and creating a looping agent that does a job for 100x cheaper!

Let me know if you have any ideas, I am quite keen to try this on new domains!

Repo: github.com/osolmaz/switch

man I don't know what the new herdr app did but I am loving the low latency and higher responsiveness to falkenstein from south east asia

I just want them to publish reverse engineered legacy ms windows/office source

It is clear that to have an agent work well with MS Office suite, you have to have these programs work very fast in RL environments

Imagine the agent waiting 10 seconds for MS Word to open while trying to solve a task (let alone have it run on linux)...

So openai and anthropic have most certainly already reverse engineered them or are about to

In an ideal world, they would just publish that so that we don't have to waste time having to do an open source version

Request for Reverse Engineering

We are months into a reverse engineering cambrian explosion, and there is still no github repo I could find that implements a Word DOCX renderer???

Reverse engineering and creating rust libdocx, libxlsx and libpptx using abliterated deepseek or glm is one of the highest services anyone can do to mankind these days

Agents MUST be able to read and write these obnoxious file formats

Here is a starting point, a good samaritan already created a test suite for rendering docx

huggingface.co/datasets/arthr…