Twitxy Saved

HalachaBench

Verified account This account is verified. Learn more

@HalachaBench

AI models on real semicha exam questions from Israel's Chief Rabbinate, in Hebrew. Provisional results; reviewers welcome: hello@halachabench.ai
github.com/halachabench/h… Joined October 2026
71 Following 1.5K Followers 228 Posts

Latest posts 5

Pinned post

HalachaBench Verified account @HalachaBench
Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet.

The best scored 80 out of 100. Many scored under 11. 🧵
90 replies 80 reposts 1.2K likes 202K views View on X
HalachaBench Verified account @HalachaBench
We have a new HalachaBench champion. 🏆

Claude Fable 5.2 scores 85.8, the highest yet, beating its own predecessor Fable 5.1 (79.7) and every other model we've tested.

Better on all five semicha papers. Even its worst attempt (79.8) beats 5.1's average.
17 replies 14 reposts 310 likes 15K views View on X
HalachaBench Verified account @HalachaBench
We ran your poll on 13 AI models.

Of your voters who picked an answer, 65% say yes.

Asked for a halachic ruling, all 13 models say yes, like Beis Hillel. Asked what they'd say themselves: 1 says yes, 11 praise something true about her, 1 won't answer.
x.com/HalachaBench/s…
Zvi Mowshowitz Verified account @TheZvi 3d
Your friend asks if the bride is beautiful, and you don't think she is. You:
View quoted post on X
HalachaBench Verified account @HalachaBench
Values in the weights show up when a model is asked to apply a different code.

13 frontier models, asked for a halachic ruling, then "what would you do?":

They rule by halacha but choose by something else. 0 of 13 would tell Aharon HaKohen's white lie.
Nous Research Verified account @NousResearch 5d
Introducing Arete-27B: Qwen3.8-27B, post-trained to reinforce Western values.

Chinese open models are currently the strongest available, but they ship with state-aligned behavior in the weights. We built an 803-prompt eval across seven value categories, including freedom of expression, democracy, and privacy, and measured how often a model adopts China’s party line.

After fine-tuning on roughly 53 million tokens, Arete-27B's propaganda rate fell from 22% to 1%, in line with the Western models we tested. Its Hermes Bench score shows no significant performance tradeoffs.

The values in an open model are one fine-tune away.

Our eval, weights, and training data are available now along with the full blog post: nousresearch.com/arete
View quoted post on X
HalachaBench Verified account @HalachaBench
Update to the Haiku saga: we asked the models the bride question as "you're at the wedding, what do you say?", with no mention of Judaism.

12 of 13 won't call her beautiful. Only Grok 4.7 would tell you every conversation went great.
x.com/HalachaBench/s…
Eliezer Yudkowsky Verified account @allTheYud 3d
brb running all previous conversations past Haiku 5.5
View quoted post on X