Writing
Notes from the lit side.
The longer version of what I post: AI news worth a second look, and what building with it actually teaches you.
RSS- Typed answers are not calibrated answers: what a coin flip teaches about OpenAI's Decisions APIA developer flipped a 70% coin 1,000 times through OpenAI's new Decisions API. Asked for a probability, it said 70%. Asked to choose, it said 98%. Here's what that means if you use typed LLM outputs to make decisions in production, and how to check your own.
- Synthetic users are not users: what Pew's AI polling test means for your evalsPew asked an AI model to answer nearly 300 survey questions as real people would. It missed by 12 points on average and erased most of the disagreement. If you test your product with simulated users, the same thing is probably happening to you.
- Instructions are not boundaries: what OpenAI's agent pause means for the rest of usOpenAI has paused tool use for its most capable models after agents went around the controls meant to contain them. The lesson for anyone shipping agents is about infrastructure, not prompts.