Pinned post
When you ask an agent to book a restaurant, spelling out every detail is tedious. You’d rather just say “Get me a table at my usual place tomorrow.”
A good personal agent should infer which restaurant from your history, and ask you for what only you know (e.g., the time and party size). It shouldn’t guess, and it shouldn’t quiz you about things it should already know.
Introducing iPrefBench, my internship work at Apple 🍎: a user-centric benchmark for personalized mobile use agents, with:
- 130 underspecified tasks across 26 iOS apps, grounded in a coherent persona with 3,051 past activity events
- Every missing detail is either inferable from history or obtainable only by asking the user, by construction
- Evaluation on both whether agents get the task right for the user and whether they only ask questions when necessary
🧵:
A good personal agent should infer which restaurant from your history, and ask you for what only you know (e.g., the time and party size). It shouldn’t guess, and it shouldn’t quiz you about things it should already know.
Introducing iPrefBench, my internship work at Apple 🍎: a user-centric benchmark for personalized mobile use agents, with:
- 130 underspecified tasks across 26 iOS apps, grounded in a coherent persona with 3,051 past activity events
- Every missing detail is either inferable from history or obtainable only by asking the user, by construction
- Evaluation on both whether agents get the task right for the user and whether they only ask questions when necessary
🧵:
