Farm Fresh Human Data, back to the market

Evals and held-out test sets

We write every task from scratch for the capability you want to track, and each comes with a reference answer and scoring notes.

None of it comes from the web or a public benchmark, so it is not in any existing model's training data.

Get your free sample

What you can order

  • Eval sets built to your spec
  • Held-out test sets
  • Your model's answers, graded by an expert

Who grows it

The experts who write each task also write its reference answer and scoring notes.

What to tell us on the order slip

  • The capability you want to measure.
  • The answer format, and how you plan to score it.
  • How many tasks, and how hard.
  • When you need the set, and whether you will rerun it on new models.

Say on the order slip whether any of the set may later go into training, and decide which tasks stay held out before delivery.

How to get a free sample

  1. Send us an order slip.
  2. Agree the spec with a farmer.
  3. Experts grow a free pilot batch to your spec.
  4. Run it through your evals, then order by the crate.

Get your free sample

Also at the stall