Farm Fresh Human Data, back to the market

Rubrics and graded answers for RL

For verifiable tasks, where a script can check the final answer, we write the reference answer. For open-ended tasks, an expert writes a rubric for that task alone: what a full-credit answer must include and how many points each part earns.

We can also grade your model's responses against the rubric.

Get your free sample

What you can order

  • Rubrics written for your tasks
  • Answers graded against a rubric
  • Tasks with reference answers

Who grows it

Experts in each task's field write the rubrics and do the grading.

What to tell us on the order slip

  • The tasks, or a few examples of the kind you have in mind.
  • Whether answers can be checked automatically or need a rubric.
  • Pass or fail, or points per criterion.
  • How many tasks and graded answers, and your deadline.

If you will train against a model-based grader, order graded answers along with the rubrics, and check its scores against the expert's on the same responses before training.

How to get a free sample

  1. Send us an order slip.
  2. Agree the spec with a farmer.
  3. Experts grow a free pilot batch to your spec.
  4. Run it through your evals, then order by the crate.

Get your free sample

Also at the stall