Research Scientist - LLM Evaluation & Benchmarking (Remote)

About the role

Anyone AI is hiring a Research Scientist for its internal team, reporting to the CEO, working remotely for the US and LATAM markets. The role focuses on LLM evaluation and benchmarking: designing benchmarks that measure frontier model capability, building evaluation packages across reasoning, coding, agents, tool use and multi-modal tasks, and defending the methodology under lab scrutiny.

Key responsibilities

• Evaluation research: turn eval targets into original benchmark designs and tackle hard measurement questions such as construct validity, item discrimination, headroom, reliability, contamination and capability elicitation.
• Benchmark development: build evaluation packages with subject-matter experts, expert-verified ground truth, multi-model results and rigorous QC (calibration layers, severity-weighted rubrics, deterministic verifiers).
• Experts: recruit, calibrate and review a pool of experts in coding, agentic/tool-use and STEM/reasoning, acting as the final arbiter of correctness and frontier difficulty.
• Laboratory relations: act as technical point of contact for labs, understanding what they want to measure and translating it into an evaluation design.
• Delivery and dissemination: turn lab requests into sample packages, own pilots end to end, and help turn generalizable work into public benchmarks and papers (e.g. NeurIPS, ICLR, ACL).

Required profile

• Research background in ML evaluation or benchmarking, with published or open benchmarks, or equivalent hands-on work trusted by labs.
• Deep expertise in LLM and frontier-model benchmarking, with strong code-model and agentic evaluation.
• Fluency with the measurement problem: construct validity, psychometrics, rubrics, pass rates, headroom, contamination and task discrimination.
• Interest in the safety side of evaluation, capability elicitation, robustness and hard-to-measure capabilities.
• Proven ability to hold a team or expert pool to a rigorous standard.
• Comfort with the full research loop: framing the question, running the study and writing it up.
• Fluent English; Spanish is a plus.