AI Evaluation & Data Engineering Specialist

About the role

We are seeking an AI Evaluation & Data Engineering Specialist to design, curate, and operationalize datasets and evaluation frameworks that assess the performance of AI products, particularly large language models. The role combines data engineering expertise with a deep understanding of AI model evaluation.

Key responsibilities

• Build and maintain evaluation datasets for AI models across multiple programming languages (Python, Golang, JavaScript, Java).
• Develop and apply data labeling and scoring guidelines based on Google’s evaluation framework.
• Implement LLM‑judge calibration workflows to align automated and human evaluations.
• Perform error analysis, drift detection, and regression testing of AI model outputs.
• Collaborate with automation engineers to integrate datasets into evaluation pipelines.
• Support rater training, inter‑rater reliability checks, and dataset validation reviews.
• Manage data quality assurance and documentation for contributions to Google‑maintained repositories.

Required profile

• 4–8 years of experience in AI/ML data operations, evaluation, or data engineering.
• Proficiency in Python for dataset manipulation, analysis, and scripting.
• Experience with LLM evaluation, prompt engineering, or text‑generation quality assessment.
• Familiarity with Gemini CLI, Vertex AI, or LangChain evaluation tools.
• Strong understanding of data curation, annotation workflows, and labeling quality metrics.
• Hands‑on experience with Git‑based repositories and CI/CD data workflows.
• Excellent analytical and problem‑solving skills with attention to detail.

Required skills

• Python
• Gemini CLI
• Vertex AI
• LangChain
• Git
• CI/CD pipelines