We are seeking a Data Scientist II to join our Quick Data team, focusing on evaluation and benchmarking data development for Quick Suite features. Our mission is to engineer high-quality datasets that are essential to the success of Amazon Quick Suite. From human evaluations and Responsible AI safeguards to Retrieval-Augmented Generation and beyond, our work ensures that Generative AI is enterprise-ready, safe, and effective for users at scale. As part of our diverse team—including data scientists, engineers, language engineers, linguists, and program managers—you will collaborate closely with science, engineering, and product teams. We are driven by customer obsession and a commitment to excellence.

Design and develop comprehensive evaluation and benchmarking datasets for Quick Suite AI-powered features

Leverage LLMs for synthetic data corpora generation; data evaluation and quality assessment using LLM-as-a-judge settings

Create ground truth datasets with high-quality question-answer pairs across diverse domains and use cases

Lead human annotation initiatives and model evaluation audits to ensure data quality and relevance

Develop and refine annotation guidelines and quality frameworks for evaluation tasks

Conduct statistical analysis to measure model performance, identify failure patterns, and guide improvement strategies

Collaborate with ML scientists and engineers to translate evaluation insights into actionable product improvements

Build scalable data pipelines and tools to support continuous evaluation and benchmarking efforts

Contribute to Responsible AI initiatives by developing safety and fairness evaluation datasets