About the job
We are looking for a Senior AI Architect to define and lead the architecture for Commerce Factory’s Agent Harness and Learning Layer. You will shape how agents learn, evaluate, and improve, collaborating with cross-functional teams. Ideal candidates bring extensive experience in AI and machine learning at scale.
Responsibilities
Define the architecture for Commerce Factory’s agent learning and feedback systems, including how agents evaluate outcomes, capture quality signals, and improve over time.
Design feedback and evaluation pipelines that measure agent outcomes, identify meaningful quality signals, and translate those signals into improvements such as prompt and skill updates, routing changes, and reward or preference data.
Apply AI and machine learning techniques such as preference learning, reward modeling, reinforcement learning, and progress guided by analysis.
Define how the Learning Layer integrates with the Agent Harness, evaluation infrastructure, skills layer, tool calling, context management, and execution loop.
Establish architecture standards and technical direction across Commerce Factory through architecture reviews, design documents, technology evaluations, and build-versus-buy decisions.
Develop prototypes and proof-of-concepts using current AI models, agent frameworks, training approaches, and evaluation techniques.
Qualifications
Minimum
12+ years of software engineering experience, including experience designing and architecting AI, ML, or data-intensive systems at scale.
Experience crafting and delivering LLM and agentic systems, including tool use, context and state management, output evaluation, and improvement based on user input.
Experience with techniques such as reinforcement learning, preference learning, RLHF, reward modeling, or related approaches, with experience applying these techniques to production systems.
Experience designing evaluation and learning systems that connect agent actions, measurable outcomes, feedback signals, and system improvements.
Demonstrated ability to translate ambiguous business or technical objectives into clear architecture, technical strategy, and execution plans.
Experience applying AI or machine learning to products or platforms, with measurable results such as improved quality, adoption, reliability, productivity, or cost efficiency.
Experience providing technical leadership through architecture reviews, technical standards, mentoring, or guidance for engineering teams.
Proficiency in Python and at least one additional programming language.
Experience with cloud platforms such as AWS or Azure, data pipeline technologies, and ML experimentation infrastructure.
Ability to collaborate effectively across engineering, product, architecture, and platform teams.
Preferred
No preferred qualifications listed.