Engineering Manager, Agent Oversight

The role

This role sits within the Applied Intelligence Systems group, which builds the tooling that runs agentic AI systems in live environments, alongside applied ML research and evaluation work. The manager will lead the team responsible for the platform's monitoring, evaluation, and continuous-improvement capabilities, coordinating with clients and other internal groups. The team operates across New York and San Francisco and is still expanding.

What you would do

• Direct a cross-functional group of software and ML engineers responsible for delivery on the Scale Generative AI Platform
• Set the platform roadmap across deployment, observability, evaluation, and model-driven refinement of agents
• Turn requirements from enterprise and public-sector buyers into product capabilities alongside PMs and field engineers
• Guide features from initial design through implementation, debugging, and validation
• Run fast experiments grounded in how customers actually use the platform
• Define engineering standards, team norms, and growth plans for a quickly scaling group
• Coach engineers and ML practitioners and shape how the organization expands

What they asked for

• Seven or more years of engineering experience, including two or more years leading people who run production systems built on models, beyond simply integrating an external model service
• Working knowledge of agent design patterns such as tool calling, planning plus step execution, and coordinating multiple agents
• Ability to critique ML experiment design and evaluation methods credibly with scientists and ML engineers
• Proven ownership of platform infrastructure from design through scaling across several internal or external teams
• History of partnering with product managers, field engineering, and clients to convert needs into priorities and shipped capabilities
• Track record of hiring, retaining, and raising the output of high-performing engineering teams

Nice to have

• Built or led evaluation, monitoring, or observability tooling for production LLM products
• End-to-end understanding of the ML and agent lifecycle, from experimentation through deployment and iteration
• Solid grasp of contemporary LLM behavior and agent architecture, including prompt and system tuning and integration with outside tools and services
• Publicly shared work such as papers, open-source projects, or granted intellectual property related to agents, language models, or applied machine learning
• Comfort operating in unclear problem spaces while weighing research ideas against practical product constraints