Lead Engineer, AI Platform
SKILLS
FULL DESCRIPTION
Lead Engineer, AI Platform
[Employer hidden — sign up to reveal] is building the world’s leading AI-powered, all-in-one platform for digital businesses. We make it possible for creators, coaches, educators, and businesses to bring together their audience with engaging discussions, live streams, events, chat, courses, and payments — all in one place, all under their own brand.
We’re proud to be a fully remote company of around 270 (and growing!) team members from 30+ countries around the world. We seek exceptional individuals around the world, set them up to do the best work of their lives, and in turn, create a meaningful impact in their own lives. We don't track hours, but we do manage for high expectations very closely. We collaborate across time zones, are highly async, and like to document a lot.
Compensation: $170K a year, offers equity. U.S.-benchmarked compensation and equity, regardless of where you live.
About the Role
The AI Quality engineering team at [Employer hidden — sign up to reveal] owns the foundation for measuring, diagnosing, and improving the quality of [Employer hidden — sign up to reveal]'s AI-powered features. We're looking for a Lead Engineer to help us build out the evaluation frameworks, observability tooling, and diagnostic infrastructure that tell us whether our AI Agents are working well, where to improve them, and how to make them faster and more cost-efficient.
What You'll Be Doing
- Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether [Employer hidden — sign up to reveal]'s AI agents are actually working.
- Diagnose exactly where quality breaks down. Trace failures across the agent pipeline.
- Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations.
- Run structured experiments across prompts, models, and the agent harness.
- Lead and grow the AI Quality engineering team.
- Partner closely with AI Core engineering.
What You'll Need to Be Successful
- 7+ years of experience building and shipping production software, ideally including LLM-powered agents.
- Comfortable in Ruby on Rails / Python or ready to pick them up quickly.
- Experience building evaluation or observability infrastructure for ML/AI systems.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
- Learn fast, experiment aggressively, and thrive in a highly dynamic environment.
- Strong alignment with our company values.
- Proficient in English at CEFR Level C2 / ILR Level 5.
Compensation & Benefits
[Employer hidden — sign up to reveal] offers U.S.-benchmarked compensation globally, equity in the company with ongoing refresh grants, and 35 days of paid time off each year. We’re a remote-only team that comes together twice a year for company retreats.