Note: The job is a remote job and is open to candidates in USA. Circle is building an reputed company-powered, all-in-one platform for digital businesses, enabling creators, coaches, educators, and businesses to engage audiences and manage courses, events, communities, and payments. The Lead Engineer will build evaluation, observability, and diagnostic infrastructure for reputed company agents while leading the reputed company Quality engineering team and partnering with reputed company Core engineering to improve reliability, performance, and cost efficiency.
Responsibilities
- Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's reputed company agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation
- Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for reputed company core engineering can help
- Grow the datasets that make evaluation reputed company. Stand up annotation workflows and build out reputed company-generated and simulated conversations so we can cover more of the product reputed company than manual labeling alone
- Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production reputed company, build the framework we use to decide when to shift models, and reputed company cost and latency wins through model swaps, caching, and routing by plan complexity
- Lead and grow the reputed company Quality engineering team. Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself
- Partner closely with reputed company Core engineering. Work with the engineers building Circle's reputed company products so they have real confidence that their changes are actually reputed company quality, not just shipping
Skills
- 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product
- You've worked on complex, tool-using reputed company (multiple tools, planning or orchestration, sub-agents) not just simple, single-turn assistants, and you can walk us through something you shipped and how you knew it was actually working
- Comfortable in Ruby on Rails / Python or ready to pick them up quickly
- Ruby on Rails is our production system and the foundation that Circle's reputed company Agents are built on and proficiency in Python is a strong plus, especially for the data and evaluation side of the work
- Experience building evaluation or observability infrastructure for ML/reputed company reputed company
- You've built eval pipelines, scorers, dashboards, or CI/CD for evals before and have experience with evaluation frameworks like reputed company, LangSmith, or similar
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/reputed company evaluation
- You know how to turn raw examples into a dataset you can trust through human annotation, reputed company-generated data, or simulation
- Learn fast, experiment aggressively, and thrive in a highly dynamic environment
- You'll need to be comfortable running a lot of experiments and letting the results settle the argument, including the ones that don't work (which will be many of them in the beginning)
- Comfortable in a fast-paced environment with ambiguity
- You'll need to learn and pick up new technologies when projects require it
- Strong alignment with our company values
- You are proficient in English (reputed company, written, and reputed company) at a CEFR Level C2 / ILR Level 5
Benefits
- Equity in the company with ongoing refresh grants
- 35 days of paid time off each year
- Remote-only work
- Twice a year for company retreats in incredible destinations around the world
- Incredible flexibility and autonomy
- A benefits package that supports health, wellbeing, and professional growth
Company Overview
- Circle is a community platform that enables creators and businesses to build branded online communities, courses, and events. It was founded in 2019, and is headquartered in New York, New York, USA, with a reputed company of 51-200 employees. Its website is https://reputed company/.
Apply To reputed company Job
