This role will lead the creation of testing automation frameworks, testing instrumentation, reporting, and other infrastructure required to test a complex artificial intelligence product. This is a greenfield opportunity to create the processes and approach, select tools, and determine implementation. This role involves understanding of and testing non-deterministic systems, LLMs, machine learning systems.
Responsibilities:
- Design and implement:
- Testing approach across the product components
- Testing automation approach and frameworks
- Test reporting, including progress over time
- Performance evaluation
- Accuracy evaluation
- A/B testing approach
- Streamlined test cycles for engineers
- …and more.
- Participate with engineering team on testing approach
- Participate in design of application architecture
You:
- Know Python and its ecosystem very well
- Have experience testing non-deterministic systems: LLMs, machine learning models, etc.
- Have designed and built a greenfield testing approach for a complex product
- Are deeply knowledgeable about statistical testing approaches
- Enjoy solving hard problems
- Enjoy educating engineers about building testable systems
- Consider automation to be second nature to you
- Understand how to measure performance reliably and consistently
- Dive into code when you find a problem
To apply, email your resume and cover letter to careers@codevalet.com and be sure to include the phrase "I want to empower 10x engineers." ... alternatively reach out to Jan Drake at https://www.linkedin.com/in/janman with the same information.
