Description
Join one of the world’s largest virtual companies where you can enjoy the flexibility and space to grow. We seek a founding AI Evaluation Engineer to develop evaluation, testing, and observability infrastructure for production-grade AI agents. These agents manage complex reasoning, multi-turn conversations, and direct tool execution (APIs, code execution, database calls). Move beyond UI automation and manual checks to build robust, statistical, and deterministic evaluation pipelines for non-deterministic Large Language Model (LLM) behavior.
Company Culture and Environment
We foster an environment that values strong engineering, innovation, and continuous improvement. Your growth is our growth, and we prioritize a collaborative atmosphere that encourages creativity and flexibility.
Career Growth and Development Opportunities
We invest in your future with paid training and professional opportunities for development. By becoming part of our team, you will work with modern technologies and have the chance to own your evaluation architecture from scratch.
Detailed Benefits and Perks
- Flexibility to choose your location and working hours.
- A consistent workload that is engaging and challenging.
- Recognition of your hard work and strong performance with annual pay increases.
- The freedom to work from anywhere with a stable internet connection.
Compensation and Benefits
- Starting rate is negotiable based on skills and experience.
- This is an hourly, independent contractor position; compensation is based solely on hours worked and is paid monthly via wire transfer.
- This position does not include PTO as it is independent contractor work.
Why you should apply for this position today
This is an exciting opportunity to shape the future of AI evaluation and testing within a supportive and innovative company. You will have the chance to work on cutting-edge projects while enjoying a flexible work-life balance.
Skills
- Proficiency in Python or TypeScript, focusing on API testing, headless execution, and asynchronous systems.
- Experience building LLM evaluation pipelines or statistical testing frameworks.
- Familiarity with integrating tests into GitHub Actions or similar CI systems.
- Expertise in validating structured outputs: JSON schema, OpenAPI specs.
- Knowledge of tracing tools like OpenTelemetry and LangSmith.
- Ability to design and independently own an evaluation architecture.
- Strong written and spoken English.
- Experience with code execution sandboxes or agent frameworks like LangChain.
- Deep understanding of embedding models and vector databases.
Responsibilities
- Design and deploy automated CI/CD testing pipelines to validate agent behavior.
- Build and maintain golden datasets for regression prompts and edge cases.
- Implement code-based checks before depending on LLM assessments.
- Create test suites for validating multi-turn conversations and safeguards.
- Develop adversarial test suites against prompt injections and chaotic responses.
- Implement tracing using tools like OpenTelemetry and reduce LLM API costs.
Qualifications
- Hands-on experience in building evaluation pipelines for AI systems.
- Expertise in validating structured outputs and understanding of agent frameworks.
- Experience with load testing non-deterministic endpoints.
Education Requirements
- Bachelor’s degree in a relevant field or equivalent experience.
Education Requirements Credential Category
- Bachelor’s degree or equivalent industry experience.
Experience Requirements
- At least 3 hours of daily overlap with US Central Time business hours.
- Proven experience with code execution sandboxes or agent frameworks.
Why work remotely?
Working remotely provides you with the freedom to choose your time zone and preferred hours, allowing you to work from wherever you feel most productive. This flexibility supports your personal and professional growth while ensuring a balanced work-life integration.