AI Evaluation Engineer
CV-LibraryLondoncontractPosted: 13 July 2026
Apply NowRole/Job title :AI Evaluation Engineer
Work Location :London
Type work : Contract
Mode of work : Hybrid
Responsibilities:
* Define and implement end-to-end evaluation strategy for generative AI conversational systems (LLMs, RAG pipelines, agents).
* Establish evaluation metrics
* Test Dataset & Benchmarking
* Benchmark models (GPT variants, open-source LLMs) and prompt strategies
* Implement automated evaluation pipelines
* Conduct human-in-the-loop evaluation for qualitative validation.
* Prompt & Response Quality Optimization
* RAG & Knowledge Grounding Validation
* Safety, Risk & Compliance Testing
Your Profile
* Strong understanding of LLMs, RAG architecture, prompt engineering
* Python – data analysis, evaluation pipelines
* Prompt evaluation tools (PromptTools, DeepEval, etc.)
Experience with:
- Evaluation Framework Design
- Establish Evaluation metrics & NLP quality assessment
- Test Dataset and benchmarking
- LLM output evaluation and scoring
- Safety, risk and compliance testing
- Python, data analysis, experimentation frameworks
- Tooling and automation
Familiarity with:
- Responsible AI principles
- Customer support / retail conversational flows
- Analytical mindset with ability to translate model behaviour into actionable improvements
Work Location :London
Type work : Contract
Mode of work : Hybrid
Responsibilities:
* Define and implement end-to-end evaluation strategy for generative AI conversational systems (LLMs, RAG pipelines, agents).
* Establish evaluation metrics
* Test Dataset & Benchmarking
* Benchmark models (GPT variants, open-source LLMs) and prompt strategies
* Implement automated evaluation pipelines
* Conduct human-in-the-loop evaluation for qualitative validation.
* Prompt & Response Quality Optimization
* RAG & Knowledge Grounding Validation
* Safety, Risk & Compliance Testing
Your Profile
* Strong understanding of LLMs, RAG architecture, prompt engineering
* Python – data analysis, evaluation pipelines
* Prompt evaluation tools (PromptTools, DeepEval, etc.)
Experience with:
- Evaluation Framework Design
- Establish Evaluation metrics & NLP quality assessment
- Test Dataset and benchmarking
- LLM output evaluation and scoring
- Safety, risk and compliance testing
- Python, data analysis, experimentation frameworks
- Tooling and automation
Familiarity with:
- Responsible AI principles
- Customer support / retail conversational flows
- Analytical mindset with ability to translate model behaviour into actionable improvements