Lead AI QA Engineer - Manual & Functional Testing

Lead AI QA Engineer - Manual & Functional Testing

Project Foundry Resourcing Services

About the Role

We are seeking an experienced

Lead AI QA Engineer

to take ownership of quality assurance across a fast-paced, multi-agent LLM product.

This is a hands-on leadership role with responsibility for defining the QA approach, improving test coverage, coordinating testing activity and ensuring the quality, accuracy and reliability of AI-driven functionality across the platform.

The role will retain a strong focus on

manual testing, functional testing, front-end validation, answer accuracy and source-data alignment

, while also providing technical direction and leadership across the wider QA function.

The successful candidate will lead the development of test strategies, test plans and quality standards, prioritise testing activity based on risk and business impact, and ensure AI-generated responses are validated against the correct underlying data sources.

They will work closely with developers, data scientists, AI engineers, product managers and delivery teams to identify quality risks, improve development and release processes, drive defects through to resolution and provide a clear view of product readiness.

This is a complex and evolving AI environment where strong technical judgement, structured thinking, attention to detail and the ability to lead quality across multiple workstreams are essential.

Key Responsibilities

QA Leadership & Test Strategy

• Own and continuously improve the overall QA and testing strategy for the AI Assistant and associated product functionality.

• Define testing standards, processes, quality gates and release-readiness criteria across the product.

• Lead the planning and prioritisation of QA activity across multiple development workstreams and releases.

• Develop risk-based testing approaches to ensure the highest-impact areas receive appropriate coverage.

• Provide technical guidance and support to QA Engineers and Testers across the team.

• Review test plans, test cases, defect reports and test evidence to ensure consistent quality and coverage.

• Identify gaps in existing QA processes and implement practical improvements to increase product quality and testing efficiency.

• Act as the primary QA point of contact for engineering, product and delivery teams.

• Provide clear recommendations on release readiness, quality risks and areas requiring further validation.

Manual Test Planning & Execution

• Lead the development, documentation and execution of detailed manual test plans and test cases for the AI Assistant's front-end UI, conversational flows and core product functionality.

• Perform and oversee functional, regression, exploratory, smoke and usability testing across new features, enhancements and production fixes.

• Validate that user journeys, prompts, responses, data retrieval and system behaviours operate as expected across a wide range of scenarios.

• Ensure Jira tickets for defects, test findings and follow-up actions are clear, reproducible and appropriately prioritised.

• Work directly with developers to investigate and reproduce complex defects and validate fixes once resolved.

• Ensure test evidence, execution results and release validation documentation are maintained to a consistent standard.

AI Response Validation & Accuracy Testing

• Lead the approach to validating LLM-generated responses against structured and unstructured data sources.

• Ensure AI responses are accurate, complete, relevant and grounded in the correct source information.

• Define repeatable methods for testing answer accuracy, source alignment and consistency across different prompts and scenarios.

• Identify, document and escalate hallucinations, incorrect responses, source mismatches, missing context, reasoning gaps and inconsistent outputs.

• Develop comprehensive test scenarios covering realistic user behaviour, adversarial prompts, edge cases and complex business scenarios.

• Analyse recurring AI quality issues and work with AI Engineers, Data Scientists and Developers to identify underlying causes.

• Support the development of quality metrics and validation frameworks to improve confidence in AI-generated outputs.

• Help establish clear acceptance criteria for AI response quality and expected product behaviour.

Functional & Front-End Testing

• Lead testing across front-end screens, workflows, navigation, forms, conversational interfaces and user interactions.

• Validate business rules, permissions, access controls and end-to-end user journeys.

• Ensure appropriate cross-browser, UI consistency and usability testing is performed.

• Coordinate regression and release testing across new functionality, fixes and impacted product areas.

• Identify areas of increased risk following product or architecture changes and adjust testing coverage accordingly.

Automation & Test Efficiency

• Define where automated testing can add value while maintaining strong manual validation for complex AI behaviours.

• Work with engineering teams to develop an appropriate balance between manual and automated testing.

• Support the creation and maintenance of automated tests for repeatable UI, API, regression and performance scenarios.

• Identify opportunities to automate repetitive QA activity and improve overall testing efficiency.

• Ensure automated testing complements rather than replaces the detailed manual validation required for AI-generated responses.

• Drive improvements in reusable test assets, regression suites and overall test coverage.

Performance, Security & Access Testing

• Coordinate testing of response times, system performance and behaviour under different workloads.

• Work with engineering and security teams to validate data privacy, access control, permissions and compliance requirements within AI interactions.

• Lead testing to identify scenarios where users could receive incorrect, unauthorised, sensitive or inappropriate information.

• Ensure security and permission-related defects are appropriately prioritised and escalated.

• Support validation of AI behaviour across different user roles, permission levels and data-access scenarios.

Defect Management & Release Quality

• Own the QA view of defect severity, priority and overall product quality.

• Ensure critical and high-impact issues are appropriately investigated, prioritised and resolved.

• Lead defect triage discussions with development, engineering and product teams.

• Monitor defect trends and identify recurring quality issues or systemic weaknesses.

• Coordinate release validation and provide clear quality recommendations ahead of deployment.

• Support post-release validation and ensure production issues are rapidly assessed and prioritised.

• Provide stakeholders with a clear view of outstanding quality risks before releases are approved.

Collaboration & Reporting

• Work closely with Developers, Data Scientists, AI Engineers, DevOps, Product Managers and delivery teams.

• Participate in and contribute to sprint planning, stand-ups, retrospectives, backlog refinement and release-readiness discussions.

• Represent QA considerations during product design, technical discussions and feature development.

• Communicate testing progress, quality risks, blockers and release concerns clearly to technical and non-technical stakeholders.

• Provide structured reporting on testing progress, defect status, quality trends and release readiness.

• Promote a strong quality culture across engineering and product teams.

Required Skills & Experience

Must Have

• Significant experience in software testing and QA, including experience operating at Senior, Lead or Test Lead level.

• Strong hands-on experience in

  manual and functional testing

  .

• Proven experience testing

  AI, LLM, chatbot, virtual assistant or NLP-driven products

  .

• Experience leading QA activity across complex software products or multiple development workstreams.

• Strong understanding of QA strategy, risk-based testing, test planning and release-quality management.

• Strong understanding of functional, regression, exploratory, smoke, usability and front-end testing techniques.

• Experience creating and reviewing detailed test plans, test cases, test scripts, test evidence and defect reports.

• Proven ability to validate AI-generated responses against underlying source data.

• Strong ability to identify hallucinations, inaccuracies, source mismatches, inconsistent outputs and missing context within AI-generated responses.

• Good understanding of LLMs, NLP concepts, retrieval-based AI systems and AI response validation.

• Experience leading defect triage and working directly with engineering teams to investigate complex issues.

• Experience providing release-readiness assessments and communicating quality risks to stakeholders.

• Strong attention to detail and a structured, methodical approach to testing and troubleshooting.

• Ability to prioritise QA activity based on business impact, technical risk and release urgency.

• Experience working in fast-paced environments with evolving requirements.

• Strong written and verbal communication skills.

• Ability to provide technical guidance, mentoring and support to other QA Engineers or Testers.

• Comfortable working closely with engineering, AI, data, product and delivery teams.

Desirable Skills

• Experience using Jira or similar tools for defect tracking, test management and reporting.

• Experience implementing or improving QA processes within an AI or software product environment.

• Exposure to automated testing tools or frameworks for UI, API or regression testing.

• Experience testing data-driven, API-enabled or multi-agent AI products.

• Experience with Retrieval-Augmented Generation (RAG), vector databases, embeddings or similar AI architectures.

• Experience developing structured evaluation frameworks for LLM response quality.

• Experience defining or tracking AI quality metrics such as accuracy, groundedness, relevance and consistency.

• Experience supporting performance or load testing.

• Experience supporting release readiness, UAT, production validation and post-release defect triage.

• Familiarity with privacy, security, permissions and access-control considerations within AI or data-driven applications.

• Experience working within Agile software development environments.