Automation Test Engineer (AI & Chatbot Testing)

Automation Test Engineer (AI & Chatbot Testing)

Location:

UAE

Job Summary

We are looking for an experienced

Automation Test Engineer

to test and improve AI-powered customer support and troubleshooting solutions. The role focuses on test automation, API testing, AI quality validation, and performance testing to ensure reliable and accurate customer experiences.

Key Responsibilities

AI & Chatbot Testing

• Develop automated tests for AI chatbots and RAG (Retrieval-Augmented Generation) systems.

• Validate response accuracy, relevance, and AI performance.

• Measure AI quality metrics such as hallucination rates and answer relevance.

• Perform regression testing when AI models or knowledge bases are updated.

Automation & Functional Testing

• Automate testing of troubleshooting workflows and decision trees.

• Validate end-to-end customer support scenarios.

• Simulate customer interactions and verify expected outcomes.

API & Integration Testing

• Build and execute automated API tests.

• Validate integrations between AI platforms and backend systems.

• Create mock services and test data where required.

Performance & Security Testing

• Conduct load and performance testing of conversational AI systems.

• Test AI guardrails, prompt injection risks, and unsafe responses.

• Identify performance and reliability issues.

CI/CD & Quality Assurance

• Integrate automated tests into Azure DevOps pipelines.

• Track quality metrics and generate test reports.

• Work closely with developers, AI engineers, and product teams.

Requirements

• 4+ years of experience in QA Automation or SDET roles.

• 1+ year of experience testing AI, Chatbots, LLMs, or Agentic AI solutions.

• Strong programming skills in Python, TypeScript, or C#.

• Experience with REST APIs, GraphQL, WebSockets, Postman, and WireMock.

• Experience with Azure DevOps and CI/CD pipelines.

• Knowledge of RAG, semantic search, and vector databases.

• Experience with MQTT or IoT telemetry is a plus.

• Exposure to embedded systems or consumer electronics is preferred.