QA Engineer - Agentic & Generative AI

Job title: Agentic QA Engineer – Generative AI & Agentic Systems (Agent, Multi‑Agent Testing)

Location: Dallas, TX (Onsite)

Duration: Long term Contract

Mode of interview: 2 F2F rounds interviews in Dallas, TX (Must)

No Third Party

Required Qualifications

• 7+ years in Software QA/Testing

  , with 2+ years in AI/ML or LLM-based systems

  ; hands-on experience testing agentic/multi-agent architectures.

• Strong programming skills in Python experience building test harnesses, simulators, and fixtures.

• Experience with LLM evaluation

  (exact/soft match, BLEU/ROUGE, BERTScore, semantic similarity via embeddings), guardrails

  , and prompt testing

  .

• Expertise in distributed systems testing latency profiling, resiliency patterns (circuit breakers, retries), chaos engineering

  , and message queues.

• Familiarity with orchestration frameworks

  (LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, Azure OpenAI orchestration, or similar).

• Proficiency with CI/CD

  (GitHub Actions/Azure DevOps), observability

  (OpenTelemetry, Prometheus/Grafana, Datadog), and feature flags/canaries

  .

• Solid understanding of privacy/security/compliance in AI systems (PII handling, content policies, model safety).

• Excellent communication and leadership skills; proven ability to work cross-functionally with Ops, Data, and Engineering.

Preferred Qualifications

• Experience with multi-agent simulators

  , agent graph testing

  , and tooling latency emulation

  .

• Knowledge of MLOps

  (model versioning, datasets, evaluation pipelines) and A/B experimentation for LLMs.

• Background in cloud

  (AWS), serverless

  , containerization

  , and event-driven architectures.

Prior ownership of cost/latency/SLAs for AI workloads in production.