Forward Deployed Engineer - AI Assurance

ABOUT

:

Owns quality for AI-native applications — functional testing plus the AI-specific evaluation (accuracy, drift, hallucination).

KEY RESPONSIBILITIES

• Build test plans and automation for AI-native application features (functional + AI-specific)

• Design evaluation harnesses for model/agent outputs — accuracy, consistency, hallucination rate

• Run regression testing across model/prompt/config changes to catch silent quality drift

• Red-team AI features for edge cases and adversarial inputs where relevant

• Build automated eval pipelines integrated into CI/CD

• Partner with AI Architects to define testability requirements before build starts

• Own the quality gate before any AI feature ships to production

• Communicate quality risk to delivery leadership in terms they can act on

• Train delivery teams on AI-specific testing practices

• Own the evals framework for the practice — golden datasets, scoring rubrics, LLM-as-judge calibration, and versioned benchmarks per use case

• Define eval acceptance thresholds per engagement and gate releases on them

• Build eval engineering tooling — dataset curation, trace capture, offline/online eval runs, and dashboards delivery teams can read

• Instrument production evals and drift monitoring, feeding failures back into the golden datasets

REQUIREMENTS & SKILLS

• 5–9 yrs QA/test engineering, with 2+ yrs testing AI/ML-powered features specifically

• Strong test automation skills (Python-based frameworks, CI/CD integration)

• Understands AI-specific failure modes — hallucination, bias, drift, non-determinism — and designs tests for them

• Statistically literate enough to interpret model evaluation metrics, not just pass/fail results

• Familiarity with red-teaming methodologies for AI systems

• Clear, assertive communicator — willing to block a release over a quality concern

• Detail-oriented and methodical under delivery-timeline pressure

• Collaborative but independent — doesn't rubber-stamp under delivery pressure

• Explains quality risk in business-impact terms, not just technical jargon

• Hands-on evals engineering — builds and maintains eval suites with frameworks such as OpenAI Evals, Ragas, DeepEval, LangSmith, Azure AI Foundry evaluations

• Designs golden datasets and rubrics, and calibrates LLM-as-judge scoring against human review

• Understands RAG and agent eval metrics — groundedness, retrieval precision/recall, task completion, tool-call correctness, cost/latency

• Experience wiring evals and drift monitoring into CI/CD and production observability