Job Description

AI Test Engineer / SDET – LLM & RAG | Dubai (On-site)

STATERA is looking for an AI Test Engineer to automate the testing of an AI-powered customer support and product troubleshooting solution. The role combines software test automation with hands-on evaluation of LLMs, RAG pipelines, and conversational agents.

Responsibilities

Build automated evaluations for RAG retrieval accuracy, context relevance, semantic search precision, and response quality.
Measure faithfulness, answer relevance, hallucination rates, and guardrail compliance.
Test multi-turn troubleshooting flows across product issues such as connectivity failures, battery degradation, and performance faults.
Simulate complex customer interactions to verify diagnostic state navigation and resolution paths.
Prevent regressions when product documentation, firmware specifications, or telemetry models change.
Automate API and end-to-end tests across the AI agent, customer support workflows, and warranty-related integrations.
Run load and scalability tests on conversational APIs.
Develop adversarial tests covering prompt injection, out-of-scope requests, and hallucination risks.
Integrate AI evaluation and regression suites into Azure DevOps CI/CD pipelines.

Required experience

4+ years in QA automation or SDET roles, including 1–2 years testing LLMs, chatbots, or agentic AI systems.
Advanced proficiency in Python (including pytest and asyncio) and/or TypeScript or C#.
Strong API testing and mocking experience with REST/GraphQL, WebSockets, Postman, WireMock, or custom mock servers.
Experience evaluating RAG systems, semantic search, and vector database integrations.
Ability to design reliable automated tests for multi-turn conversations and AI behavior.
Experience with CI/CD and Azure DevOps.

Valuable additional experience