Role Summary:

We are seeking a Senior QA Engineer specializing in AI and Multi-Agent Systems to ensure the quality, reliability, and governance of next-generation AI platforms. This role focuses on validating agentic AI architectures, LLM-driven systems, and enterprise AI pipelines, combining traditional QA expertise with AI-specific testing strategies.

The ideal candidate has strong experience in test automation and distributed systems, and has evolved into validating AI-driven workflows, including RAG systems, MPC's, agent orchestration, and decision-making models in regulated environments such as banking and financial services.

Key Responsibilities:

1. AI / Multi-Agent System Testing

Design and implement testing strategies for:
- Multi-agent architectures (cooperative and autonomous agents)
- Agent workflows involving planning, reasoning, and tool usage
Validate:
Test AI agent behavior across:
- Edge cases, failure scenarios, and fallback mechanisms
- Multi-step workflows and distributed execution

2. LLM & GenAI Testing

Validate LLM-powered applications, including:
- Retrieval-Augmented Generation (RAG) systems
- Prompt engineering workflows and tuning strategies
Perform:
- Prompt testing and validation
- Hallucination detection and response quality checks
- Output accuracy benchmarking aligned with business expectations
Build automated test frameworks for:
- Response validation
- Context relevance
- Semantic correctness

3. Test Automation & Framework Development

Develop AI-aware automation frameworks:
- Extend traditional automation (Selenium, API testing) with AI validation layers
Automate:
- Test case generation using AI
- Regression suites with dynamic test data
Integrate with tools such as:
- qTest, JIRA, CI/CD pipelines
Enable intelligent test case reuse and generation

4. Data & Pipeline Validation

Validate:
- Data ingestion pipelines
- Embedding processes and vector databases
- Retrieval quality from knowledge sources
Ensure correctness of:
- Search results
- Context injection into prompts
- Data lineage and integrity

5. Enterprise AI Quality & Governance

Ensure AI systems meet enterprise requirements:
- Accuracy, explainability, consistency
- Auditability and traceability
Support:
- Responsible AI (RAI) compliance
- Model risk validation and governance reviews
Implement monitoring for:
- Prompt execution traces
- Agent actions and outcomes
- Failure patterns and anomalies

6. Performance & Reliability Testing

Conduct:
- Load and scalability testing for AI services
- Latency validation for real-time decision systems
Validate:
- Cost efficiency (LLM token usage optimization)
- System resilience and fallback mechanisms

7. Collaboration & Leadership

Work closely with:
- AI engineers, architects, product teams, and business stakeholders
Mentor QA engineers in:
- AI testing practices
- Automation frameworks
Contribute to:
- Test strategy definition for enterprise AI programs
- Architecture reviews and quality governance

Required Qualifications:

Education

Bachelor's or Master's in Computer Science, Engineering, or related field

Experience

10+ years of QA / SDET experience
2-4 years of experience in AI/ML or GenAI testing
Experience in enterprise platforms (preferably banking / financial systems)

Technical Skills:

AI / GenAI Testing

Understanding of:
- LLMs, RAG architectures, agentic AI
- Prompt engineering and evaluation
Experience validating:
- Multi-agent workflows
- AI reasoning systems

Test Automation

Strong expertise in:
- Selenium / Playwright / API automation
- Test frameworks (BDD, Data-Driven)
Programming:
- Java / Python

Data & Validation

Experience with:
- SQL and data validation
- API testing and microservices validation
Exposure to:
- Vector databases and embeddings (preferred)

DevOps & Tools

CI/CD: GitHub Actions, Jenkins, Harness
Containers: Docker, Kubernetes
Tools: qTest, JIRA, Postman

Preferred Qualifications:

Experience testing:
- AI copilots / enterprise assistants
- Agent-based automation platforms
Familiarity with:
- Regulatory compliance (RBI, Basel, data governance)
- Observability tools (Grafana, Prometheus)

Key Competencies:

Analytical Thinking: Strong ability to identify AI failure patterns
Quality Mindset: Focus on reliability, accuracy, and robustness
Innovation: Adoption of AI-driven testing approaches
Collaboration: Strong stakeholder communication

Success Metrics:

High-quality AI releases with minimal production issues
Improved:
- Test coverage for AI workflows
- Accuracy and reliability of AI outputs
Reduction in:
- Manual testing effort via AI-driven automation
Faster validation cycles for AI features

About State Street

Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.

We are committed to fostering an environment where every employee feels valued and empowered to reach their full potential. As an essential partner in our shared success, you'll benefit from inclusive development opportunities, flexible work-life support, paid volunteer days, and vibrant employee networks that keep you connected to what matters most. Join us in shaping the future.

As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.

Discover more information on jobs at

Read our

Quality Assurance - Assistant Vice President