Lead/Senior Lead BigData - Quality Engineering

• Required Qualifications, Capabilities and Skills

Work on enterprise-scale data and analytics platforms to ensure the quality, accuracy, and reliability of business-critical data solutions.

Collaborate closely with business stakeholders, product owners, data engineers, and development teams to understand requirements and clarify business scenarios.

Translate business requirements and user stories into comprehensive test scenarios, test cases, and validation strategies.

Design and execute functional, integration, regression, and end-to-end testing for data pipelines and analytics applications.

Validate large-scale data transformations, data ingestion processes, and batch workflows across Hadoop and cloud-based data platforms.

Perform source-to-target data validation, data reconciliation, and data quality testing to ensure data accuracy and completeness.

Monitor and validate scheduled jobs workflow executions using Control-M.

Create and manage mock data and test datasets to support testing requirements.

Leverage automation frameworks and AI-assisted testing tools to improve testing efficiency and coverage.

Identify, track, and validate defects while working closely with development and engineering teams to ensure timely resolution.

Participate actively in Agile ceremonie including sprint planning, backlog refinement, daily stand-ups, and sprint reviews.

Ensure testing deliverables are completed within committed sprint timelines while maintaining high quality standards.

Contribute to organizational capability building through knowledge sharing, mentoring, and adoption of testing best practices.

Skills and Expertise

Minimum 5-8 years of relevant experience in Big Data Testing, Cloud Data testing, End to End Data Validation, ETL Testing, or Quality Assurance.

Strong hands-on experience in Hadoop Testing primarily with Hive and validation of data processing workflows on Cloud environment using Azure Databricks.

Experience performing Hive table validation, including schema validation, partition validation, data reconciliation, and source-to-target verification.

Strong proficiency in SQL for data analysis, validation, and testing activities.

Hands-on experience with PySpark for validating large datasets, transformations, and business rules.

Working knowledge of Unix commands for file validation, backend testing, troubleshooting, and data verification.

Experience working with Control-M for batch job monitoring and workflow validation.

Experience with Azure Databricks and Spark-based data processing environments on both ETL/ELT data pipelines.

Ability to analyze business requirementand translate them into effective testing strategies and test cases.

Experience creating mock data and managing test data for various testing scenarios.

Exposure to test automation frameworks and automation tools.

Experience using AI-powered testing tools to improve productivity and test effectiveness.

Strong analytical and problem-solving skills with attention to detail.

Experience working in Agile/Scrum delivery environments.

Excellent written, verbal, presentation, interpersonal, and stakeholder management skills.