AWS Data Engineer

Responsibilities :

Strong hands-on AWS Data Engineering experience across large-scale cloud data platforms.

Expertise in AWS S3, AWS Glue, AWS Lambda, AWS Redshift, AWS Lake Formation, AWS CloudWatch and AWS IAM.

Strong experience in real-time and streaming data processing using Amazon Kinesis Data Streams, Kinesis Firehose, Kinesis Analytics, Amazon Kafka or equivalent streaming technologies.

Hands-on coding proficiency in Python, PySpark and SQL with strong optimization skills.

Experience building high-volume, low-latency data ingestion and transformation pipelines.

Strong experience in Databricks, Delta Lake, Structured Streaming and Spark Performance Tuning.

Experience handling TB/PB-scale datasets, partitioning strategies, file optimization, schema evolution and data lifecycle management.

Strong understanding of Data Lake, Lakehouse and Modern Data Platform architectures.

Practical data modeling experience for analytical, operational and reporting workloads.

Experience implementing CI/CD pipelines using AWS CodePipeline, CodeBuild, CodeDeploy, GitHub Actions or Jenkins.

Strong troubleshooting skills for production issues, performance bottlenecks and cost optimization.

Excellent client communication, stakeholder management and problem-solving skills.

Technical and Professional Requirements:

Responsibilities and JD in brief along with additional criteria to be considered:

Design and develop scalable AWS Data Engineering solutions and cloud-native data platforms.

Build batch and real-time data pipelines using AWS Glue, PySpark, Databricks, S3, Kinesis , Redshift and Kafka/MSK.

Develop and optimize Databricks, Delta Lake and Spark workloads for large-scale data processing with Pyspark , Databricks transformations.

Implement streaming solutions using Amazon Kinesis Data Streams, Firehose and Analytics for near real-time data ingestion and processing.

Handle high-volume (TB/PB scale) data pipelines with a focus on performance, scalability and cost optimization.

Develop complex SQL transformations, data models and analytics-ready datasets.

Implement CI/CD, Infrastructure as Code (Terraform/CloudFormation) and DevOps best practices.

Ensure data quality, governance, security, monitoring and production support.

Collaborate with architects, stakeholders and cross-functional teams in Agile delivery environments.

Troubleshoot and optimize end-to-end AWS data platforms, streaming pipelines and cloud infrastructure