We are seeking an experienced Senior Data Engineer to design, build, and optimize scalable data pipelines and platforms. The ideal candidate has a deep understanding of data architecture, ETL/ELT processes, cloud technologies, and big data tools. You will work closely with analytics, data science, and product teams to ensure high-quality, reliable, and accessible data.
Key Responsibilities
- Design, develop, and maintain robust, scalable ETL/ELT pipelines for batch and real-time data processing.
- Architect and implement data storage solutions, including data lakes, warehouses, and streaming systems.
- Build and optimize data models, ensuring performance, reliability, and ease of use.
- Collaborate with data scientists, analysts, and stakeholders to understand and provide data requirements.
- Ensure data quality, validation, lineage, governance, and documentation.
- Implement best practices for data security, access control, and compliance.
- Develop automation and monitoring for data workflows.
- Optimize performance of data processing jobs and distributed systems.
- Mentor junior team members and contribute to architectural decisions.
- Work with DevOps teams to deploy and manage data workloads in cloud environments.
Required Skills & Experience
- 5+ years of hands-on experience in data engineering.
- Strong experience with programming languages such as:
- Python, Scala, or Java.
- Proficiency in SQL and working with large datasets.
- Experience with big data frameworks such as:
- Apache Spark, Hadoop, Kafka, Flink, Airflow, dbt.
- Cloud expertise in at least one platform:
- AWS (Glue, EMR, Redshift, Athena, S3)
- Azure (Data Factory, Synapse, Databricks)
- GCP (BigQuery, Dataflow, Dataproc)
- Experience working with data warehouses:
- Snowflake, Redshift, BigQuery, Synapse, etc.
- Knowledge of CI/CD pipelines, Git, Terraform, Docker, or Kubernetes.
- Understanding of data modeling, dimensional modeling, and schema design.
- Strong grasp of distributed systems and performance optimization.
Preferred Qualifications (Nice to Have)
- Experience with Databricks or Delta Lake.
- Knowledge of machine learning pipelines or feature stores.
- Hands-on experience with streaming systems (Kafka, Kinesis).
- Exposure to data governance frameworks (Collibra, Alation).
- Experience mentoring other engineers or leading technical initiatives
