Position Summary:
We are seeking a Professional Data Engineer to join our dynamic team, where you will play
a crucial role in developing and maintaining robust data solutions.
As a Professional Data Engineer, you will collaborate with data science, business analytics,
and product development teams to deploy cutting-edge techniques and utilise best-in-class
third-party products. The Data team operates with engineering precision, prioritising security, privacy, and regulatory compliance in every initiative. As a Professional Data Engineer, you will contribute to the team's commitment to utilising the latest tools and methodologies, ensuring that our data solutions align with industry best practices.
Our Tech Stack:
- Languages: SQL and Python
- Pipeline orchestration tool: Dagster (Legacy: Airflow)
- Data stores: Redshift, Clickhouse
- Platforms & Services: Docker, Kubernetes
- PaaS: AWS (ECS/EKS, DMS, Kinesis, Glue, Athena, S3 and others.)
- ETL: FiveTran & DBT for transformation
- IaC: Terraform (with Terragrunt)
- GenAI: Bedrock, LangChain, LLMs
Key Responsibilities:
- Develop and maintain ETL pipelines using SQL and/or Python.
- Use tools like Dagster/Airflow for pipeline orchestration
- Collaborate with cross-functional teams to understand and deliver data requirements.
- Ensure a consistent flow of high-quality data using stream, batch, and CDC processes.
- Use data transformation tools like DBT to prepare datasets to enable business users to self-service.
- Ensure data quality and consistency in all data stores.
- Monitor and troubleshoot data pipelines for performance and reliability.
Essential Experience:
- 3+ years of experience as a data engineer.
- Proficiency in SQL is a must
- Experience with modern cloud data warehousing, data lake solutions like Snowflake, BigQuery, Redshift, Azure Synapse.
- Experience with ETL/ELT, batch, streaming data processing pipelines.
- Excellent ability to investigate and troubleshoot data issues, providing fixes and proposing both short and long-term solutions.
- Knowledge of AWS services (like S3, DMS, Glue, Athena, etc.)
- Familiar with DBT or other data transformation tools.
- Familiarity with GenAI, and how to leverage LLMs to resolve engineering challenges
Other Desired Experience:
- Experience with AWS services and concepts (like EC2, ECS, EKS, VPC, IAM, etc).
- Familiar with Terraform and Terragrunt.
- Experience with Python
- Experience with orchestration tools like Dagster, Airflow, AWS Step functions, etc.
- Experience with pub-sub, queuing, and streaming frameworks such as AWS Kinesis, Kafka, SQS, SNS.
- Familiar with CI/CD pipelines and automation
