Job Description
Technical Competence/skills
Python
SQL
Apache Airflow
PostgreSQL / MySQL
Git / GitHub / GitLab
Tasks and job roles
Data Modeling
Designing and Architecting Data Pipelines:
ETL/ELT Development: Design, implement, and optimize ETL/ELT workflows to support both batch and real-time data ingestion from diverse sources.
Optimizing Data Storage and Performance: Analyze and improve data storage architecture to enhance performance, reduce latency, and support growing data volume. Leverage indexing, partitioning, caching, and other optimization strategies to ensure efficient querying and scalability.
Implementing Data Quality And Governance Practices
Collaborating with Data Scientists and Analysts: Work closely with data scientists, analysts, and business stakeholders to ensure data availability, clarity, and usability. Translate analytical and reporting requirements into technical data solutions and pipelines that enable insight generation and AI/ML model development.
Automation and Monitoring: Implement automated workflows for data processing, testing, and deployment. Set up real-time monitoring tools and alerting systems to proactively identify and resolve issues in data pipelines, ensuring high availability and reliability.
Data Security And Privacy
Documentation and Knowledge Sharing:
Infrastructure Management: Oversee and maintain the data engineering infrastructure, including scheduling tools (e.g., Airflow), data repositories, CI/CD pipelines, and development environments. Coordinate with IT and DevOps teams to ensure system performance, version control, and disaster recovery readiness.
