Data Engineer

Key Responsibilities:

• Analyze the client's data needs and document the requirements.
• Refine data collection/consumption by migrating data collection to more efficient channels.
• Plan, design and implement data engineering jobs and reporting solutions to meet the analytical needs.
• Develop test plans and scripts for system testing, and support user acceptance testing.
• Build reports and dashboards according to user requirements.
• Work with the client's technical teams to ensure smooth deployment and adoption of new solutions.
• Ensure the smooth operations and service levels of IT solutions.
• Support production issues.
• Apply AI coding tools such as Copilot and Coder AI to support development, testing, and code reviews.
• Use automation to improve development efficiency. This includes build, test, and deployment activities.
• Leverage AI assistance to improve code quality. This covers refactoring, defect resolution, and documentation.
• Demonstrate willingness to adopt new AI and automation tools where relevant to work outcomes.

What We Are Looking For

• Good understanding and completion of projects using Waterfall/Agile methodology.
• Working experience using Cloudera AI (Machine Learning), Plotly, and data engineering AI tool stack.
• Strong SQL, data modelling, and data analysis skills are a must.
• Hands-on experience in big data engineering jobs using Python, PySpark, Linux, and ETL tools like Informatica.
• Hands-on experience in a reporting or visualization tool like SAP BO and Tableau is a must.
• Hands-on experience in DevOps deployment and data virtualisation tools like Denodo will be an advantage.
• Hands-on experience in scripting such as Python and shell scripting will be an advantage.
• Track record in implementing systems using Hive, Impala, and Cloudera Data Platform will be preferred.
• Good understanding of analytics and data warehouse implementations.
• Ability to troubleshoot complex issues ranging from system resources to application stack traces.
• Track record in implementing systems with high availability, high performance, and high security hosted across data centres or hybrid cloud environments will be an added advantage.
• Passion for automation, standardization, and best practices.
• AI and automation experience will be an advantage.