Data Engineer(PySpark)

Greetings,

We have an immediate vacancy for a Data Engineer (PySpark) with 7+ years of experience at Synechron, based in Dubai.

Job Role: - Data Engineer (PySpark)

Job Location: - Dubai

Need Skill Set: PySpark, Python, Spark, SQL, Hadoop.

About Company:

At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading digital solutions. Synechron's progressive technologies and optimization strategies span end-to-end Artificial Intelligence, Consulting, Digital, Cloud & DevOps, Data, and Software Engineering, servicing an array of noteworthy financial services and technology firms. Through research and development initiatives in our FinLabs we develop solutions for modernization, from Artificial Intelligence and Blockchain to Data Science models, Digital Underwriting, mobile-first applications and more. Over the last 20+ years, our company has been honored with multiple employer awards, recognizing our commitment to our talented teams. With top clients to boast about, Synechron has a global workforce of 17000+, and has 58 offices in 21 countries within key global markets. For more information on the company, please visit our website or LinkedIn community.

Diversity, Equity, and Inclusion

Synechron's Diversity, Equity, and Inclusion (DEI) program, Same Difference, was developed because we believe in a culture of listening, respect, and opportunity.

We each bring unique backgrounds, thoughts, talents, and experiences with us to work every day, and we know that by embracing them, we are creating an even greater Synechron. The best way to build a strong team is to value individual differences. So, it doesn't matter where you're from or what you've had to do to get here if you have the skills, enthusiasm, and drive to make your mark, we'll support you like we support each other. Choose a career with us and let's pursue innovation, together.

Job Descriptions:

About the Role:

We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role requires a strong background in big data ecosystems, cloud-native tools, and advanced data processing techniques.

The ideal candidate has hands-on experience with data ingestion, transformation, and optimization on the Cloudera Data Platform, along with a proven track record of implementing data engineering best practices. You will work closely with other data engineers to build solutions that drive impactful business insights.

Responsibilities:

• Data Pipeline Development: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy.
• Data Ingestion: Implement and manage data ingestion processes from a variety of sources (e.g., relational databases, APIs, file systems) to the data lake or data warehouse on CDP.
• Data Transformation and Processing: Use PySpark to process, cleanse, and transform large datasets into meaningful formats that support analytical needs and business requirements.
• Performance Optimization: Conduct performance tuning of PySpark code and Cloudera components, optimizing resource utilization and reducing runtime of ETL processes.
• Data Quality and Validation: Implement data quality checks, monitoring, and validation routines to ensure data accuracy and reliability throughout the pipeline.
• Automation and Orchestration: Automate data workflows using tools like Apache Oozie, Airflow, or similar orchestration tools within the Cloudera ecosystem.
• Monitoring and Maintenance: Monitor pipeline performance, troubleshoot issues, and perform routine maintenance on the Cloudera Data Platform and associated data processes.
• Collaboration: Work closely with other data engineers, analysts, product managers, and other stakeholders to understand data requirements and support various data-driven initiatives.
• Documentation: Maintain thorough documentation of data engineering processes, code, and pipeline configurations.

Education and Experience:

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field.
• 3+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform.

Technical Skills:

• PySpark: Advanced proficiency in PySpark, including working with RDDs, DataFrames, and optimization techniques.
• Cloudera Data Platform: Strong experience with Cloudera Data Platform (CDP) components, including Cloudera Manager, Hive, Impala, HDFS, and HBase.
• Data Warehousing: Knowledge of data warehousing concepts, ETL best practices, and experience with SQL-based tools (e.g., Hive, Impala).
• Big Data Technologies: Familiarity with Hadoop, Kafka, and other distributed computing tools.
• Orchestration and Scheduling: Experience with Apache Oozie, Airflow, or similar orchestration frameworks.
• Scripting and Automation: Strong scripting skills in Linux.

Soft Skills:

• Strong analytical and problem-solving skills.
• Excellent verbal and written communication abilities.
• Ability to work independently and collaboratively in a team environment.
• Attention to detail and commitment to data quality