Sr Data Engineer

Job description

• Juniper Experience and Operations (JXO) organization is incubating a Technology group to transform CX (customer experience) and EX (employee experience). The charter of this group is to discover, evaluate and leverage technology to enhance and simplify the experience of stakeholders. Knowledge and data will be central to this journey of creating a proactive and predictive support experience. The use of automation, AI and other modern technology will enable reduction of time taken to resolve issues or perform tasks.
• This role is part of Juniper s strategic future of support team and will involve design of automated solutions. The role requires enabling business transformation projects using technology, review of data to enable process, systems and tool re-engineering in customer support and services. In addition, the role requires to support the enhancement of self-service, automation, omnichannel strategy by seeking solutions and drivers to achieve seamless customer experiences and increased customer loyalty.

Primary Tech skills:

• AWS , Databricks , Python, Pyspark, SQL, Web-Crawling

Secondary Tech skills:

• Snowflake, MLOps

Responsibilities:

• Develop, maintain, and optimize data pipelines and workflows using Databricks Job workflows and Feature Store to ensure seamless data ingestion and transformation as a scalable data solutions.
• Architect Lakehouse Solutions: Design, implement, and architect Lakehouse solutions on Databricks, leveraging Delta Lake, and Feature Store to enhance data storage and processing.
• In-depth Databricks Expertise: Demonstrate a deep understanding of Databricks platform features, including Spark SQL, Delta Lake, Feature Store and Databricks notebooks, to optimise data engineering processes.
• Data Transformation: Implement advanced data transformations and quality checks within the Lakehouse architecture to ensure data accuracy, completeness, and consistency.
• Data Integration: Seamlessly integrate data from diverse sources, aligning with Lakehouse principles for data ingestion and storage, leveraging AWS S3 Storage and possibly Snowflake as a SQL Data Warehouse.
• Data Security: Implement and maintain comprehensive data security and access controls within the Lakehouse architecture, utilising AWS IAM policies as security features to safeguard sensitive data.
• Performance Optimisation: Architect data pipelines for performance and scalability, making efficient use of Databricks clusters and AWS storage resources through S3 Life cycle.
• Data Modelling: Create and implement advanced data models and schemas on Databricks, aligning with Lakehouse principles for analytical and reporting needs.
• Documentation: Create detailed documentation for data pipelines, data models, Lakehouse architecture configurations, AWS integrations, and data migration plans.
• Troubleshooting: Proactively identify and resolve complex data pipeline and architecture issues, ensuring data integrity and availability, with a focus on Databricks/AWS monitoring and diagnostics.
• Performance Monitoring: Employ advanced monitoring techniques and tools to maintain the performance and health of the Lakehouse architecture, taking proactive measures to address potential bottlenecks or issues. Should also have understanding of cluster monitoring and sizing.
• Data Governance: Ensure strict adherence to data governance and data management best practices within the Lakehouse architecture, utilising AWS-based data governance solutions.

Qualification and Desired Experiences:

• 7+ years of data analysis and engineering experience
• Bachelors degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.
• Experience with big data tools: Hadoop, Spark, Kafka, Spark & Kafka Streaming, Python
• Familiarity with Snowflake environment
• Advanced working SQL experience working with relational databases, query authoring (SQL) and working familiarity with a variety of databases.
• Hands-on experience in Databricks for data pipeline development and AWS services like S3, Glue, EMR, EC2.
• Experience building big data data pipelines, architectures and data sets. In-depth knowledge of Model and Design of DB schemas for read and write performance.
• Working knowledge of API or Stream-based data extraction processes like Salesforce API and Bulk API and have hands-on experience in web crawling.
• Experience performing root cause analysis on all data and processes to answer specific questions and identify opportunities for improvement.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management. A successful history of manipulating, processing and extracting value from large disconnected datasets.

Personal Skills:

• Ability to collaborate cross-functionally in a fast-paced environment and build sound working relationships within all levels of the organization
• Ability to handle sensitive information with keen attention to detail and accuracy. Passion for data handling ethics.
• Ability to solve complex, technical problems with creative solutions while anticipating stakeholder needs and providing assistance to meet or exceed expectations
• Able to demonstrate perseverance and resilience to overcome obstacles when presented with a complex problem.
• Assist in combining large data sets and data analysis to create optimization strategies
• Comfortable with ambiguity and uncertainty of change when assessing needs for stakeholders
• Have effective time management skills which enable you to work successfully across functions in a dynamic and solution-oriented environment while meeting deadlines
• Self-motivated and innovative; confident when working independently, but an excellent team player with a growth-oriented personality
• Will be required to routinely or customarily troubleshoot items related to applications that require independent judgement, decision-making, and unique approaches