Senior Data Engineer – Customer Platform & Data
Position Overview
We are seeking Senior Data Engineers with deep experience in distributed data systems, Spark-based data processing, and production-grade data platform engineering.
As a Senior Data Engineer, you will design, build, optimize, and operate large-scale batch and real-time data pipelines supporting critical customer data initiatives, including customer data, identity resolution, bookings, loyalty programs, and AI-powered customer insights.
The ideal candidate is a highly autonomous engineer who can own technical solutions end-to-end, from system design and implementation through production operations, reliability, performance, and cost optimization. You will work in a highly automated, engineering-focused environment that embraces AI-assisted software development and spec-to-code methodologies.
Key Responsibilities
- Design, develop, and optimize large-scale batch and streaming data pipelines using Scala and Apache Spark.
- Build and support real-time data processing solutions using Kafka, Kafka Streams, Flink, or similar technologies.
- Develop and maintain Apache Airflow DAGs, production workflows, and backfill processes.
- Design robust data models, schemas, and source-to-target mappings for large-scale data platforms.
- Implement data quality, validation, reconciliation, and monitoring controls.
- Troubleshoot production issues and drive root-cause analysis and resolution.
- Ensure data platforms meet required standards for reliability, scalability, performance, cost, and data quality.
- Apply strong software engineering practices, including testing, version control, CI/CD, code review, and automated deployment.
- Review, validate, and improve AI-generated code, ensuring that generated solutions meet production engineering and quality standards.
- Take end-to-end ownership of datasets, pipelines, and platform components throughout their lifecycle.
- Contribute to technical design discussions and make pragmatic architecture and implementation decisions.
- Work effectively with configuration-heavy code and multiple programming languages, including Scala, Java, and Python.
- Adopt modern spec-to-code and AI-assisted development practices, including emerging frameworks, agent skills, and specification-led engineering approaches.
Required Qualifications
- 5+ years of professional Data Engineering experience.
- Strong hands-on experience with Scala and Apache Spark in production environments.
- Proven experience building, deploying, and owning production ETL/ELT pipelines.
- Strong experience with distributed data processing and large-scale data systems.
- Practical experience with Kafka, Kafka Streams, Flink, or comparable streaming technologies.
- Hands-on experience with Apache Airflow, including DAG development, scheduling, backfills, and production troubleshooting.
- Strong knowledge of data modeling, schema design, and source-to-target mapping.
- Experience implementing data quality, validation, reconciliation, and monitoring.
- Strong software engineering fundamentals, including automated testing, CI/CD, version control, and code review.
- Experience troubleshooting and supporting production data systems.
- Ability to independently own technical solutions from design through production.
- Ability to critically review and validate AI-generated code rather than relying on generated output without verification.
- Strong communication skills and the ability to clearly explain technical decisions and trade-offs.
Nice-to-Have Qualifications
- Strong Apache Flink expertise.
- Experience with ScyllaDB, Cassandra, DynamoDB, or other NoSQL technologies.
- Experience with Customer Data Platforms (CDP), identity resolution, loyalty systems, clickstream data, customer data, or booking data.
- Experience working with SLAs, SLOs, observability, monitoring, and production reliability.
- Experience with AI-assisted development tools such as GitHub Copilot, Claude, Cursor, or similar platforms.
- Familiarity with spec-led/spec-to-code development, Spec Kit, agent skills, or comparable engineering methodologies.
- Experience working with large-scale customer-facing or data-intensive platforms.
Success Profile
The successful candidate will be a senior, highly autonomous engineer who can confidently demonstrate:
- Experience designing and optimizing large-scale Spark jobs.
- Strong, hands-on Scala production experience.
- Practical experience building and operating streaming systems using Kafka Streams, Flink, or similar technologies.
- Ability to design and maintain Airflow DAGs, production workflows, and backfill strategies.
- Strong understanding of data quality and pipeline validation.
- Comfortable working across Java, Scala, Python, and configuration-heavy codebases.
- Strong production troubleshooting and deployment discipline.
- Experience reviewing and improving AI-generated code with sound engineering judgment.
- An ownership mindset covering systems, datasets, reliability, performance, cost, scalability, and quality.
- Ability to work effectively in a spec-to-code, highly automated engineering environment.
Engineering Environment
The Customer Platform & Data organization operates at significant scale, supporting data-intensive capabilities across customer experiences and business operations.
Engineers will work with technologies including:
Scala | Apache Spark | Kafka | Kafka Streams | Apache Flink | Apache Airflow | Java | Python | NoSQL | CI/CD | AI-Assisted Development
The environment places strong emphasis on automation, engineering quality, observability, reliability, and responsible use of AI development tools.
Why This Position
- Work on large-scale customer data platforms supporting critical customer-facing experiences.
- Solve complex distributed data engineering challenges involving both batch and real-time processing.
- Own systems end-to-end and have meaningful influence over technical architecture and engineering standards.
- Work with modern technologies including Spark, Scala, Kafka, Flink, and Airflow.
- Be part of an engineering culture embracing AI-assisted and spec-to-code development.
- Help build highly reliable, scalable, and data-quality-focused platforms used across a global organization.
- Collaborate with experienced engineers on technically challenging initiatives spanning customer data, identity, bookings, loyalty, and AI-powered insights.
Education
Specialist / Professional Certified
