Kafka Architect – (For Banking Data Lakehouse Program)
Location: Jeddah, KSA
Experience : 10 - 12 Years
Key Responsibilities
1. Architecture & Design
- Design enterprise-scale Kafka architecture aligned with Bank Lakehouse (Medallion - Bronze/Silver/Gold).
- Define event-driven architecture patterns for real-time ingestion and processing.
- Establish topic strategy, partitioning, replication, retention policies.
- Architect multi-cluster Kafka setup (dev/test/prod, DR strategy).
- Ensure exact point-in-time recovery and fault tolerance.
2. Platform Implementation
- Lead setup and configuration of:
- Kafka brokers, Zookeeper / Kraft mode
- Kafka Connect framework
- Schema Registry
- Stream processing (Kafka Streams / Spark Streaming / Flink)
- Integrate Kafka with:
- Cloudera Data Platform (CDP)
- Informatica (for ETL/ELT)
- Data Warehouse (Teradata)
- Denodo (for virtualization layer)
3. Real-Time Data Engineering
- Design pipelines for:
- Event streaming ingestion
- Near real-time batch ingestion
- Build CDC (Change Data Capture) pipelines using Kafka Connect (Debezium).
- Enable ingestion from source systems .
- Support incremental and historical loads integration.
4. Performance & Scalability
- Ensure:
- High throughput and low latency
- Support for concurrent consumers without degradation
- Optimize:
- Partition strategy
- Consumer group scaling
- Producer configurations (acks, batching, compression)
5. Resilience & Reliability
- Implement:
- Failover strategies
- Data replication and durability (<99.9% uptime requirement)
- Handle:
- Hardware failures
- Data corruption scenarios
- Design retry, dead-letter queues (DLQ), error handling frameworks
6. Data Governance & Security
- Enforce:
- Role-Based Access Control (RBAC)
- Encryption (in transit & at rest)
- Data masking/tokenization support (aligned with classification)
- Integrate with:
- Metadata and lineage tools
- Governance frameworks (NDMO/NDI compliance)
7. Monitoring & Operations
- Implement monitoring using:
- Cloudera Manager
- Kafka metrics (JMX, Prometheus, Grafana)
- Define:
- Alerting thresholds
- SLA monitoring for ingestion pipelines
- Ensure observability for:
- Lag monitoring
- Throughput and error rates
8. Phase 1 Remediation Support
- Assess existing Kafka pipelines and components.
- Identify:
- Defective pipelines
- Missing ingestion flows
- Redesign/rebuild where required.
- Ensure alignment with target architecture and data models (FSLDM/FSAS).
9. Reporting Enablement Support
- Ensure:
- Real-time data availability for downstream reporting
- Support:
- Query performance optimization
- Data consistency across Bronze/Silver/Gold
- Resolve issues impacting reporting (data mismatches, delays).
10. Documentation & Governance
- Deliver:
- High-level architecture design
- Low-level design (LLD) for Kafka pipelines
- Topic catalog and schema definitions
- Maintain:
- Data lineage documentation
- Source-to-target mappings
- Participate in:
- Architecture governance boards
- Change management processes
Key Technical Skills
Core Technologies
- Cloudera Kafka (mandatory)
- Kafka Connect, Kafka Streams
- Schema Registry (Avro/Protobuf/JSON)
Ecosystem Tools
- Cloudera (CDP)
- Apache Spark / Spark Streaming
- Apache Flink (preferred)
- Informatica (integration awareness)
- Teradata
- Denodo
Data & Integration
- CDC tools (Debezium preferred)
- REST APIs, Microservices integration
- ETL/ELT Design patterns
DevOps & Infra
- Kubernetes / Docker (optional but preferred)
- CI/CD for pipeline deployment
- Linux environments
Banking Domain Experience (Mandatory)
- Experience in BFSI domain
- Understanding of:
- Regulatory reporting
- Data reconciliation (≥99% accuracy)
- AML, Payments, Trade finance systems
Soft Skills
- Strong stakeholder management (business + IT)
- Ability to work in multi-vendor environment
- Experience in large-scale transformation programs
- Strong problem-solving and RCA skills
