Platform Engineer

Databricks Platform Lead (AWS)

Location: UAE and Saudi arabia, Europe

About the Role

We are looking for a Senior Databricks Platform Lead to design, build, and manage an enterprise Databricks Lakehouse platform on AWS. This is a hands-on technical leadership role where you will define the platform architecture, improve security, optimize cloud costs, mentor engineers, and work closely with business and technical teams.

Key Responsibilities

• Design and maintain the Databricks Lakehouse platform on AWS.
• Build secure, scalable, and high-performance data platform architecture.
• Manage AWS infrastructure including VPC, IAM, EC2, S3, networking, and security.
• Configure and administer Databricks workspaces, Unity Catalog, clusters, and permissions.
• Develop standards for data ingestion, transformation, and orchestration using PySpark and Delta Lake.
• Build and support batch and streaming data pipelines using Databricks, Kinesis, SNS/SQS, and Auto Loader.
• Create cluster policies, autoscaling, and cost controls to optimize cloud spending.
• Monitor platform health, audit logs, security, and performance.
• Support ML workloads using MLflow and integrations with SageMaker or AWS Bedrock.
• Lead migration of legacy data warehouses and on-premise platforms to Databricks Lakehouse.
• Mentor engineering teams, review designs, and establish coding best practices.
• Work closely with business stakeholders and AWS teams to deliver platform improvements.

Required Skills

• Strong experience with Databricks Platform Administration and development.
• Deep knowledge of AWS including VPC, IAM, EC2, S3, PrivateLink, and networking.
• Hands-on experience with Unity Catalog, SCIM, and data governance.
• Strong PySpark and Delta Lake development experience.
• Experience building large-scale ETL/ELT and streaming data pipelines.
• Knowledge of Kinesis, SNS, SQS, Glue, and AWS DMS.
• Experience managing Databricks clusters, autoscaling, and cloud cost optimization.
• Experience migrating legacy data platforms to cloud-based Lakehouse architecture.
• Strong leadership, mentoring, and stakeholder management skills.

Nice to Have

• MLflow, SageMaker, or AWS Bedrock experience.
• Databricks Certification.
• AWS Solutions Architect or AWS Data Analytics certification.
• Experience with SOC 2 or similar security compliance standards.