Databricks Infrastructure and DevOps Engineer

Job Role Name : Databricks Infrastructure and DevOps Engineer

Key Responsibilities

Infrastructure Management:

-Design, deploy, and maintain Databricks clusters and workspaces on AWS GCC environment

- Manage Infrastructure-as-Code (IaC) using Terraform OR Databricks components using Databricks Asset Bundle (DAB)

-Configure and Optimise AWS services including S3, IAM, VPC, and security groups specifically for Databricks

- Ensure compliance with Confidential Cloud Eligible classification and Sensitive High data requirements

CI/CD Pipeline Management:

-Develop and maintain CI/CD pipelines using SHIP-HATS tools ecosystem

-Configure GitLab for version control and automated deployment workflows

-Implement code quality gates using SonarQube for static code analysis

- Manage security scanning using Fortify on Demand (FOD) for vulnerability assessment

-Oversee dependency scanning and management using Nexus IQ

-Coordinate with Cloud SCAPE for security compliance scanning and remediation

Security and Compliance:

-Implement and maintain Automated Baseline Log Review (ABLR) systems

-Coordinate with Gov Tech Cyber Security Operations Centre (GCSOC) for security monitoring

- Manage Central Accounts Management (CAM) integration for user access control

-Coordinate and resolve Vulnerability Assessment and Penetration Testing (VAPT)findings.

- Ensure compliance with Personal Data Protection Impact Assessment (PDPIA) requirements

-Implement and maintain Vulnerability Management System (VMS)

Platform Operations:

-Monitor system performance and implement optimization strategies

- Manage secrets and encryption key rotation for data protection

-Configure email notifications for data processing status and error alerts

-Implement and maintain data retention policies

-Support SaaS whitelisting processes and approvals

-Coordinate System Security Acceptance Testing (SSAT) activities

Collaboration and Support:

- Work closely with Data Engineers, Data Analysts and security technical team.

-Support multiple data marts

-Provide technical guidance for Raw-to-Tech and Tech-to-Business data pipelines

-Collaborate with various government divisions and stakeholders

-Support training environment setup for user onboarding and skill development

Qualifications

Technical Skills:

- 3+years of experience with Databricks platform administration and optimization

- Strong expertise in AWS services, particularly in GCC environment

-Proficiency in Infrastructure-as-Code tools, preferably Terraform

-Experience with GitLab CI/CD pipeline development and management

-Knowledge of security scanning tools (SonarQube, Fortify, Nexus IQ)

-Understanding of government security frameworks and compliance requirements

-Experience with containerization and orchestration technologies

- Strong scripting skills in Python, Bash, or PowerShell

Security and Compliance:

-Experience with government security clearance processes

-Knowledge of data classification and handling procedures (Sensitive High)

-Understanding of encryption mechanisms and key management

-Familiarity with vulnerability management and remediation processes

-Experience with security monitoring and incident response procedures

Certifications(Preferred):

-Databricks Certified Associate or higher

- AWS Certified DevOps Engineer

Required Experience

-Minimum 3 years of DevOps/Infrastructure engineering experience

-Experience working in government environments, esp in GCC.

-Experience in agile development methodologies