Summary
The Azure DevOps Engineer plays a crucial role in modernizing data operations by focusing on implementing Continuous Integration and Continuous Deployment (CI/CD) for Azure data services.
Main Responsibilities
The core responsibilities include:
- Design, build, and maintain CI/CD pipelines for Azure Databricks and Azure Data Factory.
- Collaborate with data engineers and software development teams to integrate code changes into the deployment pipeline.
- Automate deployment processes and workflows to enhance efficiency and reduce manual intervention.
- Monitor and troubleshoot issues in the CI/CD pipeline, ensuring minimal downtime and quick resolution.
- Implement best practices for asset bundling and version control within Azure DevOps.
- Maintain and optimize the Azure DevOps environment for scalability and performance.
- Ensure security and compliance standards are met in all deployment processes.
- Document processes, procedures, and configurations for CI/CD pipelines and Azure DevOps.
Key Requirements
- Proven experience with Azure DevOps and implementing CI/CD pipelines.
- Proficiency in Databricks Asset Bundles (DABs).
- Strong knowledge of Azure data services, specifically Azure Databricks and Azure Data Factory.
- Experience with scripting and automation tools (e.g., PowerShell, Python).
- Minimum of 5 years of experience in DevOps or related roles.
Nice to Have
- Understanding of containerization and orchestration tools (e.g., Docker, Kubernetes).
- Experience in implementing selective deployment for ADF data pipelines.
- Excellent problem-solving skills and ability to troubleshoot complex CI/CD issues.
- Knowledge of security best practices for CI/CD pipelines and Azure environments.
Other Details
This position involves collaboration with cross-functional teams and requires a Bachelor's degree in Computer Science, Information Technology, or a related field.
