Key Responsibilities

Data Architecture & Engineering

Design, develop, and manage end-to-end data pipelines using Microsoft Fabric (Data Pipelines, Lakehouse, Warehouse, OneLake) and Azure Data Factory (ADF)
Build scalable ETL/ELT solutions using PySpark, Python, SQL, and Fabric Notebooks
Implement Lakehouse and Medallion architecture (Bronze, Silver, Gold layers) for analytics workloads

Data Processing & Transformation

Develop optimized SQL-based solutions including T-SQL, stored procedures, and performance-tuned queries
Handle data transformation, enrichment, and processing across batch and near real-time pipelines
Implement incremental loads, CDC (Change Data Capture), and watermarking techniques

Cloud Data Platform Engineering

Work extensively with Azure Data Lake Storage Gen2, Azure Blob Storage, and OneLake
Integrate data from multiple sources including RDBMS, APIs, streaming systems, and cloud storage
Enable analytics and reporting through integration with Power BI and Fabric datasets

Streaming & Real-Time Data (Preferred)

Work with streaming or near real-time data pipelines using Azure Event-based services
Design event-driven data ingestion architectures where required

Data Governance & Quality

Implement data quality checks, monitoring, logging, and error handling frameworks
Ensure adherence to governance, security, and compliance standards
Work with Microsoft Purview for data lineage and governance (preferred)

DevOps & CI/CD for Data Engineering

Support CI/CD implementation for data pipelines using Azure DevOps and Git
Collaborate with DevOps teams for deployment, version control, and release management
Monitor pipeline performance and ensure reliability across environments

Migration & Modernization

Support migration from Azure Data Factory and Databricks to Microsoft Fabric where applicable
Contribute to enterprise-scale data platform modernization initiatives

Leadership & Collaboration

Work closely with architects, analysts, and business stakeholders to translate requirements into scalable solutions
Provide technical guidance and mentorship to junior engineers
Drive best practices in data engineering, architecture, and performance optimization

Lead Data Engineer - R01558812