Project Role : Custom Software Engineer

Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact.

Must have skills : Data Architecture Principles

Good to have skills : NA

Minimum 5 Year(s) Of Experience Is Required

Educational Qualification : 15 years full time education

Summary:

We are looking for an experienced Data Architect / Senior Data Engineer with expertise in modern cloud data platforms, data engineering, and emerging AI technologies. The ideal candidate will have strong hands-on experience in AWS, Databricks, PySpark, Python, Snowflake, and AWS Glue, along with exposure to Generative AI, Large Language Models (LLMs), and Agentic AI frameworks.

This role will be responsible for designing scalable data architectures, building enterprise data pipelines, enabling AI-ready data platforms, and providing technical leadership across data engineering initiatives. The candidate should be comfortable working across architecture, development, optimization, and mentoring teams while collaborating with business and technology stakeholders.

Roles & Responsibilities:

Design and implement scalable cloud-based data architectures to support enterprise analytics, reporting, AI, and machine learning workloads.

Develop, optimize, and maintain ETL/ELT pipelines using PySpark, Python, Databricks, AWS Glue, and Snowflake.

Design and implement modern data lake, lakehouse, and data warehouse solutions on AWS.

Build robust ingestion frameworks for structured, semi-structured, and unstructured data from multiple enterprise systems.

Design reusable data engineering frameworks and standards for scalable data processing.

Optimize Spark workloads, SQL queries, and Snowflake objects for performance and cost efficiency.

Implement data quality, governance, lineage, metadata management, and security best practices.

Collaborate with Data Scientists, AI Engineers, Application teams, and Business stakeholders to deliver data solutions for analytics and AI use cases.

Support AI initiatives by preparing high-quality datasets for Generative AI and Machine Learning applications.

Develop and integrate Python-based solutions for AI-enabled data processing and workflow automation.

Contribute to the design and implementation of LLM-powered applications, Agentic AI workflows, Retrieval-Augmented Generation (RAG), and intelligent automation use cases where applicable.

Lead architecture discussions, conduct technical reviews, and establish engineering best practices.

Build and maintain CI/CD pipelines and automate deployment of data engineering solutions.

Monitor production environments, troubleshoot issues, and drive continuous platform improvements.

Mentor data engineers and promote coding standards, performance optimization, and architectural best practices.

Professional & Technical Skills:

8+ years of experience in Data Engineering, Data Architecture, or Cloud Data Platforms.

Strong hands-on expertise in Python and PySpark.

Extensive experience with Databricks.

Strong experience with AWS services including AWS Glue, S3, Athena, IAM, Lambda, CloudWatch, Step Functions, Lake Formation, and Redshift.

Hands-on experience with Snowflake including data loading, transformation, optimization, and performance tuning.

Strong SQL development and query optimization skills.

Experience designing scalable ETL/ELT frameworks and enterprise data pipelines.

Strong understanding of Data Lake, Lakehouse, and Data Warehouse architectures.

Experience with Git, CI/CD, and DevOps practices.

Knowledge of data modeling, partitioning strategies, metadata management, and data governance.

Experience working in Agile delivery environments.

Preferred Skills

Working knowledge of Generative AI, Large Language Models (LLMs), and Agentic AI concepts.

Experience with AI orchestration frameworks such as LangChain, LangGraph, Semantic Kernel, or similar frameworks.

Exposure to Retrieval-Augmented Generation (RAG), vector databases, embeddings, and prompt engineering.

Experience integrating LLM APIs from platforms such as Azure OpenAI, OpenAI, or Amazon Bedrock into enterprise applications.

Knowledge of MLOps and AI deployment best practices.

Experience with Delta Lake, Apache Iceberg, or similar table formats.

Exposure to streaming technologies such as Kafka or Spark Structured Streaming is an added advantage.

Infrastructure as Code using Terraform or CloudFormation is desirable.

Experience leading technical teams and mentoring engineers.

Additional Information:

Position: Associate Manager (Level 8)

Experience: 8–12 Years

Education: Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.

Domain experience in Banking, Financial Services, Healthcare, Retail, Manufacturing, Utilities, or Pharma will be an added advantage.