Job Description
The Data Engineer is responsible for finding trends in data sets and developing algorithms to help make raw data more useful to the enterprise. This role requires a significant set of technical skills, including a deep knowledge of SQL and NoSQL database design and multiple programming languages.
Key Responsibilities
- Help data scientists and analytics modelers prepare data for use in analytics models
- Manage, assess, and provision open-data requests
- Define and monitor data retention requirements and their categorization
- Create and maintain optimal data pipeline architecture
- Enable access to internal and external data sources, ensuring data availability
- Set up and provide ongoing operational support for ETL environments (development, test, production)
- Design, implement, and deploy data loaders
- Assist in pulling, filtering, tagging, joining, parsing, and normalizing datasets
- Monitor and optimize performance and stability of automated pipeline components
- Selected data engineers may handle L3 support tickets for IT operations as needed
- Develop ETL (Extract, Transform, Load) processes to integrate data from multiple sources
- Implement security measures to protect data at rest and in transit
Key Accountabilities
- Maintain the integrity and security of data pipelines and architecture
- Optimize data workflows to ensure efficient data processing
Key Qualifications
- Experience with Big Data platforms such as AWS, Azure, Spark, CouchDB, Hive, Pig, etc.
- Experience integrating data into analytical platforms, including ingestion technologies, data profiling, source-to-target mappings, ETL development, SQL optimization, testing, and implementation
- Experience building processes for data transformation, data structures, metadata, dependency, and workload management
- Experience working with cross-functional teams in a dynamic environment
- Strong analytical skills, especially with unstructured datasets
