Responsibilities
- Create new and maintain existing Scala/Spark jobs for data transformation and aggregation from simple to complex data transformations involving structured and unstructured data.
- Produce unit tests for Spark transformations and helper performance optimisation methods.
- Develop data processing pipelines, data storage, and management architecture.
- Define scalable calculation logic for interactive and batch use cases
- Interact with infrastructure and data teams to produce complex analysis across data
- You'll be working on a unique and challenging big data ecosystem with a focus on storage efficiency, data security and privacy, scalable and performant queries, expandability and flexibility, etc., and to help better measure the quality of map data.
- You will work with engineers to build a big data platform that processes and manages exabytes of data and enables efficient access to that data.
Requirements
- Minimum 7 years of working experience in Big Data and Hadoop platforms, and building large-scale distributed systems with high availability.
- Experience in developing Spark Applications using Spark RDD API, Hadoop, Spark SQL, Spark Streaming API, Spark MLlib API, and DataFrames APIs
- Should have broad knowledge of Spark advantages, Spark workflows, how to write Spark jobs, Spark query tuning and performance optimisation.
- Solid in data structures and algorithm basics.
- Should have good hands-on experience with any one programming language (Scala/Java 8 - 1st preference, OR Python - 2nd preference OR Java - 3rd preference).
- Strong investigative and problem-solving skills.
- Data ingestion, optimisation techniques, data transformation and aggregation pipeline design/development knowledge are required.
- Experience working on cutting-edge Big Data storage systems and technologies like Hadoop, HDFS, AWS S3 AWS Lambda, Storm/Heron, Cassandra, Apache Kafka, Solr/ElasticSearch, MongoDB, DynamoDB, Postgres, MySQL, etc.
This job was posted by Ambica Badghaiya from Unify.
