| Description :
|
JOB DESCRIPTION: - Big Data Engineer Designing and implementing highly performant data ingestion pipelines from multiple sources using Spark/Scala in Hadoop
- Direct experience of building data pipelines using Spark/Scala
- Developing scalable and re-usable frameworks for ingesting of data sets
- Integrating the end to end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times
- Working with other members of the project team to support delivery of additional project components (API interfaces, Search)
- Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.
- Strong knowledge of Data Management principles
- Knowledge on Azure cloud, ADLS is a plus
|