Big Data Engineer Python

Big Data Engineer 

This role focuses on developing, integrating, and optimising large-scale data flows. It involves close collaboration with data scientists to translate analytical models into practical technical solutions, requiring hands-on expertise in data cleansing, pipeline design, component orchestration, and modern data technologies such as Hadoop, Spark, and API development.

Mission & Responsibilities

● Technical Collaboration: work closely with data scientists to develop effective technical solutions

● Data Optimisation: perform data cleansing, preparation, and integration

● Pipeline Engineering: design and build robust data pipelines

● System Integration: manage component orchestration

● Large-Scale Data Processing: responsible for batch Spark ETL, including data wrangling, joining, and aggregation

Technical Competencies

● AI Ecosystems & Frameworks, including:

○ Modern Data Stack (MDS)

○ On-premises Data Lake technologies (Hadoop, Spark, Hive)

○ Large-scale data processing (Spark)

○ Orchestration tools (Apache Airflow)

○ Containerisation (Docker / Kubernetes)

○ Python programming

● Architecture & Integrations:

○ API development

○ Cloud architectures (AWS, Azure)

Experience with real-time architectures and event streaming infrastructure is advantageous.

Behavioural Competencies

● Collaboration: able to work closely and effectively with data scientists to translate complex analytical needs into functional technical solutions

● Analytical & Organised: precise and methodical approach to data cleansing, preparation, integration, and pipeline design

● Problem Solving: quick and efficient in addressing bottlenecks in data processing or orchestration errors

We require full-time availability (100%), although 80% availability will be considered.

,