Accessibility Links

Big Data Engineer - Spark, SQL, Hive, HDFS, Python, Oozie

  • Job reference: DO32674
  • Location: Brighton
  • Job type: Contract
  • Start date: Not specified
  • Contact: Not specified
  • Sector: BI, Data & Analytics
  • Salary: £300 - £350 per day

Sorry, this vacancy has now expired.

Please see our job results page to find our current open vacancies or take a look at our Specialisms below and go straight to your area.

BI, Data & Analytics, Change Management, Digital and Development, ERP and CRM Systems, Executive Leadership, Information and Cyber Security, Infrastructure & Cloud Services, Interim Management, Strategy & Architecture, or Work For La Fosse.




 Return to homepage.

Big Data Engineer - Spark, SQL, Hive, HDFS, Python, Oozie

We are looking for some incremental Big Data Engineers to support the CS work we are doing for Regulatory Reporting - Issues Management and KYCC.


The Engineer/s will be responsible for the designing and building of scalable data pipelines and processing applications on BIG Data Hadoop platform. This role will be involved in all phases of the software development life cycle including coding, testing and operational maintenance activities. The candidate must be passionate about data, innovative software development, cloud solutions and agile methodologies. They must be eager to advance their and the teams knowledge, and have a proven ability to deliver quality products.


  • Ownership and development of data pipelines including data ingestion, transformation and presentation following Agile methodology.
  • Development of the data platform to enable scalable pipelines including support and monitoring.
  • Hands on technical role contributing to all phases of the software development lifecycle including analysis, design, implementation, unit testing and QA.
  • Performance enhancements to code to improve speed and scalability.
  • Build internal knowledge base by documenting best practises, standards and governance.

Essential Knowledge and Skills:

The Engineer/s must be able to demonstrate advanced skills and knowledge at the depth and scope required for the position.

  • Bachelor's Degree in Computer Science, Engineering, or related discipline
  • Good experience in Hadoop technologies

Must haves

  • Experience in following software packages: - All of below skills must have
  • Spark, Spark SQL
  • Hive
  • Oozie or Similar
  • HDFS
  • Python
  • Advanced SQL

Basic knowledge must have

  • Unix shell scripting
  • Knowledge and experience of using Cloud based technology
  • Experience with Test Driven Development
  • Knowledge and understanding of Java programming language
  • Experience with source code control tools such as Git to manage production and development code
  • Agile Experience using Scrum or Kanban
  • Understanding of reporting tools such as Tableau
  • Excellent analytical and problem solving skills
  • Good prioritisation skills and be flexible enough to adapt
  • A focus on end goals and good team-playing mentality
  • An ability to work to tight deadlines and within constraints
  • Familiarity with the Atlassian and Rally suite
  • Ability to interact comfortably with the other members of a large, geographically dispersed team

Desirable Knowledge and Skills

  • Apache Flume
  • Experience with enterprise relational database systems like Oracle, DB2
  • Understanding of IBM Mainframe
  • Financial AML knowledge
  • Knowledge of secure software coding practices
  • Working experience of a defect tracking syste

If interested, please contact me directly or apply via the link!

Related jobs
Principal Data Engineer - Python
  • Contract
  • City of London
  • £800 - £900 per day
  • Reference 34191
  • I'm currently looking for a Principal Data Engineer who has a strong background using Python, working with demanding data scientist communities and incorporating algorithms into new frameworks. Technical requirements: Expert level experience with PythonExperience with Spark streaming, Kafka, Hive/Impala...
Read more
Senior BI Engineer
  • Contract
  • London
  • £500 - £600 per day
  • Reference 36248
  • Senior BI Engineer A Senior BI Engineer is required for a London based client. Must have experience with SQL, data modelling and the design & build of EDW (enterprise Datawarehouse) You will be expected to have strong experience in: Strong experience in data warehousing & BIStrong experience in dimensional...
Read more
Data Architect - Cloud/Big-Data
  • Contract
  • Barnet
  • £700 - £800 per annum
  • Reference LFA 35968
  • Big Data Architect - Cloud/Big-Data - Barnet Urgent requirement for a Data Architect (technical) to work as part of a digital transformation. Experience with both designing big-data solutions in cloud environments (AWS/Azure), and with cloud-hosted technologies, is hugely desirable...
Read more