Accessibility Links

Big Data Engineer - Spark, SQL, Hive, HDFS, Python, Oozie

  • Job reference: DO32674
  • Location: Brighton
  • Job type: Contract
  • Start date: Not specified
  • Contact: Not specified
  • Sector: BI, Data & Analytics
  • Salary: £300 - £350 per day

Sorry, this vacancy has now expired.

Please see our job results page to find our current open vacancies or take a look at our Specialisms below and go straight to your area.

BI, Data & Analytics, Change Management, Digital and Development, ERP and CRM Systems, Executive Leadership, Information and Cyber Security, Infrastructure & Cloud Services, Interim Management, Strategy & Architecture, or Work For La Fosse.




 Return to homepage.

Big Data Engineer - Spark, SQL, Hive, HDFS, Python, Oozie

We are looking for some incremental Big Data Engineers to support the CS work we are doing for Regulatory Reporting - Issues Management and KYCC.


The Engineer/s will be responsible for the designing and building of scalable data pipelines and processing applications on BIG Data Hadoop platform. This role will be involved in all phases of the software development life cycle including coding, testing and operational maintenance activities. The candidate must be passionate about data, innovative software development, cloud solutions and agile methodologies. They must be eager to advance their and the teams knowledge, and have a proven ability to deliver quality products.


  • Ownership and development of data pipelines including data ingestion, transformation and presentation following Agile methodology.
  • Development of the data platform to enable scalable pipelines including support and monitoring.
  • Hands on technical role contributing to all phases of the software development lifecycle including analysis, design, implementation, unit testing and QA.
  • Performance enhancements to code to improve speed and scalability.
  • Build internal knowledge base by documenting best practises, standards and governance.

Essential Knowledge and Skills:

The Engineer/s must be able to demonstrate advanced skills and knowledge at the depth and scope required for the position.

  • Bachelor's Degree in Computer Science, Engineering, or related discipline
  • Good experience in Hadoop technologies

Must haves

  • Experience in following software packages: - All of below skills must have
  • Spark, Spark SQL
  • Hive
  • Oozie or Similar
  • HDFS
  • Python
  • Advanced SQL

Basic knowledge must have

  • Unix shell scripting
  • Knowledge and experience of using Cloud based technology
  • Experience with Test Driven Development
  • Knowledge and understanding of Java programming language
  • Experience with source code control tools such as Git to manage production and development code
  • Agile Experience using Scrum or Kanban
  • Understanding of reporting tools such as Tableau
  • Excellent analytical and problem solving skills
  • Good prioritisation skills and be flexible enough to adapt
  • A focus on end goals and good team-playing mentality
  • An ability to work to tight deadlines and within constraints
  • Familiarity with the Atlassian and Rally suite
  • Ability to interact comfortably with the other members of a large, geographically dispersed team

Desirable Knowledge and Skills

  • Apache Flume
  • Experience with enterprise relational database systems like Oracle, DB2
  • Understanding of IBM Mainframe
  • Financial AML knowledge
  • Knowledge of secure software coding practices
  • Working experience of a defect tracking syste

If interested, please contact me directly or apply via the link!

Related jobs
Business Intelligence Developer
  • Permanent
  • London
  • £55000 - £60000 per annum + Bonus
  • Reference 39136
  • I am currently working with a fast growth, technology driven disruptor of the Fin-Tech industry who are looking for a technically gifted Business Intelligence Developer to develop, manage and enhance their BI solutions. They are looking for: A Business Intelligence Developer with a strong background in Microsoft BI...
Read more
Data Engineer - Google Cloud Platform
  • Contract
  • City of London
  • £650 - £700 per day
  • Reference 38541
  • I'm currently looking for a Data Engineer who has experience with Google Cloud Platform as well as designing and building applications to process data in the Hadoop ecosystem. Technical requirements: Experience with Google CloudHDFSHadoopExpert level experience with Python, Scala or JavaExperience with Spark...
Read more
Big Data Engineer - Scala / AWS / Spark
  • Contract
  • London
  • £600 - £700 per day
  • Reference LFA - 38133
  • Big Data Engineer - Scala / AWS / Spark Big Data Engineer with experience with Scala, AWS and Spark is required for a client based in London. Must have experience designing real-time analytics data pipelines, designing data schemas and developing AWS environements. Experience overview: ScalaSparkAWSHiveElasticSearch...
Read more