Azure Data Engineer Job Description

What is the role of an Azure Data Engineer?

In your role as an Azure Data Engineer, you will be responsible for expanding and optimising data and pipeline architectures, and for optimising data collection and flow across functional teams. Your responsibilities include assisting software developers, database architects, data analysts, and data scientists with data initiatives and ensuring a consistent data delivery architecture is put in place throughout ongoing projects.

Responsibilities

Assure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements

Develop and maintain innovative Azure solutions

Solution design using Microsoft Azure services and other tools

The ability to automate tasks and deploy production standard code (with unit testing, continuous integration, versioning etc.)

Load transformed data into storage and reporting structures in destinations including data warehouse, high speed indexes, real-time reporting systems and analytics applications

Build data pipelines to collectively bring together data

Other responsibilities include extracting data, troubleshooting and maintaining the data warehouse

Tech Stack

Python

SQL and NoSQL databases

Scala

Spark-SQL

Experience with Azure: ADLS, Databricks, Stream Analytics, SQL DW, COSMOS DB, Analysis Services, Azure Functions, Serverless Architecture, ARM Templates

Types of Azure Data Engineer

SQL Developer

Data Engineer

Data Scientist

Artificial Intelligence (AI) Engineer

Qualifications

Previous experience as an Azure Data Engineer or similar role

Experience building and optimising 'big data' data pipelines, architectures, and data sets

Strong analytic skills related to working with unstructured datasets

The ability to design and implement well written code

Ability to work to tight deadlines

Ability to test the data from source to the presentation layer