The client is a global leader in the development and management of solar energy projects, their purpose is to deliver affordable and sustainable solar power for businesses and communities around the world.
As a Data Engineer, you will be responsible for architecting, building, launching and maintaining scalable data pipelines to support our data processing, analytics and product needs. You will own the systems that collect and transform data from several operational sources, creating data integration and transformation services and APIs. You will also maintain ongoing reliability, performance, and support of the data infrastructure. This includes monitoring the computing environments and providing solutions based on application needs and anticipated growth.
DUTIES AND RESPONSIBLITIES
- Evangelize high quality software engineering practices to create and maintain optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Identify, design, and implement business process improvements by automating manual processes, optimizing data delivery and re-designing infrastructure for better performance and greater scalability.
- Ensure reliable, accurate and consistent datasets appropriately serve the business demand.
- Work alongside analytics analysts to ensure datasets surface meaningful, actionable and timely insights.
- Forge strong partnerships with busines stakeholders and strive for greater functionality in our data systems.
- Provide excellent customer focused support by reviewing and troubleshooting data issues.
- Contribute to data governance initiatives to define and implement policies for data modelling, quality, security, architecture, change management and devops.
- Implement robust practices and procedures to ensure compliance with data protection and privacy obligations.
- Develop and maintain standard operating procedures, training guides and data catalogues.
Qualifications, Knowledge and Experience
- Bachelor's degree in computer science, software engineering or equivalent
- Extensive experience designing, building and launching highly available, distributed data platforms
- Working knowledge of message queuing, stream processing and 'big data' stores
- Strong software engineering principles, including continuous integration and deployment pipelines and technologies, preferably Azure DevOps and GIT
- Proficiency with cloud services, ideally Azure and AWS
- Advanced database expertise is essential, specifically Snowflake, Cassandra and HBase
- Expert SQL skills, and fluidity with modern scripting language is essential (Python and Node.js)
- Extensive experience working with Azure Data Factory and Logic Apps
- Proficiency with Big Data systems such as Hadoop, Spark, Databricks, and technologies such as Azure Functions
- Experience designing and developing solutions that employ microservices architecture and containerization approaches for packaging applications, operating systems and infrastructure (preferably Docker)
- Familiarity managing data platform infrastructure as code (Terraform, Ansible or Chef)
- Familiarity with BI tools such as PowerBI and Tableau
- Experience handling sensitive data