Lead Data Engineer

Lead Data Engineer

Employment Type: Permanent

Location: London - Hybrid (3 days in the office)

Salary: £95,000 - £110,000 + benefits & bonus

About the Client

Our client is a leading specialist insurance provider, recognised for its innovative approach to technology & data. Operating globally, they are investing heavily in their modern data platform, leveraging cloud technologies to deliver scalable, high-quality data solutions that support underwriting, analytics & business decision-making.

Role Overview

As a Lead Data Engineer, you'll play a key role in designing & evolving a modern cloud-based data platform built on Databricks & Azure. You'll lead the development of scalable data pipelines using Python & PySpark, enabling high-quality data to power underwriting, analytics & business-critical decision-making across the organisation.

Working closely with Product Owners, Architects & business stakeholders, you'll provide technical leadership throughout the full data lifecycle. Alongside remaining hands-on, you'll mentor a team of Data Engineers, champion engineering best practices & help shape the future of the organisation's cloud data platform.

Key Responsibilities:

• Lead & mentor a team of Data Engineers, supporting the successful delivery of data engineering projects from design through to production.
• Design, develop & optimise scalable data pipelines using Python, PySpark & Databricks.
• Build & maintain cloud-native data solutions across the Azure ecosystem, including Azure Functions & related services.
• Collaborate with Product Owners, Architects & business stakeholders to define technical solutions & deliver against business priorities.
• Drive engineering best practices through code reviews, mentoring & continuous improvement initiatives.
• Identify opportunities to improve platform performance, data quality, scalability & operational efficiency.
• Champion automation, testing, CI/CD & modern engineering practices across the data engineering function.

Key Requirements:

• Strong commercial experience building data engineering solutions using Python, PySpark & Databricks.
• Proven experience designing & optimising complex SQL queries & working with large-scale datasets.
• Strong experience across the Azure technology stack, including Azure Functions & cloud-native data services.
• Previous experience leading or mentoring Data Engineers within an Agile environment.
• Strong understanding of distributed data processing, modern data engineering practices & CI/CD.
• Experience building scalable, reliable & high-performing data platforms.
• Experience within Financial Services or Insurance is highly advantageous.

Why Join:

• Opportunity to shape the future of a modern, cloud-native data platform built on Databricks & Azure.
• Hands-on leadership role combining technical delivery with engineering strategy.
• Work with modern technologies including Python, PySpark, Databricks & the wider Azure ecosystem.
• Collaborative engineering culture focused on innovation, quality & continuous improvement.
• Competitive salary, bonus & comprehensive benefits package.
• Hybrid working from a modern London office.

Data & Application Notice

We process personal data submitted as part of your application for recruitment purposes in line with UK GDPR. All applications are reviewed by a member of our team, we do not use automated decision-making in our selection process.
Your application will be assessed against the requirements of this role.

By applying to this advert, you consent to receiving future relevant roles & industry news & insights from us. To opt out of this, please contact: compliance@pioneer-search.com.

For more information on how we use your data, including how long we retain it & your rights, please refer to our Privacy Policy: www.pioneer-search.com/privacy-policy.