Data Engineer
Contract Data Engineer - Specialty Insurance
Contract Length: 6 Months
Location: Hybrid / London
Rate: £475 - £500 per day
IR35 Status: Outside IR35
Start Date: ASAP
An experienced Data Engineer is required to take ownership of the build of a data pipeline to consolidate bordereaux reporting data into a standardised, reporting-ready structure. This is a hands-on delivery role sitting within the BI team, where you will act as the sole Data Engineer on the project, responsible for end-to-end pipeline development, design, & delivery.
You will be building scalable Azure Databricks-based data pipelines, working independently to define & implement robust, production-ready solutions that support reporting & downstream analytics.
Key Responsibilities
- Take ownership of the end-to-end build of a scalable ETL pipeline in Azure Databricks to ingest & consolidate bordereaux reporting data
- Design & implement data ingestion processes leveraging existing email-based ingestion tooling for multiple file formats from shared mailboxes
- Develop Databricks notebooks, jobs, & pipelines to support data transformation & orchestration using a medallion architecture approach
- Build & implement data validation, quality controls, logging, monitoring, & error handling to ensure reliable pipeline performance
- Integrate solutions with Azure DevOps CI/CD pipelines across development, test, & production environment
Key Requirements:
- Strong hands-on experience as a Data Engineer (5+ years required)
- Proven experience with Azure Databricks, including building ETL pipelines & working with Delta Lake / Lakehouse architecture
- Strong proficiency in Python, PySpark, & Spark S
- Knowledge of Unity Catalog & modern Databricks governance & platform feature
- Experience implementing CI/CD pipelines using Azure DevOps
- Experience supporting BI/reporting layers such as Power BI or similar tools
- Insurance experience is advantageous
Data & Application Notice
We process personal data submitted as part of your application for recruitment purposes in line with UK GDPR. All applications are reviewed by a member of our team, we do not use automated decision-making in our selection process.
Your application will be assessed against the requirements of this role.
By applying to this advert, you consent to receiving future relevant roles & industry news & insights from us. To opt out of this, please contact: compliance@pioneer-search.com.
For more information on how we use your data, including how long we retain it & your rights, please refer to our Privacy Policy: www.pioneer-search.com/privacy-policy.
