Senior Databricks Engineer

Job Information

- Contract Type Permanent

- Department [Employer hidden — view at passion-project.co.uk] Advanced

- Required Skill []

- Number of Positions 1

- City Mansion House

- Province Greater London

- Postal Code EC4

- Country United Kingdom

- Job Type NA

- Industry Technology

Job Description/ Summary

Who we are

[Employer hidden] is a next-generation technology consultancy, designed to accelerate delivery and build amazing teams. We deliver services across data, AI, and cloud and we’re building the next generation of tech leaders. Since 2017, we have established a market leading position supporting our clients build their data and technology teams and deliver enduring solutions.

The Role

We are seeking a skilled and experienced Data Engineer to join our growing Databricks community of developers. A successful applicant will have a strong background in delivering scalable, cloud-based data solutions, particularly within the Databricks platform. Additionally, they will have experience in designing, building, and optimising data pipelines and architectures, leveraging technologies such as Apache Spark, Delta Lake, and cloud platforms like AWS, Azure, or GCP. Advanced proficiency in Python and SQL is essential, ideally with experience in integrating CI/CD practices and automated testing into data workflows.

As a Squad Leader working in our [Employer hidden] Advanced team, you will play a key role in delivering high-quality data engineering projects to our clients, often in collaboration with Databricks' professional services team. In addition to hands-on technical work, you will often play a leadership role in [Employer hidden]’s delivery squads, providing technical guidance and ensuring best practices are followed throughout the project lifecycle. You will work closely with clients and internal stakeholders to translate business requirements into robust technical solutions, ensuring projects are delivered on time, within scope, and aligned with client expectations.

You will also support the growth and capability development of [Employer hidden] Advanced, particularly with respect to our Databricks delivery capabilities.

Skills & Experience

Proven Experience in Data Engineering: Hands-on experience building and optimising data pipelines, architectures, and large-scale data processing systems, with a focus on cloud platforms such as AWS, Azure, or GCP.

Expertise in Databricks and Spark: Strong working knowledge of the Databricks platform and Apache Spark for developing, scaling, and optimising data engineering workflows, including Delta Lake and Databricks SQL. This knowledge should ideally be evidenced with Databricks Certified Data Engineer Professional qualification.

Proficiency in Programming Languages: proficiency in Python and SQL, with the ability to write clean, efficient, and well-documented code for ETL/ELT processes and data transformations.

Experience with Cloud-based Architectures: Sound understanding of cloud-native data solutions, including working with cloud storage (e.g., AWS S3, Azure Data Lake), data warehousing (e.g., Snowflake, Redshift), and other cloud services to enable scalable data processing.

CI/CD and Automation: Familiarity with CI/CD pipelines and automated testing for data engineering workflows, ensuring efficient and reliable deployment of data pipelines and models.

Data Governance and Security: Experience working within data governance frameworks and security best practices, ensuring that data solutions comply with industry standards and regulations.

Collaborative Mindset: Experience working closely with cross-functional teams, including data scientists, BI developers, and product owners. Proven ability to work within team settings to deliver data projects.

Problem-Solving and Analytical Skills: Strong analytical skills for troubleshooting complex data-related issues, optimizing performance, and ensuring the accuracy and quality of data outputs.

Client-Facing Experience: Demonstrated ability to work directly with clients, translating business requirements into technical solutions, and ensuring alignment with client expectations and project goals.

Key Responsibilities

• Leading technical delivery within [Employer hidden]’s squads deployed on client project engagements.

• Work with internal and client stakeholders to understand and refine business requirements, turning these into functional and non-functional requirements to inform and plan project delivery.

• Seek, build, and maintain effective client relationships contributing to [Employer hidden]’s commercial priorities and supporting our partnering approach to working with clients.

• Participating in self-directed or group learning and upskilling (gaining [Employer hidden] funded certifications) to ensure your technical skills stay up to date and industry relevant.

Our commitment to Diversity, Equity and Inclusion (DEI):

At [Employer hidden], we not only strive to bridge the skills-gap in data and technology, but we are also committed to playing a key role in improving diversity in the industry. To that effect, we welcome candidates from all backgrounds, and particularly encourage applications from groups currently underrepresented in the industry, including women, people from black and ethnic minority backgrounds, LGBTQ+ people, people with disability and those who are neurodivergent.