Data Engineer

What you'll do 

Working with members of the Technology team you will:

• design, develop and maintain data pipelines and ETL/ELT processes
• implement Lakehouse and Warehouse structures
• develop PySpark notebooks for data transformation and enrichment
• design and implement data models optimised for downstream analytics (e.g. star/snowflake schemas)
• optimise Lakehouse performance with partitioning, indexing and caching strategies
• ensure integration solutions comply with relevant standards and regulations, including information security best practices
• maintain a good understanding of the technologies that are relevant to [Employer hidden] data and analytics initiatives
• support key stakeholders and users
• contribute to CI/CD automation pipelines using Azure DevOps or GitHub actions
• contribute to the establishment and maintenance of agreed standards, patterns and designs applicable to the work undertaken and to ensure that they are adhered to
• contribute to data quality, lineage and cataloguing initiatives to ensure enterprise-wide trusted datasets
• occasional support of legacy data integrations e.g. SSIS.

What you'll need

Essential:

• ETL/ELT design with a focus on incremental and streaming data pipelines
• Azure Data Services (Data Factory, Synapse Analytics, Event Hubs, and Logic Apps)
• PySpark, Spark SQL, and Python for data processing and transformations
• knowledge of Data Lake, Delta Lake, Parquet and OneLake storage concepts
• SQL Server and T-SQL
• experience with data security, encryption and compliance standards
• excellent teamwork skills for collaboration with highly technical colleagues
• the ability to translate business requirements into technical data solutions
• familiarity with testing techniques used to plan and execute software tests of all application components (functional and non-functional) to verify that the software satisfies specified requirements and to detect errors.

Desirable:

• Fabric (Data Engineering, Data factory and Lakehouse). Training can be provided
• familiarity with machine learning integration and predictive analytics pipelines
• power BI integration, including semantic models and DirectLake
• KQL (Kusto Query Language) for Eventstream and real-time analytics.

What you'll receive

• A combined pension contribution of up to 20%.
• Career progression and professional development opportunities.
• 25 days' holiday rising to 28 with length of service.
• The opportunity to sell up to five days of holiday every year.
• The opportunity to buy up to ten days of holiday each year (subject to conditions).
• A healthcare package that allows you to claim back healthcare costs.
• Life assurance of up to eight times your salary.
• The opportunity to lease a new electric car through salary sacrifice (subject to conditions).
• Cashback and discounts from more than 3,000 retailers.
• One paid volunteering day each year.
• Enhanced family leave and pay arrangements.
• Access to an interactive health and wellbeing platform.
• Support from trained mental health first aiders.
• A £1,000 referral fee if you recommend someone who is successfully recruited by us. 

Who we are

[Employer hidden] is part of the international YTL Group based in Kuala Lumpur. The UK companies include:

• Wessex Water – one of the top-performing water and sewerage companies in England and Wales, serving 2.9 million people across the South West
• YTL Developments – a major UK developer currently redeveloping a 350-acre former airfield into an award-winning, exciting mix of houses, apartments, schools, commercial space, restaurants and hotels, to make a truly sustainable new community
• YTL Construction UK – a top 20 UK contractor providing fully integrated services to infrastructure, residential, commercial, industry, energy and environmental sectors
• YTL Arena – the development and operation of an entertainment complex that includes a 20,000 capacity arena, conferencing and exhibition space
• Plus, a number of other retail, environmental and specialist businesses.

Our people tell us that [Employer hidden] is a great place to work, which is why so many of them stay with us! You will have a unique opportunity to develop and progress your career within such a diverse group.

We are passionate about diversity and inclusion – with that in mind, all applicants are welcome. We are delighted to have signed the Armed Forces Covenant and are a Disability Confident Employer.

If you require reasonable adjustments to be made during the recruitment process, please inform a member of our Recruitment team.