Data Engineer
SKILLS
FULL DESCRIPTION
Data Engineer
[Employer hidden — view at passion-project.co.uk]
London
£65-75k
Mid and Senior level
2+ days a week in office
Who you are
- Proven experience as a Data Engineering using SQL and Python
- Previous experience with data lakes in AWS, Glue Catalog and Athena (or equivalent)
- Good understanding of Spark, optimisation and performance tuning
- Capable of using popular data modelling tools to create a diagram of proposed tables to enable discussion
- Good communicator and comfortable with presenting ideas and outputs to technical and non-technical users
- Worked on Apache Airflow before to create DAGS
- Ability to work within Agile, considering minimum viable products, story pointing and sprints
What the job involves
At the heart of our Data Team, Data Engineers play a pivotal role by creating pipelines and tables that power impactful dashboards, enable self-service via SQL, and support innovative machine learning models and real-time data products
As a Data Engineer, you will be involved in engineering pipelines that will drive key decisions and give data science powerful datasets to enable and drive new business insights
Data Engineers work alongside Machine learning engineers, BI Developers and Data Scientists in cross-functional teams with key impacts and visions
Using your skills with SQL, data modelling and Spark to ingest and transform high volume complex raw event data into user-friendly high impact tables
As a department we strive to give our Data Engineers have high levels of autonomy and freedom to innovate and continually refine their technical and soft skills with clear progression plans and training opportunities with Data Camp!
Be key to making our data lake more accessible and insightful breaking down the barriers to access by working on data marts and designing data models that even the most basic SQL users can use
- Connect with team at standup to catchup on the latest
- Build data pipelines with Spark or DBT on Starburst
- Use SQL to transform data into meaningful insights
- Build and deploy infrastructure with Terraform
- Implement DDL, DML with Iceberg
- Do a code review for your peers
- Orchestrate your pipelines with DAGs on Airflow
- Participate in SCRUM ceremonies (standups, backlogs, demos, retros, planning)
- Secure data with IAM and AWS Lake formation
- Deploy your changes with Jenkins and GitHub actions