Data Engineer - Operational Data

We are currently looking for a Mid-Senior Data Engineer to join us in our growing Data Engineering team; to help develop, maintain, support, and integrate our growing number of data systems. You will be instrumental in designing, building, and maintaining robust and scalable data pipelines that power our operational subscribers and analytical platforms. You will work with a diverse range of data sources, integrating both real-time streams and micro-batches, connecting with varied end points to move data at speed and at scale.

The right candidate will have a wealth of knowledge in the data world with a strong focus on Databricks, and will be keen to expand upon their existing knowledge, learning new technologies along the way as well as supporting both future and legacy technologies and processes.

You will be coding, testing, and documenting new or modified data systems; creating scalable, repeatable, secure pipelines and applications for both operational data and analytics, both internally and externally to the business. You will grow our capabilities, solving new data problems and challenges every day.

Key Responsibilities:

• Design, Build, and Optimise Real-Time Data Pipelines: Develop and maintain robust and scalable stream and micro-batch data pipelines using Databricks, Spark (PySpark/SQL), and Delta Live Tables.
• Implement Change Data Capture (CDC): Implement efficient CDC mechanisms to capture and process data changes from various source systems in near real-time.
• Master the Delta Lake: Leverage the full capabilities of Delta Lake, including ACID transactions, time travel, and schema evolution, to ensure data quality and reliability.
• Champion Data Governance with Unity Catalog: Implement and manage data governance policies, data lineage, and fine-grained access control using Databricks Unity Catalog.
• Enable Secure Data Sharing with Delta Sharing: Design and implement secure and governed data sharing solutions to distribute data to both internal and external consumers without data replication.
• Integrate with Web Services and APIs: Develop and manage integrations to push operational data to key external services, as well as internal APIs.
• Azure Data Ecosystem: Work extensively with core Azure data services, including Azure Data Lake Storage (ADLS) Gen2, Azure Functions, Azure Event Hubs, and CI/CD.
• Data Modelling and Warehousing: Apply strong data modelling principles to design and implement logical and physical data models for our analytical and operational data stores.
• Monitoring and Performance Tuning: Proactively monitor data pipeline performance, identify bottlenecks, and implement optimizations to ensure low latency and high throughput.
• Collaboration and Mentorship: Collaborate with cross-functional teams, including software engineers, data scientists, and product managers, and mentor junior data engineers

What We're Looking For:

• Proven Databricks Expertise: Extensive hands-on experience with the Databricks Lakehouse Platform is essential.
• Strong Spark and Python/SQL Skills: Proficiency in Spark programming (PySpark and/or Scala) and expert-level SQL skills.
• Real-Time Data Processing: Demonstrable experience in building and managing stream and micro-batch processing pipelines using technologies like Spark Structured Streaming or Delta Live Tables.
• Deep Understanding of Delta Lake Concepts: Thorough knowledge of Delta Lake architecture and features (ACID transactions, time travel, optimization techniques).
• Experience with Databricks Advanced Features: Practical experience with Change Data Capture (CDC), Unity Catalog for data governance, and Delta Sharing for secure data collaboration.
• Web Service and API Integration: A proven track record of integrating data pipelines with external web services and REST APIs.
• Solid Azure Experience: Strong experience with core Azure data services (ADLS Gen2, Event Hubs, Azure Functions).
• Data Modelling and Warehousing Fundamentals: A strong understanding of data modelling concepts (e.g., Kimball, Inmon) and experience with data warehousing principles.
• CI/CD and DevOps Mindset: Experience with CI/CD pipelines for data engineering workloads using tools like Git.
• Excellent Problem-Solving and Communication Skills: The ability to troubleshoot complex data issues and communicate technical concepts effectively to both technical and non-technical audiences.

Desirable

• GCP and BigQuery Knowledge: Utilize your experience with GCP and BigQuery for analytical data workloads and to ensure seamless interoperability within our multi-cloud strategy.

Along with your benefits package we also offer a wide range of perks for our colleagues: 

Reward, Recognition and Opportunities 

Frasers Champion- Our employees are at the heart of our business and we ensure individuals are recognised every single month for their hard work. Frasers Champion is a peer nominated scheme where 8 winners will receive double their pay for a month where they have thought without limits, owned it or been relevant. 

Retail Reconnect – In order to build the planets most admired and compelling brand ecosystem, all employees must understand our business, product and customers. Each financial year, Head Office employees will gain insights by spending two days in one of our stores or the Warehouse. The goal is to learn how the work you do impacts our teams on the frontline, and to bring ideas back to the office which will improve how we work. 

Employee Welfare 

Frasers Fit – Our Everlast Gyms Team are on a mission to make our workforce the best, and fittest on the planet! We run free gym classes for employees as well as discounted memberships to our clubs. Frasers Fit is our wellbeing programme which aims to support and improve colleagues Physical, Financial & Mental wellbeing. The app is accessible for every employee and includes training, nutrition and lifestyle advice- all completely free. 

Retail Trust – We know that its not just about physical health, mental wellness is equally important which is why all of our employees get free access and support from the Retail Trust charity. This includes a 24 hour wellbeing helpline, wellness hub, counselling and financial/legal support. 

What’s next? 

Our Recruitment Team will be reviewing applications and all candidates will receive a response, whether you are successful or unsuccessful. Shortlisted applicants may be asked to confirm a few key details before being booked in for a first stage interview with the Recruiter- this will be behaviourally focussed and centred around how you align with our Culture and Values. If successful we anticipate two further interview stages with the Hiring Manager/wider team which will be more technically focussed and could include a presentation/task so we can see your skills in action.