Data Engineer

ESSENTIAL ROLES & RESPONSIBILITIESIdentify and understand customer data-centric use cases within regulated financial services environmentsDesign and implement data ingestion, processing, and transformation pipelines on AzureBuild and maintain data pipelines for cleaning, normalisation, enrichment, and preparationApply appropriate data modelling techniques and architecture patterns, with a strong focus on medallion architectureOrchestrate, monitor, and optimise Azure Databricks jobs and Azure Data Factory pipelines across development, UAT, and production environmentsConfigure platforms, clusters, and compute resources to optimise performance, cost, and reliabilityUse automated CI/CD pipelines to manage, deploy, and version data artefacts and pipelinesOperationalise workflows developed by analysts and data scientistsSupport customers in adopting Azure data, analytics, and machine learning servicesEnsure secure storage, processing, and quality of customer dataEnsure networking and security best practices are applied when designing and operating data solutionsDesign solutions for processing large volumes of data using batch and streaming approachesCollaborate with analytics teams on data visualisation best practices and reporting enablementEnsure all solutions are well-documented, including pipelines, schemas, transformations, and operational runbooksGOVERNANCE & REPORTINGMaintain accurate documentation of data pipelines, schemas, transformations, and deployment processesSupport data governance initiatives including lineage, metadata management, and access controlContribute to service reporting, risk tracking, and continuous improvement actionsEnsure data environments are audit-ready and aligned with governance standardsTECHNOLOGY STACK (AZURE)Cloud Platform:Microsoft AzureData Engineering & Analytics:Azure Databricks (development, UAT, and production)Azure Data FactoryAzure Synapse Analytics (where applicable)Machine Learning & AI:Azure Machine Learning (limited non-production usage)Azure Document IntelligenceDatabases:Microsoft SQL Server / Azure SQL Database (primary platforms)PostgreSQL (limited use)MySQL (limited use)Data Processing:Batch and streaming data pipelinesSecurity & Governance:Role-based access control (RBAC)Data encryption and key managementAudit logging and monitoringDevOps:CI/CD pipelines for data artefacts and infrastructureBEHAVIOURAL COMPETENCIES - ORGANISATIONAL & BEHAVIOURAL FITPositive mindset and enthusiasm for learning new technologiesCollaborative and supportive team playerStrong sense of ownership and accountabilityMethodical, analytical approach to problem-solvingStrong understanding of ethical data usage in regulated environmentsCRITICAL COMPETENCIES - TECHNICAL FITEssential:Strong SQL skillsProgramming experience with Python and/or ScalaHands-on experience with Azure-based data platformsExperience designing, building, and maintaining data pipelinesStrong understanding of data modelling (relational and analytical), including medallion architectureExperience orchestrating and optimising Databricks and Data Factory workloadsExperience using CI/CD pipelines for data and analytics solutionsStrong awareness of security, networking best practices, GDPR, and PII handlingDesirable:Experience with Azure Databricks in production environmentsFamiliarity with Azure Machine Learning and AI servicesExposure to data visualisation tools (e.g. Power BI)Experience with big data frameworks (Spark, Kafka)Knowledge of data governance, lineage, and metadata toolingSHIFT & WORKING PATTERNStandard business hours, with participation in an on-call rota as requiredOccasional weekend engineering coverage will be required, typically limited to a small number of planned weekends per year to support business continuity, resilience testing, or disaster recovery activities