16.10.2026 aktualisiert
Premiumkunde
100 % verfügbarLead Databricks Data Engineer | Azure | PySpark | Delta Lake | Unity Catalog | GenAI
Kent, Großbritannien
Nur Remote
Bachelor of Engineering, Anna University, India - First Class Honours, 2005Skills
Data ArchitectureDatenbankentwicklungData Access LayerDataOpsAzure Data LakeDatenbank-EnginesEvent-driven ArchitectureAzure Data FactoryApache SparkEvent Driven ArchitectureData LakePySparkTerraformAzure Data AnalyticsAzure Synapse Analytics
Databricks & Delta Lake Engineering
Deep expertise in Databricks, Delta Lake, Unity Catalog, Medallion Architecture and Lakehouse Federation for building scalable, governed and high-performance data platforms across complex enterprise environments.
Azure Data Engineering
Extensive hands-on experience with Azure Data Factory, Azure Functions, Event Hubs, Service Bus, Azure SQL, Cosmos DB, Durable Functions and Logic Apps for designing and delivering end-to-end cloud-native data solutions.
PySpark & Batch/Streaming Pipeline Development
Strong proficiency in PySpark, Python, SQL and C# for developing batch and streaming pipelines, event-driven architectures and data contracts at enterprise scale.
DevOps & Infrastructure as Code
Solid experience with Azure DevOps, Git, Terraform, ARM Templates, Bicep, YAML CI/CD, PowerShell, Azure CLI and Bash for automating deployments and managing infrastructure across multi-environment setups.
Quality Engineering & Automated Testing
Practical application of PyTest, SpecFlow, TSQLt and Moq for automated validation, acceptance testing and code review, ensuring high release confidence and engineering standards.
Database Technologies
Experience with SQL Server, Netezza, Cosmos DB and Azure Data Explorer (KQL) for managing and querying large-scale structured and semi-structured datasets.
GenAI & LLM-Assisted Engineering
Application of GenAI and LLM-assisted approaches to accelerate code analysis, test-case generation, documentation and engineering productivity within data platform modernisation projects.
Kafka & Event-Driven Architecture
Design and delivery of event-driven ingestion and publishing frameworks using Kafka, Azure Event Hubs and Service Bus, supporting millions of records and events at scale.
Legacy Platform Modernisation
Track record of modernising legacy SQL/Synapse-based solutions into Databricks and Delta Lake architectures, including migration of data-lake layers and implementation of reusable cloud-native engineering patterns.
Deep expertise in Databricks, Delta Lake, Unity Catalog, Medallion Architecture and Lakehouse Federation for building scalable, governed and high-performance data platforms across complex enterprise environments.
Azure Data Engineering
Extensive hands-on experience with Azure Data Factory, Azure Functions, Event Hubs, Service Bus, Azure SQL, Cosmos DB, Durable Functions and Logic Apps for designing and delivering end-to-end cloud-native data solutions.
PySpark & Batch/Streaming Pipeline Development
Strong proficiency in PySpark, Python, SQL and C# for developing batch and streaming pipelines, event-driven architectures and data contracts at enterprise scale.
DevOps & Infrastructure as Code
Solid experience with Azure DevOps, Git, Terraform, ARM Templates, Bicep, YAML CI/CD, PowerShell, Azure CLI and Bash for automating deployments and managing infrastructure across multi-environment setups.
Quality Engineering & Automated Testing
Practical application of PyTest, SpecFlow, TSQLt and Moq for automated validation, acceptance testing and code review, ensuring high release confidence and engineering standards.
Database Technologies
Experience with SQL Server, Netezza, Cosmos DB and Azure Data Explorer (KQL) for managing and querying large-scale structured and semi-structured datasets.
GenAI & LLM-Assisted Engineering
Application of GenAI and LLM-assisted approaches to accelerate code analysis, test-case generation, documentation and engineering productivity within data platform modernisation projects.
Kafka & Event-Driven Architecture
Design and delivery of event-driven ingestion and publishing frameworks using Kafka, Azure Event Hubs and Service Bus, supporting millions of records and events at scale.
Legacy Platform Modernisation
Track record of modernising legacy SQL/Synapse-based solutions into Databricks and Delta Lake architectures, including migration of data-lake layers and implementation of reusable cloud-native engineering patterns.
Erfahrung
Berufserfahrungmehr als 10 Jahre
Freelancer seit2022
Sprachen
EnglishMuttersprache
Projekthistorie
Designed and delivered a Medallion Architecture data platform evolving a monolithic livestock-movement solution to a scalable multi-species model with GenAI/LLM-assisted engineering. Built event-driven ingestion and outbound publishing frameworks using Databricks, Azure Functions and Service Bus. Led migration to Databricks Unity Catalog, implemented Lakehouse Federation, optimised event-publishing runtime from 24+ hours to under 60 minutes, and developed Azure DevOps CI/CD frameworks using ARM Templates and YAML.
Led Azure data-engineering delivery and designed end-to-end solutions for large-scale Journeys, Demand and Wi-Fi data migrations. Migrated legacy Synapse data-lake layers to Delta Lake and built enterprise ADF, Event Hub and real-time ingestion patterns. Created reusable PySpark modules and automated CI/CD using Azure DevOps, ARM and Bicep. Led team delivery including peer review, technical guidance and feature demonstrations, and strengthened automated acceptance and unit testing using SpecFlow and PyTest.
Architected direct-to-consumer ingestion pipelines using lakehouse/medallion patterns and established data contracts with third-party providers. Integrated SAP data through ADF into Delta Lake and built orchestration supporting API pull and batch-file push delivery models. Delivered production-ready PySpark applications, Azure Functions, automated testing and CI/CD using Azure DevOps, ARM and Terraform. Mentored engineers on unit testing, code quality and release practices.
Zertifikate
Databricks Certified Data Engineer Professional
Databricks2020
Implementing Big Data Analytics Solutions(70-475)
Microsoft2018