08.04.2026 aktualisiert

**** ******** ****
Premiumkunde
40 % teilweise verfügbar

Data Engineer | Data Vault 2.0 Certified | AWS/Azure Databricks | Terraform IaC | Data Architecture

Berlin, Deutschland
Nur Remote
Bachelor of Science in Computer Science, University of Central Punjab, Lahore Pakistan 2001 - 2005.
Berlin, Deutschland
Nur Remote
Bachelor of Science in Computer Science, University of Central Punjab, Lahore Pakistan 2001 - 2005.

Profilanlagen

CV_MuhammadMoizAhmed_042026.pdf

Über mich

Lead Data Engineer & Architect | 15+ yrs | Kafka streaming (100K+ msg/hr) | PySpark/Python | AWS/Azure | Terraform | Databricks | Airflow | OpenSearch | Data Vault 2.0 Certified | Recent: TotalEnergies, Deutsche Bahn, Circle K | Berlin | Freelance

Skills

Apache AirflowData ArchitectureData GovernanceData VaultSqlalchemyAWS LambdaInfrastructure as Code (IaC)Data LakeData LineageAWS GlueApache KafkaTerraformDatabricks
Senior Data Professional mit 15+ Jahren Expertise in Enterprise Data Modeling, Data Governance und Cloud Platforms. Spezialisiert auf Common Data Models nach Data Vault 2.0 für regulierte Branchen. Projekterfolge: Konsolidierung 20+ Entities, 40% Kostensenkung, 60% Runtime-Verbesserung, 10K+ Messages/Hour Kafka Streaming.

Data Governance & Enterprise Architecture Certified Data Vault 2.0 Practitioner. 2,5 Jahre Group Data Office (globale Versicherung): Hub-Link-Satellite-Strukturen, Business Vault, Point-in-Time Tables, Dimensional Models für Insurance (Policy, Claims, Reinsurance, CAT Risk, Cyber). Global Business Glossary (500+ Terms), Metadata Management, Data Lineage, Compliance (GDPR, IFRS17, Solvency II). 30+ Workshops (DE, CH, UK, US, Asien). Tech: SQLDBM, Informatica Axon, Data Vault 2.0

Apache Kafka & Real-Time Streaming Enterprise-scale Kafka-to-PostgreSQL Streaming: 10,000+ Messages/Hour, zero data loss, sub-second latency. Consumer Group Management, Offset Tracking, Circuit Breaker Patterns. CDC-Pipelines: 100,000+ Records/Minute mit AWS Lambda, OpenSearch Ingestion v2. Tech: Apache Kafka, AWS Glue Streaming, Lambda, OpenSearch, Python

Apache Airflow Orchestration Production Monitoring DAGs mit Database Sensors, XCom, Branching Logic. 18h-Timeout-Resilienz, automated alerting. Complex scheduling (daily/monthly segregation). Tech: Apache Airflow, PostgreSQL, SQLAlchemy

Multi-Cloud Architecture & Migration AWS-to-Azure Migrationen: Circle K (600+ locations, 6M+ daily transactions), Couche-Tard, TotalEnergies. Ergebnis: 40% Kostensenkung, 60% Runtime-Verbesserung. Medallion Architecture, Unity Catalog, Delta Lake/Delta Tables. Tech: AWS (Lambda, Glue, S3, Redshift, Aurora), Azure (Databricks, Synapse, Data Lake Storage Gen2)

PySpark & Python Engineering Abstract Base Class ETL patterns, Schema-as-Contract design, pure transformation functions. Pandas/PySpark DataFrames, pytest TDD. Docker-based local development (AWS Glue containers). API Integrations (Microsoft Graph, RESTful), Liquibase migrations. Tech: Python, PySpark, Pandas, pytest, Docker, Liquibase

Infrastructure as Code Terraform: 15+ Module, modular architecture, security-first IAM. 100% Compliance (tfsec, tflint, Bandit, Semgrep). GitOps mit automated scanning, drift detection. Tech: Terraform, GitHub Actions, GitLab CI/CD

Middleware & Integration IBM App Connect Studio: SAP↔Salesforce, DocuSign integrations. iDoc, JSON/XML transformations, HMAC security. Swagger API specifications. Tech: IBM App Connect, SAP, Salesforce, GCP

BI & Performance Power BI: Semantic Models, Row-Level Security, DAX. Redshift tuning (distribution keys, WLM), Aurora PostgreSQL. Tech: Power BI, Tableau, Redshift, PostgreSQL

Stakeholder Management Internationale Teams, 30+ Workshops mit Legal, Compliance, BI, IT. 100+ Seiten Guidelines, 50+ Presentations. Konsensbildung, kulturelle Transformation.

Erfahrung

Berufserfahrungmehr als 10 Jahre
Freelancer seit2017

Sprachen

DeutschverhandlungssicherEnglischverhandlungssicher

Projekthistorie

Lead Data Engineer -- Data Platform für Appian

Transport - Public Sektor

Transport und Logistik

500-1000 Mitarbeiter

Standort: Frankfurt (Remote from Berlin)

Leitung Design und Implementierung enterprise-weiter Datenplattform mit Fokus auf skalierbare Architektur-Patterns und Compliance-getriebene Entwicklung.

Kernverantwortlichkeiten:
  1. Designed Medallion Architecture (Bronze/Silver/Gold) mit serverless ETL pipelines (AWS Glue, Lambda, PySpark)
  2. Architected OpenSearch CDC ingestion pipeline (Öffentlicher Verkehrssektor) processing 100,000+ records/minute mit AWS Lambda (Python 3.13, 1,000 events/batch, 100 concurrent executions), OpenSearch Ingestion Pipeline v2, RDS Proxy
  3. Built nested document architecture mit fuzzy search analyzers, edge n-gram tokenization
  4. Implemented Terraform modules: 15+ reusable modules mit modular architecture, conditional resources, lifecycle management, security-first IAM, templatefile() for YAML configuration
  5. Resolved security compliance (tfsec, tflint, Bandit, Semgrep) achieving zero vulnerabilities
  6. Established GitOps workflow mit automated security scanning, drift detection, CI/CD
  7. Created reusable components for Lambda, OpenSearch, Aurora PostgreSQL, RDS Proxy
  8. Established CloudWatch monitoring for pipeline health and performance metrics
Projektergebnis: Enabled scalable data platform supporting multiple business domains with real-time search capabilities and standardized patterns ensuring compliance and operational excellence.

Technologien: AWS (Lambda, Glue, S3, API Gateway, Aurora RDS, OpenSearch, RDS Proxy, CloudWatch), Python, PySpark, Terraform, PostgreSQL, Apache TIKA

Data Engineer (Python/PySpark)

Simon-Kucher Engine Ltd

Internet und Informationstechnologie

500-1000 Mitarbeiter

Developed PySpark ETL pipelines for revenue management data platform, implementing Abstract Base Class pattern with schema-driven data migration to PostgreSQL.

PySpark ETL Development:
- Implemented ABC ETLJob pattern with enforced method signatures: extract(), transform(), load(), run()
- Built pure transformation functions: business rule filters, column renaming, date parsing, type casting
- Developed session factory supporting both local Spark and AWS Glue environments
- Engineered ordered pipeline: filter on source columns → rename to target schema → parse/cast → add metadata.

Schema-as-Contract Pattern:
- Created dual schema classes separating source (CSV) and target (DB) with StructType definitions ,
- Implemented column mapping dictionaries for traceable source-to-target transformations
- Built schema-driven CSV reader handling column selection, null values, type enforcement

Database Operations:
- Developed idempotent load using TRUNCATE + INSERT for full snapshot data
- Created Liquibase changesets for PostgreSQL migrations with generated columns and indexes

Local Development & Testing:
- Built Docker environment using AWS Glue container + PostgreSQL for rapid iteration
- Implemented local-first workflow: test with pytest, deploy to cloud unchanged
- Documented ETL patterns comparing ABC approach with DataFrame chaining alternatives

Technologies:
PySpark, Python (ABC, StructType), PostgreSQL, Liquibase, pytest, Docker, AWS Glue

Project Data Manager / Data Engineer

Circle K

Energie, Wasser und Umwelt

500-1000 Mitarbeiter

Architected and executed AWS-to-multicloud migration for Circle K's data platform (AWS/Azure Databricks), designing cross-cloud data transfer solutions that modernized infrastructure while maintaining zero-downtime operations for 600+ retail locations.
Cross-Cloud Migration Architecture:
  • Architected S3-to-Azure Blob pipeline via AWS Databricks for cross-cloud data transfer from legacy AWS to Circle K Azure
  • Phased migration strategy maintaining single legacy AWS account for synchronization while transitioning workloads to Circle K environments
  • Unity Catalog volumes collaboration with Circle K metastore admins, defining schema structures and access patterns
  • Migrated from low-level Spark RDDs to high-level DataFrame APIs, enabling Photon acceleration for 3-5x performance gains
Infrastructure as Code & Automation:
  • Implemented Terraform Redshift stack and modules, with RA3 node configuration
  • Implemented Redshift native schedules with associated IAM roles for automated ETL workflows
  • Managed database scripts for database and schema definitions / user access management
Data Pipeline Modernization:
  • Refactored multiple Python notebooks from AWS Databricks to Azure Databricks, optimizing for Photon accelerator
  • Implemented Unity Catalog Delta tables for daily POS data processing with automatic schema evolution
  • Created weekly KPI / reports using Databricks views analyzing product volumes and sales metrics
  • Migrated data pipelines from sequential processing to parallel Spark operations, reducing runtime by 60%
Outcome: Successfully orchestrated zero-downtime migration serving 6M+ daily transactions; scalable cross-cloud architecture supporting both AWS and Azure workloads, reduced operational costs by 40% through infrastructure consolidation.
Technologies: Azure Databricks, AWS Databricks, Terraform, Unity Catalog, Delta Lake, Redshift (RA3), S3 Cross-Account Access, IAM AssumeRole, Photon Engine, PySpark, Azure Blob Storage

Project Data Manager

Couche-Tard

Energie, Wasser und Umwelt

500-1000 Mitarbeiter

Managed cloud data infrastructure, focusing on Redshift optimization, Unity Catalog implementation, and AWS Databricks. Led performance tuning initiatives reducing query execution times across Power BI workloads.
Redshift Performance Engineering:
  • Implemented distribution keys, sort/interleaved sort keys based on query pattern analysis
  • Established Workload Management (WLM) queues with memory allocation optimization for concurrent Power BI refresh jobs and Pipeline workloads
  • Configured RA3 node clusters with managed storage enabling independent compute/storage scaling
  • Created admin monitoring views for table statistics, query performance, and disk usage tracking
  • Advised VACUUM strategies with scheduling during pipeline runs, optimizing table performance
  • Implemented automatic snapshot schedules for recovery scenarios
Databricks Platform Establishment:
  • Secured AWS Databricks access from parent organization through governance approval process
  • Implemented Terraform modules for Unity Catalog setup with three-level namespace (catalog.schema.table)
  • Developed POCs for AWS Databricks adoption over Redshift, for performance and cost improvements, using Delta table
  • Built data pipelines for public and internal data sources, using Spark RDD parallel processing
  • Built data quality monitoring completeness, uniqueness, and timeliness KPIs
AWS Infrastructure Management:
  • Managed subnet groups and CIDR block allocations for Redshift cluster isolation
  • Configured cluster resize operations from dc2.large to RA3.xlplus nodes with minimal downtime
  • Managed Power BI Gateway Data Sources, and Network Firewall Whitelisting b/w OnPrem and Cloud using FQDNs
BOXI to Cloud Data Integration:
  • Integrated SAP BOXI SFTP Exports with AWS Cloud, using AWS Transfer Family and Fingerprint calculation approach
Technologies: AWS Redshift (RA3), Databricks, Unity Catalog, Terraform, WLM, Distribution Keys, Interleaved Sort Keys, Power BI, S3, Spark RDD, Python, Delta Lake

Data Modeler, Group Data Office

Group Data Office – Insurance Branche

Versicherungen

1000-5000 Mitarbeiter

Location: Munich

Datenbankaufbau und -strukturen nach Data Vault Design, aber auch 3NF und Data Marts (Consumption Layer).

Engaged as a Data Vault 2.0 Modeler with a Group Data Office from Insurance and Reinsurance business, for Group implementation of Commercial Insurance (PLC), Reinsurance data collection, sourced from various Operating Entities across various Geographical locations.
 
  • Updating and enhancing Group Commercial Common/Unified Data Model (nach Data Vault 2.0).
  • Updating Data Standards for the Data Model.
  • Enhancing Global Definitions (Global Business Glossary)
  • Developing business Data Examples for Operating Entities.
  • Writing Mapping Guidelines.
  • Unifying CAT Risk 3NF Data Model into group Commercial DV 2.0 Data Model.
  • Unifying Health Data Model into group DV 2.0 Data Model.
  • Participation in Enterprise Ontology clarification Workshops
  • Business examples and presentation slide decks for various Scopes in the model, Reinsurance (FAC/Treaty/Quota Share), Incident/Claim classifications, Policy and Object Terms (L&Ds), IIP (International Insurance Programs) to name a few
  • Data Mapping and Validation Example preparations, show-casing bi-directional mapping between group Commercial DV 2.0 Data Model and CAT Risk and Cyber 3NF Data Models.
  • Writing detailed Mapping Guideline for Data Deliveries into Group Commercial Common Ingestion DV2.0 Data Model.
  • Led solutioning in Data Model Workshops.
  • Unifying Claims Data Model into group Commercial DV 2.0 Data Model.
  • Participation in Workshops with Data Architects from other Business units.
  • Member of Community of Practitioners on Business Intelligence, Architecture and Data Modeling.

Outcome: Enabled Group Data Reporting for Portfolio steering on Common Data Standards and Global Business Glossary

Tools: SQLDBM, PostgresSQL, Azure Synapse Analytics Pool DB SQL, GitHub, Confluence, Informatica Axon (GBG), Sharepoint, Excel, Powerpoint, Enterprise Ontology, Protegé

Project Data Manager

TotalEnergies

Energie, Wasser und Umwelt

500-1000 Mitarbeiter

Advised on AWS cloud adoption, data management, and infrastructure architecture. Proposed architecture solutions, evaluating AWS building blocks to align with client needs, project planning and team communications.
Project and Incident Management:
  • Coordinated with Cyber Security and Group Admins for secure implementations and incident management
  • Managed IT demands and change requests in ServiceNow
  • Handled GitHub team permissions
  • Facilitated Databricks GitHub app requests, managing repos for various environments
Data Lake Management and AWS Setup:
  • S3 Data Collection: Led sessions to define data categories for S3, per company classifications
  • AWS Transfer Family with Lambda: Implemented Transfer Family with Lambda for custom authentication
  • IAM Policies: Consulted on IAM policy setup for secure, cross-account S3 access
  • Redshift Setup: Managed Redshift cluster setup, consulting on keys and Copy Commands for optimization
  • Liquibase: Advised on Liquibase adoption for database versioning in CICD
Architecture and Security Management:
  • AWS Account Documentation: environment classification, IAM policies, S3 access, and VPC connections to on-prem via Transit Gateway
  • Subnet and CIDR Blocks: Designed and documented subnet layouts for network security
  • Firewall Management: Managed firewall openings/closings per compliance standards
  • Resource Monitoring: Monitored AWS resources, optimizing costs
  • Databricks Access: Configured SCIM passthrough and meta-IAM roles
  • GitHub OIDC Integration: IAM roles for GitHub-to-AWS access via OIDC
Databricks Data Engineering and Data Transfer:
  • Unity Catalog: Deployed with Terraform for data governance
  • Delta Tables and Workflows: Created Delta Tables and workflows for data processing
  • Python Notebooks: Built notebooks for analysis, including data cleanup and public datasets (ELWIS, PEGELONLINE WSV, DWD)
  • Data Transfers: Developed Shell/SFTP scripts enabling BOXI exports over SFTP for data collection
Technologies: AWS, Databricks, Unity Catalog, Terraform, S3, Redshift, IAM Policies, Lambda, AWS Transfer Family, Delta Lake, Python, ServiceNow, GitHub OIDC, Transit Gateway, BOXI, Shell/SFTP

Data Vault / Integeration Layer Developer

Insurance Branche

Versicherungen

1000-5000 Mitarbeiter

Location: Munich

Engaged as a Data Vault 2.0 developer for implementation of IFRS17 Account Standard requirements and adaptions, at an Insurance company in Munich. As an Integration Layer developer, I was mainly responsible for:
 
  • Overseeing operations in Integration Layer
  • Analysing and Developing new Raw Vault and Business Vault hubs, links and satellites
  • Ensuring relationships between entities loaded from different source systems.
  • Integrating new file based data source for Top Adjustments for month end closings.
  • Working with Effectivity Satellites
  • Met requirements for Ledge Specific postings (GAAP Codes) and Automated Reversals.
  • Provided Loading Templates for Premiums (Policy), Claims and Cash transactions.
  • Analysing Data Quality errors
  • Generating SQL packages and carrying out deployments.
  • Preparing Visual Data Vault Diagrams!
  • Documentation in Confluence
  • Exchanging with other IL Developers, Engineering Manager and Product Managers.
  • Raising Pull Requests in Github, and Merging them to Master (after a successful review).
  • Creating Test Cases in Tricentis TOSCA
  • Working in an agile 10 days Sprint basis, creating User Stories and allocating Story Points.
  • Participcating in Feature Grooming sessions.
  • Using Jenkins to manage and schedule Runtime jobs

Tools: SQL Developer, Oracle, GitHub, Tricentics TOSCA, Eclipse, Jenkins

Project Manager GDPR Data Lake

Real State Platform company

Sonstiges

1000-5000 Mitarbeiter

Location: Berlin.

Working as Project Manager for GDPR implementation in Data Lake storage at a Real State Platform company.
 
  • Responsible for coordination between tech team and legal.
  • Engagement with external DPO, to clarify GDPR requirements.
  • Stake holder management: one on one interviews with various stakholder, communicaiton of developing topics
  • Over-seeing team plannings: publishing Road Maps, weekly/bi-weekly team meetings.
  • Communication with data consumers and producers.

Tools: Miro, Confluence, JIRA , Project documentation

Middleware Data Integration Specialist

Telekomm Branche

Telekommunikation

1000-5000 Mitarbeiter

Location: Berlin.

Developing and Maintaining Middleware Restful APIs for Integration use-cases, such as:
 
  • between SAP and Salesforce
  • DocuSign and Salesforce,
  • Microservices, legacy ERPs and Internal Systems that maintain Parts information
  • GCP

Working with Json, XML and iDoc (SAP) Formats, to develop Integrtations and Gateways. Analysis of source SAP Data structures and Data models, Salesforce and real-time micro-services. Data Mapping between SAP, Salesforce and the Micro-service.
Implementation of HMAC.
 
  • SQL development
  • Stored procedure development
  • Use of Transaction Management
  • Exception Handling.

Regularly prepared Swagger specifications, Test Evidence and UAT documentation.

Technologies: MSSQL Server 2014 (SQL/T-SQL), Middleware (IBM App Connect Studio), JSON/XML, CSV, Swagger, GCP, Restful APIs

Python Data Migration Engineer

Fashion eCommerce Shop
Worked on a migration project to migrate fashion Products data from Product Information Management system to a new Enterprise version.

DWH/Data Vault Entwickler

Payment Provider
Integrated SAP (bookings / settled payments – book keepings) data into Data Vault for a Payment Provider.

Data Integration Engineer (Python/Nifi)

Consulting House
Developed a complete Data Engineering solution using Python and Nifi for a Consulting House client.

Data Vault 2.0 Consultant

Insurance Branche
Consulted on a Big Data project with Data Vault 2.0 Modelling, using AWS, Snowflake and Cloudera (CDH) Cloud solutions for an Insurance company.

MicroStrategy Freelance Consultant

International Retail Co.
Implemented MicroStrategy Use Cases for the retail business as part of the FE team.

Member of Management Team

Crosslend GmbH
Led BI and Analytics function, collaborating with executives and various departments to implement MicroStrategy Use-cases and ETLs in MySQL.

Data Warehouse Architect

Kreditech Holding SSL GmbH
Responsible for Data Warehouse Architecture and Data Engineering Team, managing DWH technology infrastructure and service providers.

Senior Manager BI

Zalando SE
Managed Customer pipeline within Zalando BI, focusing on cohort trend analysis, website click log analysis, and customer survey data design and development.

Senior System Analyst

Redmath Pvt Ltd
Worked as an Offsite Senior Analyst for Transaction banking & Transport logistics, responsible for Requirement Analysis, Solution Definition and Technical Design.

Assistant Manager

HBL – Habib Bank Ltd
Provided reporting for multiple divisions of the bank, including setting up Oracle database, streamlining ad-hoc query requirements, and managing heterogeneous source databases.

Software Engineer

Innovation Group plc (a UK company)
Responsible for MI analytics of Insurance Solutions, working closely with Innovation Group Data Architect and Business Analysts.

Zertifikate

Databricks Fundamentals

Databricks Academy

2024

SqlDBM Fundamentals

Academy of Data Modeling

2024

dbt Fundamentals

dbt Labs

2024

Certified Data Vault 2.0 Practitioner

Data Vault Alliance

2018

Kontaktanfrage

Einloggen & anfragen.

Das Kontaktformular ist nur für eingeloggte Nutzer verfügbar.

RegistrierenAnmelden