13.10.2026 aktualisiert
YA
100 % verfügbar
Senior Data Engineer | Python, PySpark, Databricks, AWS | 5+ yrs exp
Lengerich, Deutschland
Nur Remote
Master's degree in Economics and ManagementÜber mich
Senior Data Engineer with 5+ years of experience in Python, PySpark, Databricks, and AWS/Azure. Expert in building scalable ETL/ELT pipelines, Lakehouse architecture, and data optimization. Skilled in Airflow, SQL, Snowflake, and Ray, with a strong focus on data quality and performance.
Skills
ETL/ELT Pipeline Development
Design and optimization of ETL/ELT pipelines across Ingestion, Raw, Derived, and Datamart layers, including data quality and deduplication checks for reliable data processing.
Python & PySpark
Development of Python and PySpark transformations with AWS Glue and Databricks, using chunk-based processing and data partitioning for improved performance and scalability.
Cloud & Big Data Platforms
Extensive experience with Azure Databricks, Azure Data Lake (ADLS), AWS Glue, Step Functions, Athena, Lambda, Snowflake, Aurora, and Delta Table for scalable cloud-based data engineering.
Apache Airflow
Automation and monitoring of data workflows with Apache Airflow, reducing manual processing and ensuring reliable pipeline orchestration.
SQL & Databases
Proficiency in SQL with hands-on experience in Oracle, Postgres, Redis, Snowflake, Aurora, and OpenSearch for data storage and retrieval.
Data Lakehouse Architecture
Implementation of Lakehouse architecture using Delta Lake and Delta Table, enabling unified batch and streaming data processing.
Backend Development with Flask
Development of backend services with Python, Flask, and Oracle for internal logistics platforms, including API and database integrations.
Testing & Quality Assurance
Implementation of pytest data quality checks and automated pipeline execution to ensure data reliability and pipeline stability.
Tools & DevOps
Experience with GitHub, GitLab, JIRA, Asana, and Power BI for version control, project management, and data visualization.
Agentic AI
Knowledge of Agentic AI concepts, supported by a certification from DeepLearning.ai.
Design and optimization of ETL/ELT pipelines across Ingestion, Raw, Derived, and Datamart layers, including data quality and deduplication checks for reliable data processing.
Python & PySpark
Development of Python and PySpark transformations with AWS Glue and Databricks, using chunk-based processing and data partitioning for improved performance and scalability.
Cloud & Big Data Platforms
Extensive experience with Azure Databricks, Azure Data Lake (ADLS), AWS Glue, Step Functions, Athena, Lambda, Snowflake, Aurora, and Delta Table for scalable cloud-based data engineering.
Apache Airflow
Automation and monitoring of data workflows with Apache Airflow, reducing manual processing and ensuring reliable pipeline orchestration.
SQL & Databases
Proficiency in SQL with hands-on experience in Oracle, Postgres, Redis, Snowflake, Aurora, and OpenSearch for data storage and retrieval.
Data Lakehouse Architecture
Implementation of Lakehouse architecture using Delta Lake and Delta Table, enabling unified batch and streaming data processing.
Backend Development with Flask
Development of backend services with Python, Flask, and Oracle for internal logistics platforms, including API and database integrations.
Testing & Quality Assurance
Implementation of pytest data quality checks and automated pipeline execution to ensure data reliability and pipeline stability.
Tools & DevOps
Experience with GitHub, GitLab, JIRA, Asana, and Power BI for version control, project management, and data visualization.
Agentic AI
Knowledge of Agentic AI concepts, supported by a certification from DeepLearning.ai.
Erfahrung
Berufserfahrung5 - 10 Jahre
Freelancer seit2021
Sprachen
DeutschgutEnglischverhandlungssicherRussischMuttersprache
Projekthistorie
Built and optimized ETL/ELT pipelines across Ingestion, Raw, Derived, and Datamart layers. Developed Python and PySpark transformations with AWS Glue using chunk-based processing. Automated pipeline orchestration with AWS Step Functions and implemented data quality and deduplication checks.
Built and optimized ETL/ELT pipelines using Python, PySpark, Databricks, and Delta Lake. Automated and monitored data workflows with Apache Airflow, reducing manual processing. Optimized Spark transformations using data partitioning and chunking for faster processing and analytics.
Built distributed data pipelines with Ray for scalable data processing and analytics. Optimized parallel data processing to improve pipeline performance and scalability. Implemented pytest data quality checks and automated pipeline execution.