Apache Airflow Engineer
Deutsche Bank AG
(Banken und Finanzdienstleistungen, 1000-5000 Mitarbeiter)
Schlagwörter
Skills
MapR, HPE Ezmeral Data Fabric, Hadoop, GCP, Terraform, MapR, Spark, Hive, Airflow, OPENSHIFT, Spark streaming, Flume, data lineage, SAS, BIGDATA, data access, Apache Drill, Dremio, Trino datacenters, Puppet, Cassandra, Kafka, Postgres, Redis, OEM, Icinga, IT audit, fortify, system administration, cloud, OpenStack, Linux, Linux Red Hat 6 and 7, SELINUX, backup, caching, Varnish, Nginx, Grafana, Juniper, Cisco, Vlan, User Management, LDAP, Virtualization, KVM, Sdn, Windows servers, Windows Server 2003, 2008, control panel, database, DNS, Nagios, MRTG, Windows, microcomputer, Computer Science, client-server, ASP
Projekthistorie
02/2023
-
bis jetzt
- Design and implementation of large scale, high available, fault tolerant Apache Airflow. The implementation consists of running Airflow with different executor type both on bare metal/virtual machines and OpenShift (Kubernetes). Containers are built from source using Buildah based on ubi9-micro image. Podman and systemd script were used to run containers and services on bare metal/virtual machine nodes. The setup also covers Disaster Recovery scenarios. Helm charts were used for OpenShift deployment.
- High available, disaster recoverable and load balanced Postgres setup using PgBouncer, HAProxy, Patroni with backup and recovery via BarMan on both bare metal and Kubernetes.
- High available Redis setup using Sentinel.
- S3 compatible storage for Airflow remote logging.
- Monitoring of complete stack with Prometheus.
- Support the migration of workflows from Control-M, Automic and Tivoli WS to Airflow.
01/2023
-
03/2023
Apache Cassandra Engineer
Deutsche Post AG
(1000-5000 Mitarbeiter)
- Design, and implementation of Cassandra clusters in three different environments. The production cluster consists of multi-datacenter (primary and secondary clusters) setup to safeguard datacenter disaster scenarios. Clusters were tuned based on workload.
- Reaper was setup to perform Cassandra repair jobs.
- Prometheus was installed, dashboards were configured to show metrics collected from Cassandra nodes and application.
- Complete setup was automated by Ansible.
10/2021
-
12/2022
Technical Team Lead / Data Architect
PAYBACK
(Marketing, PR und Design, 1000-5000 Mitarbeiter)
- § Leading cross-functional team including OPS and Dev members.
- § Part of core team for DWH (Oracle, Hadoop) migration to GCP.
- § Design and setup of GCP projects via Terraform.
- § Major upgrade and patching of MapR clusters with full automation via Ansible.
- § Provide trainings (Application tuning) for DATA organization.
- § Data restructuring to achieve better Application (Spark, Hive) performance.
- § Apache Airflow installation, configuration, update and maintenance on OPENSHIFT via
Ansible automation. - § Spark streaming application development to replace Apache Flume.
- § Installation and configuration of DATAHUB(DataCatalog) including in-housedevelopment of data lineage generator for different data sources (Oracle, Hive, SAS). § Delegate duties and tasks within the team.
01/2020
-
09/2021
BIGDATA Specialist
PAYBACK GmbH
* MapR cross-clusters replication setup, automated with Ansible.
* MapR cross clusters data access setup.
* Containerization of MapR client/edge node via Ansible automation.
* Monitoring application performance on MapR eco-system (Hadoop cluster).
* Tunning MapR cluster to maximize utilization.
* PoCs on different in-memory query engines (Apache Drill, Trino, Starburst).
* Setup and configure application orchestration with Apache Airflow.
* JupyterHub installation and configuration.
* MapR cross clusters data access setup.
* Containerization of MapR client/edge node via Ansible automation.
* Monitoring application performance on MapR eco-system (Hadoop cluster).
* Tunning MapR cluster to maximize utilization.
* PoCs on different in-memory query engines (Apache Drill, Trino, Starburst).
* Setup and configure application orchestration with Apache Airflow.
* JupyterHub installation and configuration.
10/2017
-
01/2020
BIGDATA system engineer
PAYBACK GmbH
* Migration of multiple MapR (Hadoop) clusters between datacenters.
* Migration of Data and Applications between clusters.
* Installation, configuration and maintenance of MapR eco-system, fully automated by Ansible
and Puppet.
* Installation, configuration and maintenance of Cassandra, Kafka, Postgres and Redis.
* Setup and configuration of monitoring tools (OEM and Icinga).
* Migration of Data and Applications between clusters.
* Installation, configuration and maintenance of MapR eco-system, fully automated by Ansible
and Puppet.
* Installation, configuration and maintenance of Cassandra, Kafka, Postgres and Redis.
* Setup and configuration of monitoring tools (OEM and Icinga).
07/2015
-
09/2017
Team lead IT Operations
Xyrality GmbH
* Delegate duties and tasks within the IT department.
* Perform regular IT audit to discover areas of weaknesses and fortify them.
* Daily system administration tasks and routines.
* Private cloud setup and administration (OpenStack).
* Perform regular IT audit to discover areas of weaknesses and fortify them.
* Daily system administration tasks and routines.
* Private cloud setup and administration (OpenStack).
11/2012
-
06/2015
Linux System Administrator
Xyrality GmbH
* OS installation, configuration, troubleshooting and tuning for Linux Red Hat 6 and 7 with
security enhancement through SELINUX.
* Configuration and administration of HA Postgres cluster with backup and recovery validation.
* Installation and configuration of load balancer and caching mechanism (Varnish) and web
servers (Apache, Nginx).
* Setup and configuration of monitoring tools (Check_mk, Graphite, Grafana). Log management
and analysis (ELK tools).
* Configuration and troubleshooting of various network components (Juniper and Cisco) and
implementation of Policy rules, DMZ and multi cross Vlan communications.
* User management and authentication via LDAP and RADIUS.
* Infrastructure automation with Ansible.
security enhancement through SELINUX.
* Configuration and administration of HA Postgres cluster with backup and recovery validation.
* Installation and configuration of load balancer and caching mechanism (Varnish) and web
servers (Apache, Nginx).
* Setup and configuration of monitoring tools (Check_mk, Graphite, Grafana). Log management
and analysis (ELK tools).
* Configuration and troubleshooting of various network components (Juniper and Cisco) and
implementation of Policy rules, DMZ and multi cross Vlan communications.
* User management and authentication via LDAP and RADIUS.
* Infrastructure automation with Ansible.
09/2011
-
09/2012
Research Assistant at Faculty of Dentistry (Linux System Administrator)
University of Malaya
* Setup and maintenance of High-Performance Computing cluster with Rocks.
* Integration of servers to Cluster machine to be used for Bio-Informatics research.
* Server Virtualization by KVM and RHEV.
* Install and manage various Bio-Informatics software.
* Integration of servers to Cluster machine to be used for Bio-Informatics research.
* Server Virtualization by KVM and RHEV.
* Install and manage various Bio-Informatics software.
06/2010
-
08/2011
Junior System Administrator (Linux/Windows)
Shinjiru Technology Sdn Bhd
* Troubleshooting servers, different kind of Operating Systems (Linux and Windows servers -
Windows Server 2003, 2008).
* Troubleshoot various problems ranging from control panel, OS, software, applications,
emails, database, DNS and many more.
* Configuring mail servers.
* Assemble and change of hardware for servers.
* Being shift coordinator and distributing tasks to team members.
* Monitoring servers and network via Monitoring tools such as Nagios and MRTG graphs.
Windows Server 2003, 2008).
* Troubleshoot various problems ranging from control panel, OS, software, applications,
emails, database, DNS and many more.
* Configuring mail servers.
* Assemble and change of hardware for servers.
* Being shift coordinator and distributing tasks to team members.
* Monitoring servers and network via Monitoring tools such as Nagios and MRTG graphs.
03/2010
-
05/2010
Service Delivery Team Member
Shinjiru Technology Sdn Bhd
* Installing and configuring server with different platform (Windows and Linux).
* Troubleshooting servers for any hardware failure.
* Service and maintain microcomputer system, including test, run Scandisk, defragment hard
drives, update antiviral programs to maintain performance and security.
* Troubleshooting servers for any hardware failure.
* Service and maintain microcomputer system, including test, run Scandisk, defragment hard
drives, update antiviral programs to maintain performance and security.
09/2008
-
01/2009
Research Assistant at Faculty of Computer Science
University of Malaya
* Developing password less authentication system to ease client-server communication on
Mobile device.
* Synchronization of self-login authentication among the paired devices.
Mobile device.
* Synchronization of self-login authentication among the paired devices.
01/2007
-
07/2007
Software Developer
TATA Consultancy Services Ltd
* Part of a team developing a self-support portal for the employee of Deutsch Bank, Australia.
* The portal was built using ASP and later was deployed in Deutsch Bank, Melbourne, Australia.
* The portal was built using ASP and later was deployed in Deutsch Bank, Melbourne, Australia.
Zertifikate
Google Cloud Platform Professional Data Engineer
Google Cloud
2023
Google Cloud Platform Certified Associate Engineer
Google Cloud
2022
Databricks Certified Data Engineer Associate
Databricks
2022
Reisebereitschaft
Verfügbar in den Ländern
Deutschland, Österreich, Schweiz, Großbritannien, Belgien, Dänemark, Finnland, Irland und Niederlande
