Senior Data Engineer (m/w/d)

Gutenbergstraße 20, 30823 Garbsen, Deutschland

Senior Data Engineer (m/w/d)

Up for a challenge?

We look for someone to support us in our data engineering and business intelligence efforts on a major replatforming project. A growing number of business units, each with rapidly growing data sets require us to rethink our data platform and gradually pursue new ways of collecting, cleaning and analyzing endless streams of data. Most of our analytical models currently run on top of Clicdata, which again utilizes Azure SQL Server for all stages of data processing.

For replatforming efforts we are using

  • Big Query, newly started, for workloads ClicData can't handle

  • Airflow, for batch-based ingestion and transformation as well as for quality monitoring

  • Pyhton/Flask-based APIs to offer our data products to other parts of the organization

  • Kafka-based streaming services for connecting downstream applications, immediate data ingestions and real-time analytics

  • Superset for data exploration

Perks & Benefits

  • A new, cloud-based system landscape

  • flat hierarchies

  • Cross-functional team with other data engineers and controllers

  • Exciting technologies, including BigQuery, Python, SQL, and Airflow

  • New projects for real-time analysis with technologies such as Kafka and stream-based analysis

  • Top employer - certified by the German Institute for Quality Standards e.V.

  • Internal trainings and offers for personal development

  • Corporate benefits

  • Development perspective to team leader

  • Fresh fruit from the region and good coffee every day

Tasks and Deliverables

  • Work on exisiting data pipelines and create new data pipelines in the existing toolset (ClicData, Azure SQL Server, Airflow, BigQuery)

  • Shape new ways of working for continuous collection, cleanup, harmonization and analysis of never-ending streams of data from multiple sources

  • Identify bottlenecks in the current ClicData-based implementations and rework the SQL-based pipelines to improve performance and reduce processing delays (currently a full pipeline run takes 3-4 hours)

  • Recreating, implementing, and documenting new (conceptual) models for new use cases

  • Automate data quality testing and processes around making sure we act on problematic data

  • Prepare, monitor and continuously refine data products for consumption by Data Analysts or Data Scientists

  • Maintaining continuous delivery through version control and automatic testing and deployment

Required Experience

  • At least 3 years in Advanced SQL in any major Database

  • At least 3 years in Python around data handling and APIs

  • Experience with code-based ETL pipelines (Airflow, Dagster, Prefect or alike)

  • Basic experience with cloud-based hosting using Docker and Kubernetes

Recommended Experience 

  • Experience with Streaming-based ingestions and data handling with tools like Kafka, Google PubSub, Amazon Kinesis or Azure EventGrid

  • Experience with Big Data analysis, preferably Stream analytics, with tools like Flink, Spark Structured Streaming or Apache Beam

  • Experience with Data Warehouse technology aka Big Query, Snowflake or Redshif

Deine persönliche Ansprechperson

Helen Roth freut sich auf Deine Onlinebewerbung. Das Recruiting wird von ALPHATOP HR GmbH übernommen. Hier erfährst Du mehr über die Verarbeitung Deiner Daten.

Senior Data Engineer (m/w/d)

Helen Roth

Tel: +49 7191 22081-13

Gutenbergstraße 20, 30823 Garbsen, Deutschland

Worauf wartest Du?

Du hast Interesse an dieser Stelle? Bewerbe Dich jetzt online.
Gemeinsam finden wir heraus, wie gut die Stelle zu Dir passt.

// OMS Prüfservice als Arbeitgeber

Unsere motivierte Mitarbeiter bestimmen unseren Erfolg

OMS Prüfservice im Profil

OMS Prüfservice ist eines der führenden Unternehmen für elektrische Sicherheit. Unsere Mission ist, Deutschlands Arbeitsplätze sicherer zu machen.

Unsere Werte bilden das Fundament für unser Denken und Handeln.

Werte und Kultur

Unsere Werte bilden das Fundament für unser Denken und Handeln. Sie begleiten uns in unserem Alltag, prägen unser Tun und sind die Grundlage für die hohe Qualität unserer Leistungen.

OMS wurde mit "Great Place to Work" als einer der besten Arbeitgeber in Deutschland ausgezeichnet

Ausgezeichneter Arbeitgeber

Wesentlicher Bestandteil der Erfolgsgeschichte von OMS sind die Mitarbeiter. Das bestätigen Auszeichnungen wie "Great Place to Work" und Kununu. Was sagen Andere über uns?