Data Engineer

Description:

We are looking for a skilled and hands-on Data Engineer to join the Data & Analytics team and contribute to building, enhancing, and supporting scalable enterprise data solutions on Microsoft Azure.

In this role, you will be responsible for developing and maintaining data pipelines, integrating data from SAP and non-SAP source systems, and implementing data transformations using Azure Databricks.

You will work extensively with Azure Data Factory, Databricks, PySpark, Python, and SQL to deliver reliable, high-quality, and well-structured datasets that support enterprise reporting, business intelligence, and advanced analytics.

The ideal candidate combines strong technical expertise with practical experience in enterprise-scale data engineering, data integration, performance optimization, and production support.

Key Responsibilities

1. Data Ingestion & Integration

• Design, develop, and maintain scalable data ingestion pipelines using Azure Data Factory (ADF).
• Integrate data from multiple sources, including SAP systems, relational databases, REST APIs, flat files, and cloud applications.
• Implement incremental data loading and Change Data Capture (CDC) mechanisms.
• Monitor, troubleshoot, and resolve data integration and pipeline-related issues.

2. Data Transformation & Processing

• Develop and maintain data transformation workflows using PySpark and SQL within Azure Databricks.
• Implement and manage the Medallion Architecture (Bronze, Silver, and Gold layers).
• Perform data cleansing, validation, enrichment, and transformation to meet business requirements.
• Optimize data processing jobs for performance, scalability, and cost efficiency.
• Work with Delta Lake to manage reliable and efficient data processing.

3. Data Warehousing & Modeling

• Support the implementation of enterprise data models and Common Data Models (CDM).
• Develop curated and structured datasets for reporting and analytics.
• Apply data warehousing and dimensional modeling principles.
• Ensure data consistency, accuracy, and reliability across multiple business domains.

4. Analytics Enablement

• Deliver trusted and optimized datasets for Power BI reporting and self-service analytics.
• Collaborate with Business Analysts, BI Developers, and Data Architects to understand business and technical requirements.
• Support improvements in reporting and dashboard performance.

5. Data Governance & Quality

• Implement data quality checks, validation rules, and monitoring frameworks.
• Ensure compliance with data governance standards and security policies.
• Document data flows, business rules, and technical processes.

6. Production Support & Operations

• Monitor scheduled jobs, workflows, and data pipelines.
• Investigate and resolve production incidents and data processing failures.
• Perform root cause analysis and implement corrective actions.
• Participate in deployment, release management, and CI/CD activities.

Required Technical Skills & Experience

• Strong hands-on experience with Azure Databricks (4+ years).
• Practical experience with Azure Data Factory (2+ years).
• Strong programming experience in PySpark (2+ years) and Python (2+ years).
• Strong SQL development skills, including complex queries and data transformations (3+ years).
• Good understanding of Azure Data Lake Storage (ADLS) and Delta Lake.
• Solid understanding of ETL/ELT processes and enterprise data warehousing.
• Experience implementing Medallion Architecture (Bronze, Silver, Gold).
• Knowledge of incremental data loading, data modeling, and data quality management.
• Experience building and maintaining enterprise-scale cloud data pipelines.
• Familiarity with Git, Azure DevOps, and CI/CD concepts.
• Understanding of Power BI datasets and reporting requirements.

Preferred Qualifications

• Experience integrating data from SAP ECC or SAP S/4HANA.
• Experience with REST API integrations.
• Familiarity with event-driven architectures.
• Knowledge of Unity Catalog and Microsoft Fabric.
• Experience in the Retail or Automotive industry.
• Experience supporting enterprise analytics and BI environments.

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related technical field.