Description:
The Azure Engineer will build and maintain data pipelines, storage layers, and data products based on the architecture design while meeting organizational requirements. This role involves working closely with the end client and a team of backend engineers, cloud engineers, and data analysts to enable seamless data integration and build robust data platforms on Azure Cloud.
Responsibilities:
- Design, develop, and manage data pipelines on Microsoft Azure for internal and external data sources.
- Develop ingestion frameworks for both batch and real-time data.
- Ensure data pipelines are optimized for reliability and scalability.
- Ensure the scalability, security, and performance of cloud architecture while aligning with business goals.
- Collaborate closely with solution architects, backend developers, and data governance specialists.
- Collaborate with business stakeholders to translate functional requirements into scalable technical solutions.
- Hands-on experience with Azure Databricks platform for building end-to-end data solutions.
- Proficient in Azure Data Services, including Azure Data Factory and Azure SQL Database.
- Commercial experience with Microsoft Fabric is a must.
- Build and optimize data lake zones (raw, curated, consumption).
- Develop business and dataflow logic for data processing using tools like PySpark, T-SQL, and Stored Procedures.
- Implement message queues using Azure Service Bus.
- Integrate cloud-based solutions with on-premises systems.
- Implement Infrastructure as Code (IaC) using Terraform for automated and consistent provisioning.
- Strong understanding of Azure architecture components, including compute, storage, networking, and security.
- Optimize cloud storage solutions for both performance and cost.
- Establish monitoring and logging solutions using Azure Monitor, Log Analytics, Application Insights, and third-party tools.
- Configure automated scaling solutions to handle workload fluctuations.
- Create and maintain comprehensive architecture documentation, including diagrams, processes, and technical specifications