Description:
Exciting Opportunity Alert! 🌟 HTC Global Services is hiring Azure Data bricks - Lead (Airline Project) for an 1-year extendable contract in Abu Dhabi, UAE (Onsite).
HTC Global Services - a leading CMM level 5 global provider of innovative IT and Business Process Services and Solutions since 1990 with headquarters in Troy, Michigan, USA.
- Lead and manage a team of Data Engineers, providing technical guidance, mentorship, and performance coaching.
- Design, develop, and optimize large-scale data pipelines using Azure Databricks and PySpark.
- Architect and implement modern data lake and lakehouse solutions using Azure Data Lake Storage (ADLS).
- Build and maintain batch and real-time streaming data pipelines using technologies such as Azure Event Hubs, Kafka, Spark Streaming, or Structured Streaming.
- Collaborate with Solution Architects, Data Scientists, Product Owners, and business stakeholders to define data requirements and technical solutions.
- Establish data engineering best practices, coding standards, CI/CD processes, and governance frameworks.
- Ensure high availability, scalability, security, and performance of data platforms.
- Drive Agile delivery, sprint planning, effort estimation, and technical reviews.
- Troubleshoot complex data engineering challenges and optimize platform performance.
- Evaluate emerging technologies and recommend improvements to the organization's data architecture.
Required Skills & Experience
Technical Skills
- Strong experience with Azure Data Engineering ecosystem.
- Expert-level proficiency in Azure Databricks.
- Strong hands-on experience with PySpark and Apache Spark.
- Experience with Azure Data Lake Storage (ADLS Gen2).
- Expertise in designing and implementing streaming data solutions using:
- Spark Structured Streaming
- Azure Event Hubs
- Apache Kafka
- Real-time data processing frameworks
- Experience with Azure Data Factory (ADF) for orchestration and ETL workflows.
- Strong SQL and data modeling skills.
- Experience with Delta Lake and Lakehouse architecture.
- Knowledge of CI/CD pipelines, Azure DevOps, Git, and Infrastructure as Code.
- Experience working with large-scale enterprise data platforms.
- Strong experience with Databricks Unity Catalog for enterprise data governance, access control, metadata management, and secure data sharing across multiple workspaces.
- Hands-on experience with Lakeflow Declarative Pipelines (formerly Delta Live Tables) for building scalable, reliable, and maintainable data ingestion and transformation pipelines.
- Expertise in implementing data quality checks, monitoring, lineage, and orchestration using Databricks-native pipeline capabilities.
- Experience working with Databricks Genie and AI-powered conversational analytics to enable business users to interact with enterprise data through natural language interfaces.
- Knowledge of integrating Generative AI capabilities within the Databricks ecosystem, including leveraging Databricks AI/BI, Genie Spaces, and Lakehouse AI features.
- Strong understanding of Delta Lake optimization techniques, including partitioning, clustering, performance tuning, and data lifecycle management.
- Experience in designing and implementing modern Lakehouse architectures using Databricks, Unity Catalog, Delta Lake, and Azure Data Lake Storage (ADLS).