Senior Data & BI Engineer

Role Title:

Senior Data Engineer

(Azure | Databricks | End-to-End Delivery)

Experience:

8–12 years (with strong Databricks depth)

Location:

Client location - Dubai

Employment Type:

Contract - Full time

We are looking for a lead, hands-on

Databricks Lead / Lakehouse Architect

who can

drive project kickoff

, define the

implementation strategy

, and lead

end-to-end delivery

of an

Azure Databricks Lakehouse

along with

Microsoft Fabric Power BI semantic modeling and reporting

for hospitality analytics. This role will own architecture, technical decisions, delivery planning, stakeholder alignment, and initial build-out, and then help ramp the broader team for implementation and rollout.

Key Responsibilities

Project Kickoff & Strategy

• Lead discovery workshops, clarify business objectives, define scope, milestones, and delivery approach.

• Lead discovery workshops with

  hospitality business/tech stakeholders (Revenue Management, Operations, Finance, Loyalty, Digital).

• Define target-state lakehouse architecture and delivery roadmap (phased onboarding of domains: reservations, stays, POS, loyalty, digital).

• Create end-to-end

  implementation strategy

  : architecture, design standards, data onboarding plan, governance, and operating model.

• Define best practices, coding standards, branching strategy, CI/CD approach, and environments (dev/test/prod).

Architecture & Platform Build

• Design and implement

  Azure Databricks Lakehouse

  architecture using

  Delta Lake

  and

  Medallion (Bronze/Silver/Gold)

  .

• Design data ingestion patterns for batch (and streaming if needed) from varied sources (APIs, DBs, files, event streams).

• Build scalable transformation frameworks using

  PySpark / Spark SQL

  with performance and cost optimization.

Data Engineering Delivery

• Implement and review complex pipelines: incremental loads, CDC patterns, SCD Type 1/2, data quality rules, and reconciliation.

• Own performance tuning: partitioning, file sizing, Z-ORDER, caching, adaptive query execution, cluster sizing, autoscaling.

• Implement monitoring/alerting, job orchestration, dependency management, and restart/recovery strategies.

Security, Governance & Operations

• Define security model: RBAC/ACLs, service principals, Key Vault integration, secrets management.

• Implement Unity Catalog strategy (if applicable): catalogs, schemas, lineage, permissions, data sharing.

• Establish operational readiness: runbooks, SLAs, logging, audit, cost governance.

Collaboration & Leadership

• Act as primary technical point of contact for client stakeholders.

• Produce design documentation (HLD/LLD), implementation roadmap, and knowledge transfer plan.

• Mentor engineers, conduct code reviews, and ensure delivery quality.

Must-Have Skills

• Deep hands-on

  Databricks

  experience in

  end-to-end implementations

  (not only development).

• Strong

  Spark

  expertise:

  PySpark + Spark SQL

  , distributed processing concepts, optimization.

• Strong experience with

  Delta Lake

  (MERGE, OPTIMIZE, VACUUM, schema evolution, time travel).

• Experience designing

  lakehouse

  architecture (medallion layers, curated zones, semantic datasets).

• Strong

  Azure

  data platform exposure:

  ADLS Gen2

  ,

  Azure Data Factory

  (or alternative orchestration),

  Key Vault

  ,

  Azure SQL/Synapse

  (any).

• Power BI development experience is mandatory.

• Strong understanding of:

• Incremental ingestion, CDC, SCD, data modeling for analytics

• Data quality frameworks, reconciliation, and observability

Nice-to-Have (Strong Plus)

• Unity Catalog

  implementation experience (access control, lineage, governance).

• CI/CD for Databricks: Azure DevOps/GitHub, Databricks Repos, asset bundles, Terraform.

• Streaming patterns (Kafka/Event Hubs), Auto Loader, DLT.

• Exposure to BI semantic layers

  (Microsoft Fabric Power BI)

• Databricks certifications (Data Engineer Associate/Professional).

• AI/Copilot Integration experience is plus