Senior Databricks Engineer – Lakehouse Architecture

About the role

We are looking for a Senior Databricks Engineer to lead the design and implementation of a modern Lakehouse architecture. You will migrate legacy data solutions into scalable, governed pipelines using Databricks, PySpark, and Delta Lake, while collaborating with data engineers, architects, and business stakeholders.

Key responsibilities

• Architect and build end‑to‑end data solutions within the Databricks Lakehouse environment.
• Design Bronze, Silver, and Gold data layers and implement batch and near‑real‑time pipelines.
• Migrate legacy Azure SQL, SSIS, and ADF workflows to Spark‑based processes.
• Implement Auto Loader, Delta Lake Change Data Feed, and streaming ingestion patterns.
• Define governance using Unity Catalog, RBAC, and data lineage.
• Optimize performance with Photon, partitioning, Z‑Order indexing, and cluster tuning.
• Guide validation between legacy SQL outputs and Delta pipelines, conduct code reviews, and mentor junior engineers.

Required profile

• Bachelor’s degree in Computer Science, Engineering, Information Systems or equivalent.
• 6+ years of data engineering experience, including 3+ years with Databricks.
• Proven track record migrating legacy data platforms (SQL Server, SSIS, ADF).
• Experience leading technical implementations and mentoring teams.

Required skills

• Databricks, PySpark, Spark SQL, Delta Lake
• Azure ecosystem – ADLS, Azure Data Factory, Azure SQL, SSIS
• Unity Catalog, RBAC, data lineage
• Photon, Z‑Order indexing, cluster tuning, performance optimization
• Bronze/Silver/Gold medallion architecture, Auto Loader, Change Data Feed
• Batch and streaming pipeline development
• ETL/ELT design, data modeling