Middle Databricks Data Engineer

About the role

AgileEngine is looking for a Mid‑level Data Engineer to modernize a 15‑year‑old data warehouse by building a governed Databricks Lakehouse. You will create both batch and streaming pipelines using PySpark and Delta Lake, following a medallion architecture, and leverage AI tools such as Claude and GitHub Copilot to accelerate development.

Key responsibilities

• Design, develop, and operate batch and streaming pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Implement a bronze‑silver‑gold medallion architecture to serve analytics, reporting, and ML consumers.
• Migrate legacy ETL workloads to the Lakehouse while ensuring data parity and minimal disruption.
• Use Claude or GitHub Copilot to generate code scaffolding, tests, documentation, and prototypes.
• Write clean, well‑tested Python and SQL code and enforce code‑review standards.
• Optimize Spark jobs and Delta tables for performance and cost (partitioning, clustering, caching, cluster sizing).
• Apply data quality, lineage, and governance controls with Unity Catalog and automated validation.
• Troubleshoot pipeline failures, data defects, and production incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance.

Required profile

• 3+ years of professional data‑engineering experience.
• Hands‑on expertise with Apache Spark, Databricks, and cloud‑based data architectures.
• Proven experience modernizing legacy ETL systems and ensuring data hygiene.
• Strong problem‑solving, collaboration, and communication skills.
• Upper‑intermediate English proficiency.

Required skills

• PySpark
• Delta Lake
• Databricks (Workflows, Unity Catalog)
• Python
• SQL
• Structured Streaming, Auto Loader, Kafka, Event Hubs
• Airflow or Azure Data Factory
• dbt (or equivalent transformation framework)
• GitHub Copilot, Claude (AI coding assistants)
• Spark performance tuning and cost optimization