Job Description

Lead Data Engineer Databricks & MLOps Focus

We are seeking a highly experienced and strategic Lead Data Engineer with deep expertise in Databricks, MLOps, and cloud-native data engineering. This role is ideal for a technically
strong leader who thrives in consulting environments, excels at client engagement, and can drive complex data initiatives from architecture to execution.

Key Responsibilities:

Lead Architecture & Development: Design and implement scalable, high-performance data pipelines and MLOps workflows using Databricks, Delta Lake, and MLflow.
Migration Leadership: Oversee the migration of AWS EMR Spark jobs to Databricks, optimizing for performance, cost, and maintainability.
Client Consulting: Act as a trusted advisor to clients, providing strategic guidance on Databricks architecture, MLOps best practices, and data platform modernization.
Model Operationalization: Collaborate with data scientists to productionize machine learning models and manage the full ML lifecycle.
Team Leadership: Mentor and lead data engineering teams, fostering a culture of innovation, quality, and continuous improvement.
Cross-functional Collaboration: Work closely with cloud architects, DevOps, and business stakeholders to align technical solutions with business goals.
Innovation & Optimization: Identify opportunities to improve data workflows, automation, and platform capabilities using modern tools and frameworks.
Governance & Compliance: Implement data governance practices using tools like Unity Catalog, ensuring data quality, lineage, and access control.

Required Skills & Experience:

7+ years of experience in data engineering, with at least 2 years in a technical leadership or consulting role.
Expert-level proficiency in Databricks, including Delta Lake, MLflow, Job Workflows, and Unity Catalog.
Proven experience migrating and optimizing Apache Spark workloads from EMR or other platforms to Databricks.
Strong understanding of MLOps principles, CI/CD for data pipelines, and model deployment strategies.
Hands-on experience with cloud platforms such as AWS, Azure, or GCP.
Proficiency in Python, SQL, and Spark.
Experience with streaming data technologies like Kafka, Structured Streaming, or Delta Live Tables.
Excellent communication, leadership, and consulting skills, with a proven ability to engage clients and stakeholders.
Strong problem-solving skills and a passion for innovation and continuous learning.

Preferred Qualifications:

Certifications such as:

- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate
- AWS Certified Data Analytics Specialty
Experience implementing medallion architecture (bronze-silver-gold layers).
Familiarity with data governance and observability tools.
Background in consulting firms or professional services with a track record of delivering high-impact solutions.

Note: Job Description and Background Check

Candidates may be subjected to a Background Check /Drug Test as required by the end client before the assignment starts.

Job Tags

Similar Jobs