Lead Data Engineer - Databricks
Coimbatore · Staff/Principal
Applicants who checked fit first are 3.1× more likely to hear back
Your score for this role already exists
ASAI compared this JD against 41 signals - skills, seniority, domain, stack overlap etc. Add a resume and it unlocks in about 30 seconds.
No credit card · 1 tap with Google
What we know about this role
MEDIUM
ShyftLabs is reviewing applications at a steady pace. Expect a standard response time as they evaluate the current pool.
First 72 hours
Window passed - posted 12d agoEarly applicants get seen before the pile builds.
Not a repost
The first time we've seen this listing - it hasn't been closed and reopened.
Skills required
18 listedYou almost certainly match several of these already. Unlock your skill map to see the matches, the gaps, and what to fix first.
Job description
Position Overview
We are looking for a Data Engineer with hands-on experience in building scalable data pipelines and data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate should have strong expertise in Python, PySpark, SQL, Databricks, AWS, and REST API integrations for data ingestion, managing large volumes of data, and data export
Job Responsibilities:
PySpark, and SQL.
● Integrate data from multiple sources, including databases, Amazon S3, files, and REST APIs.
● Build data pipelines with Databricks Unity Catalog.
● Implement business logic, data transformations, and dimensional data models.
● Create, schedule, monitor, and optimize Databricks Jobs and Workflows.
● Design and manage Delta Lake tables using Medallion Architecture (Bronze, Silver,Gold).
● Ensure data quality through validations, error handling, logging, and monitoring.
● Optimize Spark workloads for performance, scalability, and reliability.
● Collaborate with cross-functional teams to deliver production-ready data solutions.
Basic Qualification:
● Hands-on experience with the Databricks Lakehouse Platform.
● Good understanding of Unity Catalog, Delta Lake, Databricks Workflows/Jobs,
Clusters, Notebooks, Repos, and Medallion Architecture.
● Experience integrating with REST APIs for data ingestion and data export.
● Strong knowledge of ETL/ELT development, batch processing, incremental loading,
and data transformation.
● Experience with data modeling (Star Schema, Snowflake Schema, Fact & Dimension
tables, SCD concepts).
● Understanding of data warehousing concepts and best practices.
● Experience working with structured and semi-structured data (CSV, JSON, Parquet,
Delta).
● Knowledge of partitioning, file optimization, Spark performance tuning, and query
optimization.
● Experience with Git and CI/CD best practices
Preferred Qualifications:
Free · no signup
Get tomorrow's jobs before you have to search
Daily job drops, skill trends and free resources - posted straight to the group. Leave any time.
No spam. Just jobs and resources.
Why people use ASAI
Someone shared one job with you. ASAI keeps finding the rest.
Scored, not searched. Every role ranked against your actual profile.
Alerts as often as hourly. Reach new roles while the pile is still small.
Skill gaps, spelled out. See exactly which requirements you don't meet yet.
Verified jobs, only. Say no to ghost jobs. Your time deserves respect.
Keep browsing