Designation : Assistant Manager - Data Engineering

Level : L3

Location : Chennai

Experience : 5 to 8 Years

Job Role :

Build and maintain scalable data pipelines and data warehouse solutions using Python, PySpark, SQL, Databricks, and Airflow.
Drive data quality, workflow orchestration, performance optimization, and Linux-based data engineering, while collaborating with cross-functional teams to deliver reliable analytics solutions.

Responsibilities :

Data Pipeline Development:

· Design, develop, and maintain scalable data pipelines using Python, PySpark, and SQL to ingest, process, and transform data from various sources.

· Implement data quality checks and monitoring to ensure the reliability and integrity of data pipelines.

Data Warehouse Management:

· Build and optimize data warehouse solutions for efficient storage and retrieval of large volumes of structured and unstructured data.

· Collaborate with cross-functional teams to design and implement data models for analytical purposes.

Databricks Expertise:

· Utilize Databricks platform for data engineering tasks, including data ingestion, ETL processing, and data analysis.

· Optimize Databricks jobs and clusters for performance and cost efficiency.

Airflow Workflow Orchestration:

· Configure and manage Airflow workflows to automate data pipeline orchestration and scheduling.

· Implement error handling and monitoring within Airflow DAGs to ensure reliable execution of data workflows.

Linux Administration:

· Manage Linux-based servers and environments for data engineering tasks, ensuring security, reliability, and performance.

· Troubleshoot and resolve issues related to Linux infrastructure and server configurations.

Collaboration and Documentation:

· Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

· Document data engineering processes, workflows, and best practices for knowledge sharing and future reference.

Required skills :

· Bachelor's degree in Computer Science, Engineering, or related field.

· Proven experience as a Data Engineer or similar role, with a minimum of 6 to 8 years of relevant experience.

· Proficiency in Python, PySpark, SQL, and Databricks.

· Good to have familiarity with Claude Code or Github Copilot or Codex ,GEN AI - Databricks Genie, LLM , Prompt engineering

· Familiarity with Airflow for workflow orchestration.

· Experience with Linux administration and shell scripting.

· Strong understanding of data modelling, ETL processes, and data warehousing concepts.

· Excellent problem-solving skills and attention to detail.

· Strong communication and collaboration skills, with the ability to work effectively in a team environment.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status.