Trantor- Lead Data Engineer

Job Description

Lead Data Engineer

Experience

10+ years of relevant data engineering experience, with a strong track record of designing and leading large-scale data platforms and a working knowledge of Large Language Models (LLMs) and GenAI-enablement.

Location

12:00 Noon – 10:00 PM IST

Bangalore (Hybrid – 2–3 days in-office per week)

Profile Summary

About Company

Trantor is a pioneer in creating enterprise technology solutions and state-of-the-art CaptiveCoE™ assisting enterprises across the globe with their digital transformations and business needs. Our commitment to excellence and authenticity has led to long-term working relationships with our clients and solution partners.

Website: https://www.trantorinc.com

We are seeking an experienced Lead Data Engineer to drive the design and delivery of scalable, reliable, and cost-efficient data platforms. This role requires deep, hands-on expertise across the modern data engineering stack — distributed data processing, data modeling, ETL/ELT pipeline development, and workflow orchestration — along with a solid understanding of Large Language Models (LLMs) and their data requirements. This role requires strong, hands-on expertise with the Databricks Lakehouse Platform, including the medallion (bronze/silver/gold) architecture, Apache Spark, and Delta Lake. The ideal candidate will combine strong engineering fundamentals with technical leadership, mentoring engineers, setting best practices, and partnering with architects, data scientists, and business stakeholders to enable advanced analytics and GenAI use cases.

Key Responsibilities

• Lead the design and development of scalable, high-performance data pipelines and platforms across batch and streaming workloads
• Architect and maintain ETL/ELT workflows for data ingestion, transformation, and curation across the medallion (bronze/silver/gold) architecture
• Define and govern data models, schemas, and storage strategies for data lakes and data warehouses to support analytics, reporting, and ML/LLM workloads
• Build and curate high-quality datasets, feature pipelines, and retrieval-ready data stores (including embeddings/vector data) to support LLM and GenAI applications
• Establish and enforce engineering best practices, coding standards, code reviews, and CI/CD for the data engineering team
• Implement workflow orchestration and scheduling using tools such as Apache Airflow (or equivalent), and automate end-to-end data workflows
• Lead migration of data warehouses and pipelines from legacy on-premise and cloud platforms to modern cloud-native and lakehouse architectures
• Apply performance tuning and cost-optimization techniques across distributed processing, partitioning, caching, and compute configuration
• Establish data governance, security, lineage, and access control aligned with compliance and privacy requirements
• Build monitoring, logging, alerting, and data-quality frameworks to ensure pipeline reliability and proactive issue resolution
• Mentor and guide data engineers, and collaborate with architects, data scientists, and business stakeholders to translate requirements into robust technical solutions
• Participate in and lead Agile/Scrum ceremonies including sprint planning, daily stand-ups, and retrospectives

Required Skills (Mandatory)

• 10+ years of hands-on data engineering experience, including technical leadership of data platform initiatives
• Strong, hands-on expertise with the Databricks Lakehouse Platform, including Delta Lake, Delta Live Tables, Databricks Workflows, and Unity Catalog
• Proven experience designing and implementing the medall…