Senior / Lead Data Engineer – Tata Consultancy Services (TCS)

Opportunity Overview

Lakehouse Architecture & Technical Execution

The Senior / Lead Data Engineer at TCS architects scalable, enterprise-grade data platforms using Databricks Lakehouse and Medallion Architecture principles:

┌───────────────────────────────────────────┐
│ 1. Bronze Layer (Raw Ingestion)           │ ➔ Stream and batch ingest raw data from AWS/Azure sources into Delta Lake
└─────────────────────┬─────────────────────┘
                      ▼
┌───────────────────────────────────────────┐
│ 2. Silver Layer (Cleansed & Enriched)     │ ➔ Execute PySpark data transformations, enforce schema validation & deduplication
└─────────────────────┬─────────────────────┘
                      ▼
┌───────────────────────────────────────────┐
│ 3. Gold Layer (Business Aggregations)     │ ➔ Build production-ready, aggregated data models tailored for BI and AI workloads
└─────────────────────┬─────────────────────┘
                      ▼
┌───────────────────────────────────────────┐
│ 4. Platform Leadership & Optimization     │ ➔ Lead technical teams, optimize compute performance, tune queries & implement MLOps
└───────────────────────────────────────────┘

Key Responsibilities

• Lakehouse Platform Architecture: Design and deploy robust enterprise data platforms utilizing Databricks Lakehouse and Medallion Architecture (Bronze, Silver, Gold layers).

• Pipeline Development & Orchestration: Build, deploy, and monitor complex Spark ETL/ELT data pipelines using PySpark, Delta Lake, and orchestration tools (Databricks Workflows, Airflow, or Azure Data Factory).

• Performance Tuning & Optimization: Conduct deep-dive performance tuning on Databricks clusters, optimize Spark memory management, and tune Delta Lake tables (Z-Ordering, Liquid Clustering).

• Cloud & Real-Time Data Integration: Architect streaming pipelines (Kafka, AWS Kinesis, Azure Event Hubs) and integrate secure cloud storage solutions on AWS (S3) or Azure (ADLS Gen2).

• Technical Leadership & Mentorship: Lead team execution, enforce data quality standards, mentor junior engineers, and drive best practices across data governance and MLOps/GenAI integration.

Qualification Matrix & Technical Skill Stack

Core Requirements

Key Focus Areas for Interview Preparation

1. Databricks Enterprise Architecture: Be prepared to present complex Medallion Architecture implementations, highlighting data governance (Unity Catalog), security, and cost optimization strategies.

2. Advanced Spark Optimization: Practice scenario questions on diagnosing Spark out-of-memory errors, data skewness handling, partitioning strategy, and Delta Lake performance tuning.

3. Leadership & System Design: Review end-to-end data platform system design for multi-cloud deployments, real-time streaming architectures, and technical team mentoring approaches.