Senior Data Engineer

Role Summary NTT DATA is seeking a highly experienced Senior Data Engineer / Technical Lead for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg. The role requires a strong hands-on engineer who can lead the engineering team while continuing to design, develop, troubleshoot and optimize the solution. The candidate will own technical execution of the migration framework, Spark/PySpark engineering, Iceberg implementation, data modelling/schema mapping, Cyber/TaaS integration, migration controls, reconciliation, performance tuning and technical delivery of migration waves. This is not a coordination-only Technical Lead role. The candidate is expected to contribute to code, conduct design/code reviews, troubleshoot complex production issues and mentor Data Engineers. Experience 10+ years of Data Engineering experience, including: •             5+ years hands-on Apache Spark/PySpark. •             Strong Hadoop/Hive experience. •             Experience leading Data Engineering teams. •             Large-scale data migration/data modernization experience. •             Strong data modelling and source-to-target mapping experience. Multi-hundred-TB/PB-scale migration experience is strongly preferred. Key Responsibilities Technical Leadership •             Lead a team of Data Engineers through assessment, development, migration, validation and production execution. •             Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns. •             Own technical estimates, engineering backlog, work allocation and technical delivery. •             Conduct design and code reviews and enforce engineering standards. •             Mentor Data Engineers and provide hands-on technical guidance. •             Act as the primary engineering escalation point for complex migration and production issues. •             Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams. •             Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions. Hadoop-to-Iceberg Migration •             Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg. •             Harden and extend the reusable, configuration-driven migration framework developed during the POC. •             Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity. •             Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing. •             Design execution approaches for small, medium, large and very-large tables. •             Optimize parallel migration across tables and partitions. •             Implement checkpointing, restart/recovery, idempotency and safe retries. •             Lead technical execution and troubleshooting of migration waves. Data Modelling & Schema Engineering •             Analyze existing Hive data models and translate them into approved Apache Iceberg physical models. •             Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration. •             Perform source-to-target data mapping and schema/data-type mapping. •             Define and implement table structures consistent with approved Iceberg standards. •             Analyze: o             entities and relationships, o             primary/foreign-key relationships, o             data types, o             partition keys, o             business keys, o             reference/cross-reference structures. •             Identify schema incompatibilities and recommend remediation. •             Work with the Data Architect on partitioning, file-layout and model-design decisions. •             Ensure migrated models preserve required business relationships and downstream compatibility. •             Support future Bronze/Silver/Gold modelling requirements where applicable. Important: enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards. Spark/PySpark Engineering & Performance •             Develop and review production-grade Python/PySpark components. •             Tune Spark jobs for PB-scale migration. •             Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks. •             Optimize table/partition/chunk parallelism. •             Tune workloads to meet migration-throughput objectives. •             Identify bottlenecks across: Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation •             Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework. Apache Iceberg Engineering •             Implement approved Iceberg table standards. •             Develop/configure Iceberg tables and Spark→Iceberg processing. •             Implement: o             partitioning, o             schema evolution, o             snapshots, o             file sizing, o             compaction, o             metadata maintenance, o             small-file remediation. •             Troubleshoot Iceberg catalog/write/commit issues. •             Optimize Iceberg table performance and storage characteristics. Cyber-3 / Tokenization •             Implement PNC-approved TaaS/Protegrity tokenization and retokenization patterns. •             Integrate tokenization into Spark migration pipelines. •             Implement batching, retry, timeout and recovery handling. •             Troubleshoot tokenization failures/performance bottle