Data Engineer Advisor

Role Summary NTT DATA is seeking experienced Data Engineers for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg. The engineer will harden, extend and operate reusable migration frameworks developed during the POC, implement Spark/PySpark migration pipelines, integrate Cyber/tokenization services, implement migration controls and automated reconciliation, optimize Iceberg storage and troubleshoot PB-scale migration workloads. This is a highly hands-on engineering role requiring strong distributed-data-processing fundamentals and experience building reliable, restartable and high-performance data pipelines. Experience 10+ years of Data Engineering experience, including 4+ years hands-on production experience with Apache Spark/PySpark. ________________________________________ Key Responsibilities Migration Framework Engineering •    Harden and extend a config-driven Python/PySpark migration framework capable of migrating thousands of Hadoop/Hive tables without table-specific code. •    Build reusable Hive metadata/schema discovery and migration-configuration components. •    Implement migration strategy selection including CTAS/bulk, partition-wise and chunk-wise processing based on table size, partition/file characteristics, tokenization requirements and execution history. •    Implement PNC-defined Data Movement Controls including: o    checkpointing, o    restart/recovery, o    idempotency, o    retries, o    control totals, o    audit logging, o    exception handling. PB-Scale Migration Execution •    Execute Hadoop/Hive → Apache Iceberg historical migrations at multi-TB/PB scale. •    Design migration patterns for small, medium, large and very-large tables. •    Tune parallel Spark execution across tables, partitions and chunks. •    Implement resumability so failed large-table migrations can restart from safe checkpoints rather than reprocessing entire tables. •    Diagnose and optimize Hadoop-read, Spark-processing, tokenization and Iceberg-write bottlenecks. •    Support migration-wave execution and production issue resolution. Cyber / Tokenization Integration •    Integrate PNC-approved tokenization capabilities, including TaaS/Protegrity where applicable, into migration pipelines. •    Implement secure processing of sensitive fields. •    Handle service failures, retries, throttling and restart scenarios. •    Measure and optimize tokenization impact on end-to-end migration throughput. •    Support technical validation of Cyber/tokenization outcomes based on PNC-defined rules. Apache Iceberg Engineering •    Create/configure Iceberg tables using approved target standards. •    Implement appropriate partitioning and file-sizing strategies. •    Optimize compaction and small-file handling. •    Work with Iceberg snapshots, schema evolution and metadata. •    Troubleshoot Iceberg write/commit/catalog issues. •    Support table maintenance and performance optimization. Data Validation & Reconciliation •    Develop automated source-target validation covering: o    row counts, o    schema conformity, o    partition completeness, o    control totals, o    aggregates/checksums, o    null/duplicate checks, o    tokenization validation where applicable. •    Isolate reconciliation failures at table/partition level. •    Implement selective retry/reprocessing of failed data. •    Produce auditable migration evidence. Production Engineering •    Monitor and troubleshoot distributed Spark jobs. •    Analyze Spark UI, executor logs and resource utilization. •    Troubleshoot Hadoop/HDFS, Spark, Iceberg catalog, object-storage and tokenization issues. •    Build operational telemetry covering migration status, throughput, retries and failures. •    Participate in code reviews, testing, CI/CD and engineering standards. •    Mentor mid-level/junior Data Engineers. ________________________________________ Mandatory Skills Strong hands-on experience with: •    Python •    PySpark •    Apache Spark / Spark SQL •    Hadoop / HDFS •    Hive / HiveQL / Hive Metastore •    SQL •    Apache Iceberg •    Parquet •    Linux •    Git •    distributed data processing •    large-volume ETL/data migration •    source-target reconciliation •    Spark performance tuning. Practical Iceberg experience should include: •    table creation/read/write, •    partitioning, •    snapshots, •    schema evolution, •    file sizing, •    compaction, •    Spark integration. Strong understanding of: •    parallelism, •    checkpoint/restart, •    idempotency, •    retry/recovery, •    large-table processing, •    small-file problems, •    distributed-system troubleshooting. ________________________________________ Preferred Skills •    Apache Polaris/Nessie/Glue or another Iceberg catalog. •    Kubernetes/OpenShift/Spark Operator. •    S3 or S3-compatible object storage; NetApp StorageGRID/ONTAP preferred. •    TaaS/Protegrity or equivalent tokenization/masking technology. •    PostgreSQL for migration configuration/control/audit metadata. •    Trino. •    Control-M/Airflow or equivalent enterprise scheduler. •    CI/CD for Python/Spark applications. •    Banking/financial-services experience. •    PB-scale migration experience. •    HiveQL→Spark SQL modernization experience. •    Bronze/Silver/Gold lakehouse experience.