Senior Data Engineer

Specific Accountability :-

• Lead the design, build, and optimization of batch and real-time data pipelines.

• Own data model design, ingestion frameworks, transformation logic, and platform reliability.

• Ensure high standards for data quality, lineage, governance, security, and compliance.

• Drive performance tuning, cost optimization, monitoring, and incident resolution across data platforms.

• Translate business and analytics needs into scalable technical solutions and delivery plans.

• Guide code reviews, technical standards, and release practices for data engineering work.

• Mentor team members, allocate work, and provide technical leadership on complex initiatives.

• Collaborate with value delivery owners, architects, analysts, scientists, and application teams to enable trusted data products

Desired Candidate Profile

Experience level: 10+ years hands-on data engineering with production pipeline ownership; at least one platform or system owned end to end.

Data transformation and modeling

• dbt (data build tool): models, tests, macros, incremental models, project structure

• Data modeling: dimensional modeling, star schema, medallion architecture (raw / curated / presentation, or bronze / silver / gold)

• Data warehousing and lakehouse architecture concepts

Orchestration

• Apache Airflow: DAG design, task dependencies, triggers, sensors, parameterized runs, failure handling, scheduling

• Workflow orchestration and monitoring of production pipelines

Data ingestion and loading

• Batch data ingestion and ETL / ELT pipeline development

• Load patterns: full load, incremental load, upsert / merge, change data capture (CDC)

• Multi-source ingestion: SFTP, REST APIs, relational databases, file-based ingestion

• Data quality, data validation, schema evolution handling

Query engine, storage, and table formats

• Dremio (or equivalent lakehouse query engine: Trino, Presto, Starburst, Athena)

• Apache Iceberg (or equivalent open table format: Delta Lake, Apache Hudi)

• S3 / S3-compatible object storage (AWS S3, MinIO, Dell ECS)

• Partitioning, table layout, and query performance tuning

Programming and databases

• Python (pipeline development, automation, scripting)

• Advanced SQL, query optimization, performance tuning

• PostgreSQL and/or SQL Server (relational databases)

Non-relational / NoSQL databases (must-have, not just exposure)

The platform ingests from and operates alongside several non-relational stores. The candidate must be able to work with them as a data engineer, meaning read from them, model their data into the lakehouse, and reason about their consistency and performance characteristics. Screen for hands-on experience with at least two of the following families:

• Key-value / in-memory stores: Redis (caching, queues, state; TTL and eviction behaviour; using

Redis as a pipeline coordination or lookup layer)

• Document databases: MongoDB, Couchbase, or DocumentDB (nested JSON documents, schema-on-read, flattening documents into relational / lakehouse tables)

• Wide-column stores: Cassandra, ScyllaDB, or HBase (partition-key design, read patterns, extracting at scale)

• Search / analytics engines: Elasticsearch or OpenSearch (index design, querying, extracting data out of an index for analytics)

Data formats and semi-structured data

• Columnar formats: Apache Parquet (required), ORC; row formats: Avro

• Text and semi-structured formats: JSON, JSON Lines, CSV / delimited files, XML; encoding, delimiter, and quoting edge cases

• Compression codecs (Snappy, gzip, zstd) and their effect on storage and query

• Handling nested and evolving schemas end to end (schema inference, drift detection, schema evolution: column add / delete / type change)

Distributed processing

• Apache Spark / PySpark (DataFrame API, partitioning, joins, skew handling, writing Iceberg /

Parquet); or equivalent distributed engine

• Understanding of parallelism, shuffles, memory tuning, and cost of distributed jobs

Scripting, Linux, and connectivity

• Bash / shell scripting and Linux command line (the deployment and operational tooling is script- based)

• SFTP / SSH: key management, automated transfers, file-based handoffs (verify / result files, archiving, retries)

• TLS certificates and secure connections (understanding certificate verification; never disabling it in production)

• Connectivity troubleshooting: DNS, firewalls, proxies, connection timeouts, service-to-service networking inside Kubernetes / OpenShift

• YAML and configuration management for pipelines, dbt, Airflow, and Kubernetes manifests