Senior Data Engineer

• We’re seeking a highly skilled Data Engineer to design, build, and maintain production-grade data pipelines that process and transform terabytes of data.

• In this role, you’ll collaborate closely with data scientists and other SWEs to ensure that our data infrastructure is scalable, reliable, and cost-effective

• Architect, develop, and deploy batch and streaming pipelines using Airflow and containerized workflows for cyber-security use-cases

• Containerize data-processing jobs with Docker, orchestrate with Kubernetes, and manage releases with Helm charts

• Build high-throughput data transformations using Dask or Apache Spark

• Maintain training data clusters across hybrid (on-prem and cloud environments)

• Optimize training jobs for performance, resiliency, and cost

• Implement observability (logging, metrics, alerting) to maintain pipeline health and SLA adherence

• Troubleshoot, debug, and resolve data-processing failures in production

• Work with cross-functional teams to define data contracts, schemas, and quality checks

• Enforce software engineering best practices: CI/CD, code reviews, automated testing, and documentation

• Design and maintain data models and schemas for AI/ML continuous training use cases

• Load data into cloud storage and lakes, ensuring performance and accessibility

Benefits

• ESOP

• Incentives and Bonuses

• Statutory Benefits: PF, ESI, Maternity Benefits, Leaves & Holidays

• VPF, NPS, Meal Vouchers, Paternity Leaves

• Health: Medical, Accidental, Term Life Insurances

• Relocation Benefits

• Flexi Working

• Reward & Recognition Policies

• Transparent Performance Management Process

• Career Progression

• Global Mobility- Containerization & Orchestration: Expertise with Docker, Kubernetes, and Helm

• ML Pipelines: Exposure to deploying cross-cluster model-training workflows using Ray or similar frameworks

• GCP Proficiency: Hands-on with Google Cloud services (e.g., Pub/Sub, Big Query, Cloud Storage, GKE). Equivalent experience in other public cloud providers is fine

• 3–5 years of professional experience designing and operating production data pipelines at scale

• Compiled Languages: Experience writing data services in Go or Rust

• Cloud Fundamentals: Familiarity with deploying and managing services in a cloud environment

• Security & Compliance: Knowledge of data governance, encryption, and role-based access control

• Programming: Strong proficiency in Python for data engineering tasks

• Distributed Frameworks: Practical experience with Dask or Apache Spark for large-scale data processing

• Workflow Management: Hands-on experience building DAG-based pipelines in Apache Airflow

• Infrastructure as Code: Familiarity with Terraform for deployment