Sr. Data Engineer

• Experience: 8+ years in data engineering, with a strong focus on large-scale, distributed, cloud-native systems.

• Snowflake depth (primary): Proven, hands-on production experience with Snowflake — warehouse tuning and cost optimization, RBAC and data governance, Streams/Tasks, Snowpipe, Time Travel, and Snowpark. You can reason about query profiles and micro-partitioning, not just write SQL.

• Core languages: Expert Python and strong, advanced SQL; PySpark/Snowpark a plus.

• Transformation & orchestration: Production experience with dbt and a workflow orchestrator (Airflow strongly preferred), designing for portability across engines.

• AWS ecosystem: Solid hands-on experience with S3, Glue, Step Functions, IAM, and event services (SNS/SQS).

• Streaming: Track record building streaming/CDC pipelines with Kinesis or Kafka and tools like Debezium, Fivetran, or Airbyte.

• Data quality: Demonstrated ownership of automated testing, data profiling, reconciliation, and pipeline validation — you own the QA of your own pipelines.

• Ways of working: Strong documentation habits (playbooks, technical specs), a bias toward reproducibility and version control, and a clear ownership mindset.

• Top desirable — Automotive domain: Direct experience with automotive repair-order data, dealership fixed-operations, or DMS feeds is a top differentiator for this role.

• Top desirable — Databricks: Hands-on Databricks experience (Spark, Delta Lake, Unity Catalog, Workflows) used alongside Snowflake. We value portability across engines and may extend the stack over time, so multi-platform depth is highly valued.

• Certifications: SnowPro Core or Advanced (Data Engineer), Databricks Certified Data Engineer Professional, or AWS Certified Data Engineer.

• Governance & lineage: Exposure to cataloging/lineage tooling (OpenMetadata) and modern CI/CD for data (tested, peer-reviewed deployments).

Key Responsibilities

1. Snowflake Warehouse Engineering

• Design and maintain governed, well-modeled schemas in Snowflake using dimensional and Medallion (bronze/silver/gold) patterns, applying dbt for modular, version-controlled transformations.

• Optimize Snowflake for performance and cost — right-size virtual warehouses, manage clustering keys and micro-partition pruning, tune query profiles, and control credit consumption through resource monitors and warehouse scaling policies.

• Leverage native Snowflake features such as Streams and Tasks (or external orchestration), Snowpipe / Snowpipe Streaming for continuous ingestion, Dynamic Tables, Time Travel, Zero-Copy Cloning, and Secure Data Sharing.

• Implement governance with role-based access control (RBAC), row-access and masking policies, and object tagging for PII and sensitive automotive/customer data.

2. Pipeline Development & AWS Data Lake Engineering

• Build and orchestrate complex ETL/ELT pipelines through a central control plane (e.g. Airflow with astronomer-cosmos) that runs consistently across AWS, Snowflake, and Databricks, favoring portable, engine-agnostic designs over transformations locked into any single platform's native constructs.

• Manage the data lake on AWS S3 as the primary landing zone, optimizing open storage formats (Iceberg, Parquet, Delta) and integrating them with Snowflake external tables and Iceberg tables.

• Build decoupled, event-driven architectures using SNS and SQS for high-throughput messaging between data services.

3. Real-Time Data Streaming & Ingestion

• Develop low-latency ingestion using AWS Kinesis or Kafka, feeding operational analytics through Snowpipe Streaming and Kafka connectors.

• Implement Change Data Capture (CDC) with Debezium, Fivetran, or Airbyte to keep Snowflake in near-real-time sync with source systems windowed on update timestamps.

4. Data Quality, Reconciliation & QA Ownership

• Own end-to-end data quality by embedding automated tests directly into ETL/ELT flows — using portable frameworks such as dbt tests and Great Expectations that apply uniformly whether transformations run on AWS, Snowflake, or Databricks — rather than treating QA as a separate downstream step.

• Build cross-engine reconciliation between source systems and Snowflake — control totals, partition/row hashing, and drill-down on discrepancies — to guarantee completeness and correctness at scale.

• Enforce data contracts and schema-evolution guidelines, and stand up proactive observability and alerting to catch data drift and pipeline anomalies before they reach users.

5. Engineering for ML/AI

• Engineer ML-ready datasets and feature pipelines to support the Data Science team, managing feature freshness and lineage.

• Operationalize AI workflows using Snowflake Cortex, Snowpark for Python, and AWS Bedrock where appropriate.

6. Technical Leadership & Collaboration

• Mentor engineers on coding standards, SQL and Snowflake optimization, dbt modeling, and Python development, and lead code reviews.

• Partner with architects, product, and analytics teams to translate designs into reliable, documented, production code, and help drive SDLC and CI/CD maturity across the data platform.