Senior Data Platform Engineer - Azure Databricks

For roles that are 100% remote or hybrid, you must have access to a reliable high-speed internet connection to support daily job responsibilities. A minimum bandwidth of 50 Mbps download and 5 Mbps upload is required. Those fully remote associates residing in states where service is required by contract, law, or regulation will be allowed to submit for reimbursement.

Role Overview

We are seeking a skilled

Senior Data Platform Engineer

with hands-on experience in Azure Databricks and a strong background in healthcare environment. The role is responsible for designing, building, and optimizing our enterprise data platform. The ideal candidate will be responsible for maintaining scalable data processing environments in Azure, while ensuring seamless integration across Azure Databricks, DevOps, HVR/ Fivetran, Snowflake, Collibra, Power BI, and Tableau ecosystems.

The ideal candidate will have deep experience in cloud engineering at least in one of the major cloud providers (Azure, AWS or GCP), strong hand-on experience of Kubernetes, containers, serverless and modern architectures, experience with Infrastructure as code (Terraform/ Ansible), solid understanding of networking security, IAM, monitoring and automation. This role bridges data architecture, data engineering, and data governance — enabling secure, scalable, efficient and resilient cloud data platform foundation.

Work Arrangement

• This is a hybrid, office based position working at our Ellis Preserve office in Newtown Square, PA. Office days are Tuesdays, Wednesday, and Thursday; Remote days are Monday and Friday. WE WILL NOT CONSIDER REMOTE CANDIDATES FOR THIS POSITION!

Responsibilties

Azure Databricks Platform Engineering

• Design, deploy, and maintain scalable Databricks clusters and workspaces within Azure

• Automate cluster management, job orchestration, and environment provisioning using Infrastructure-as-Code (IaC) tools (e.g., Terraform, ARM, or Bicep)

• Optimize Spark jobs, delta pipelines, and data lakehouse performance

• Implement robust CI/CD pipelines for Databricks workflows and notebooks

• Manage environment configurations, libraries, secrets, and access controls

Integration & DevOps

• Implement and manage Azure DevOps pipelines for Databricks code deployment, testing, and release management

• Develop reusable CI/CD templates for data ingestion and transformation workflows

• Collaborate with development teams to enable version control, code quality checks, and automated testing

Data Transformation & Orchestration

• Support and integrate Databricks and/or dbt (Data Build Tool) workflows within Databricks for modular SQL-based transformations

• Ensure efficient orchestration of ETL/ELT pipelines and integration with data lake and data warehouse environments

Data Governance & Metadata Management

• Integrate Databricks and Azure data assets with Collibra for data cataloging, lineage tracking, and governance

• Work closely with data governance teams to ensure compliance with data quality and metadata management standards

Analytics & Visualization Integration

• Enable secure and performant data access for Power BI and Tableau through optimized data models and APIs

• Collaborate with BI developers to design efficient datasets and dashboards

Team Collaboration

• Work with stakeholders, including the Product, Data and Design teams to assist with platform-related technical issues and support their data needs

• Strong team collaboration and experience working with remote teams

• Mentor and guide other platform engineers through reviews, training etc

Compliance & Best Practices

• Support implementing data quality, validation, and governance processes to ensure compliance with healthcare regulations (e.g., HIPAA)

• Understand and deploy best practices in data engineering

Education And Experience

• Bachelor’s or Master’s degree in computer science, Information Technology, Data Engineering, or related field

• At least four to seven (4 – 7) years of hands-on experience with Azure Databricks in enterprise environments

• Strong proficiency in PySpark, SQL, and data lakehouse architecture

• Deep understanding of Azure ecosystem – including Azure Data Lake, ARO, AKS, Key Vault, Azure Ansible and Azure DevOps

• Experience with CI/CD implementation using Azure DevOps

• Working knowledge of pyspark, Collibra, Power BI, and Tableau integration workflows

• Familiarity with data governance, security, and compliance best practices

• Proficiency in scripting languages (Python, Bash, PowerShell)

• Excellent problem-solving and cross-functional collaboration skills

• Experience with Terraform or other IaC tools for Azure

• Familiarity with Delta Live Tables, Unity Catalog and MLflow

Other Preferred Experience

• Familiarity with DQ frameworks, or observability tools in Azure/Databricks

• Knowledge of data governance, metadata management, and audit controls in a healthcare setting

• Experience with Lakehouse architecture and medallion design patterns

• Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203) or Azure Solutions Architect Expert

• Exposure to Agile/Scrum delivery models

Our Comprehensive Benefits Package

Flexible work solutions including remote options, hybrid work schedules, Competitive pay, Paid time off including holidays and volunteer events, Health insurance coverage for you and your dependents on Day 1, 401(k) Tuition reimbursement and more.

Your career starts now. We’re looking for the next generation of health care leaders.

At AmeriHealth Caritas, we’re passionate about helping people get care, stay well and build healthy communities. As one of the nation's leaders in health care solutions, we offer our associates the opportunity to impact the lives of millions of people through our national footprint of products, services and award-winning programs. AmeriHealth Caritas is seeking talented, passionate individuals to join our team. Together we can build healthier communities. If you want to make a difference, we’d like to hear from you.

Headquartered in Newtown Square, AmeriHealth Caritas is a mission-driven organization with more than 30 years of experience. We deliver comprehensive, outcomes-driven care to those who need it most. We offer integrated managed care products, pharmaceutical benefit management and specialty pharmacy services, behavioral health services, and other administrative services.

Discover more about us at www.amerihealthcaritas.com.