Senior Data Engineer – Private Markets Data

Location: Boston, MA

About the Opportunity

A leading global investment management firm is seeking a highly motivated

Senior Data Engineer

to join its Private Markets Data team. This individual will be responsible for designing, implementing, and maintaining sophisticated data engineering systems and governance processes while helping develop next-generation data solutions.

The ideal candidate is a well-rounded technologist with strong data engineering expertise, hands-on programming experience, and a passion for building scalable, high-performance data platforms.

This role works closely with data product managers, business analysts, technology teams, and business stakeholders and provides significant exposure to the investment management process.

Key Responsibilities

Design, develop, and maintain on-premises and cloud-based data ingestion and processing pipelines.
Administer and enhance data engineering tools including

Airflow, DBT Cloud, Snowflake, and Python

.
Support and enhance data governance processes and data quality initiatives.
Develop production-quality code and scalable data solutions supporting key business initiatives.
Conduct architecture and code reviews to ensure security, scalability, performance, and quality.
Partner with cloud migration, information security, and business analysis teams to develop new applications and migrate existing systems to cloud-native technologies.
Provide technical leadership and mentorship to onshore and offshore engineering team members.
Help define and execute the technical roadmap while identifying opportunities to improve efficiency, processes, and data quality.
Optimize data pipelines and improve overall job performance and runtime.

Technology & Technical Experience

The ideal candidate will bring experience across a modern data engineering environment, including:

Python

for data processing, automation, and production-grade development.
Advanced

SQL

, data modeling, and data warehouse design.
Snowflake

, including experience building, deploying, and optimizing Snowflake Cortex AI.
Data orchestration and transformation tools such as

Airflow and DBT Cloud

.
Cloud-native

AWS

serverless data and integration technologies.
Docker and Kubernetes

.
Deployment and infrastructure tools including

Terraform, GitHub Actions, and ECR

.
Monitoring and observability platforms such as

Datadog, Grafana, or Amazon CloudWatch

.
Data platforms including

Snowflake, Redshift, BigQuery, or Athena

.
BI/reporting tools such as

Power BI or Tableau

.
Data governance, data quality, metadata management, and lineage frameworks.
Web scraping, APIs, and data integration best practices.
Agile software development environments.

AI & Emerging Technology Experience

Candidates should be comfortable incorporating modern AI technologies into their day-to-day engineering workflow, including:

Large language models such as

Claude and ChatGPT

for data analysis and automation.
Prompt engineering and agentic/workflow automation frameworks.
AI-assisted development tools such as

Claude Code, OpenAI Codex, Cursor, or Windsurf

.
Familiarity with

Model Context Protocol (MCP)

and reusable AI skills is a plus.
Exposure to vector databases and

Retrieval-Augmented Generation (RAG)

pipelines such as Pinecone, pgvector, or Snowflake Cortex Search is a plus.
Modern lakehouse technologies such as

Delta Lake, Apache Iceberg, or Apache Hudi

are a plus.
Data quality and observability tools such as

Great Expectations, Monte Carlo, or Soda

are a plus.
Data catalog and lineage platforms such as

Alation, Collibra, or Unity Catalog

are a plus.
Streaming and real-time processing technologies such as

Kafka, Flink, or Spark Streaming

are a plus.

Qualifications

BS or MS in

Computer Science, Engineering, or a related technical discipline

.
7+ years of professional software or data engineering experience

.
Strong foundation in software design and architectural patterns.
Proven ability to translate complex business requirements into scalable technical solutions.
Excellent communication, analytical, and problem-solving skills.
Experience with Big Data and streaming technologies preferred.
Demonstrated ability to take ownership of projects from concept through completion.
Ability to collaborate effectively with both technical and non-technical stakeholders.
Self-starter who thrives in a fast-paced, highly collaborative environment.
Strong commitment to continuous learning and emerging technologies.

Job description