About the role

Cobalt builds expert data and evaluation infrastructure for AI developers. We are recruiting principal-level software engineers for a contract project that tests whether frontier AI models can carry out real software engineering work in a command line environment. You will design realistic engineering tasks that a leading AI model cannot solve, then review the code it produces to understand how and why it fails.

What you will do

Design self-contained command line tasks drawn from real software engineering work, such as fixing a subtle bug in a large unfamiliar codebase, completing a refactor that must preserve behavior, migrating a project across a breaking dependency or language version upgrade, implementing a non-trivial algorithm or data structure to a strict performance target, resolving a difficult merge or a corrupted Git history, and repairing a failing build or test suite.
Build the task environment, including the codebase and dependencies, write a reference solution, and write automated tests that verify whether a solution is correct. Tests need to check behavior thoroughly enough that a superficial or hard-coded fix does not pass.
Run your task against a frontier AI model, review its code with the same rigor you would apply to a pull request from a colleague, and refine the task until the failure reflects a real gap in the model's capability rather than ambiguity or trick wording.
Work with reviewers to bring each task to acceptance.

Who we are looking for

A PhD in computer science, software engineering, or a closely related field.
Industry or academic professional software engineering experience, including time as a senior technical reviewer on a large codebase.
At least one publication, either academic (for example a peer-reviewed paper) or professional (for example a conference talk, a detailed engineering article, or a widely used open-source project that you created or maintain).
Expert-level skill in at least two widely used languages, such as Python, TypeScript, Java, Go, Rust, or C++.
Fluency in the Linux command line, shell scripting, Docker, Git, and common build and testing tools.

Why Cobalt AI:

Advance frontier AI where it counts. Apply your research expertise to the data that frontier labs cannot obtain any other way, where your reasoning directly shapes how the next generation of models works through technical problems.
Grow professionally.

Expand your influence through evaluation projects, advisory roles, and research collaborations, while deepening your understanding of how frontier models are trained and assessed.
Work with a top-tier network.

Collaborate with researchers from leading institutions and labs on high-impact, flexible work.
Set your own schedule.

Flexible 10 to 40 hour weeks that fit around your research position and your life.
Competitive pay.

Rates vary by project and are determined by a number of factors, including scope, skillset, and experience.

Principal Software Engineer, Expert Code Reviewer for AI Models (PhD)

Job description