There are two main directions for projects: verifying existing critical software stacks (government, AI, and financial infrastructure), and developing scalable oversight for AI-driven software development. Here are some example projects that scholars might work on:

1. Measuring software verification capability of frontier models. For example, we’re working on projects in translating CompCert from Coq to Lean, which helps to uncover challenges in real world use of LLMs for software verification, and developing a benchmark suite based on the Software Foundations textbook to track progress in frontier models capability.
2. Formalizing the ML tech stack using AI assistants. For example, we’re working on formalizing StableHLO, so that we can prove security properties of custom backends for optimizing machine learning workloads.

There is some flexibility, and I’m open to scholar proposals. However, I’m primarily interested in research that relates to monitor and control measures for oversight of complex, agentic tasks. For examples in this direction, see the papers below.

This stream is focused on reducing catastrophic risks from large language models (LLMs). Their research spans several areas:

1. Developing model organisms of misalignment, e.g. of deceptive alignment, to build a better understanding of what aspects of training are more likely to lead to deceptive alignment.
2. Developing techniques for process-based supervision, such as learning from languagefeedback.
3. Finding tasks where scaling up models result in worse behavior (inverse scaling), to gain an understanding of how current training objectives actively incentivize the wrong behavior (e.g., sycophancy or reward-tampering or alignment-faking).
4. Improving the robustness of LLMs to red teaming (e.g., by red teaming with language models or pretraining with human preferences or red teaming with best-of-n jailbreaks)
5. Investigating the risks and benefits of training predictive models over training agents, e.g., understanding the extent to which the benefits of RLHF can be obtained by predictive models, and the extent to which RLHF models can be viewed as predictive models.
6. Scalable oversight – the problem of supervising systems that are more capable than human overseers
7. Advancing security through investigating adversarial machine learning, cybersecurity evals, and understanding currently possible real-world attacks
   These projects involve running a large number of machine learning experiments, to gain empirical feedback on alignment techniques and failures.

Projects I’d be particularly interested in having MATS scholars work on span the following topics:

• Reward hacking in LLMs trained with RLAIF or reasoning approaches: are misleading behaviors emergent from RL (such as those in this paper) going to be present with reasoning models (and in RLAIF models)? If so, how? The goal would be to have a good misalignment demo and try to find mitigations.
• We have found that RL can heavily distort CoT reasoning, leading to extreme “motivated reasoning”. We’re interested in studying this phenomenon in more depth. How robust is the emergence of motivated reasoning to different training setups? May this partially explain why we see scheming behaviors or alignment faking? Some reasons why motivated reasoning may be a possible explanation are here.
• AI alignment under changing and influenceable values and preferences: we want to align AI systems to values and preferences that can change (and be changed by AI systems). Doing so is hard. Can we leverage LLM’s knowledge of in this domain (potentially optimizing against them as reward models)? Can we use them to better elicitmeta-preferences, and try to reduce naturally emerging influence incentives?

I’d also be open to scholars proposing their own projects.

My MATS stream is putting interp (and other white box tools) to the test. We’re going to train hundreds of misaligned models in a red-team/blue-team capture-the-flag game.

See more here: https://www.lesswrong.com/posts/jWFvsJnJieXnWBb9r/alignment-faking-ctfs-apply-to-my-mats-stream

The goal of this project is to build a great LM agent scheming black-box monitor that works in general settings with large action spaces and for a very general definition of scheming. You can find a detailed project description here: https://docs.google.com/document/d/1FDgh4ioygjqVionP3i8i163zaXT9hqAV-gEikCNhlTE/edit?usp=sharing

Since the field moves quickly, it’s too early to list exact projects. But here are rough directions to give some flavor:

• Creating environments that can be used to evaluate and stress-test monitoring techniques (such as e.g. the APPS AI control environment)
• Creating model organisms of dangerous behavior (e.g., https://arxiv.org/abs/2501.13011 finetune a model to execute bad policies in the type of environment mentioned above, rather than requiring human-designed scaffolds)
• Building evals for capabilities related to scheming (such as Subversion Strategy Eval)
• Adversarially stress-testing monitoring techniques (e.g. extensions of Obfuscated Activations for white-box monitors)
• Improved white-box monitoring methods
• Chain of thought monitoring (e.g. building/testing monitors, studying faithfulness/steganography, testing for which tasks CoT is needed or helps improve performance)

We’ll discuss projects in areas like this with scholars at the beginning of MATS. So we encourage you to apply if a decent fraction of these directions (or the general theme) sound exciting to you; you do not need to be excited about every single potential project.

I’m mostly interested in empirical projects around control and security measures for LLM agents, automated red teaming, and capability and alignment evals for LLM agents. Some ideas of projects I might be interested in:

1. How does the offense-defense balance between malicious LLM agents and monitors shift as we increase model capability or inference-time compute?
2. How to build and evaluate agentic monitors that take actions to investigate whether a given trajectory of a suspicious LLM agent is malicious?
3. A human study of how to use a limited budget of human audits when overseeing an LLM agent
4. How far can blackbox and whitebox AI control monitors generalize?
5. How faithful does a chain-of-thought reasoning trace have to be enable oversight via chain-of-thought monitoring?
6. How capable are LLMs of latent reasoning (without chain-of-thought) and how does it scale with model size?
7. Do LLM agents have consistent goals? Evaluations of LLM agents propensity for different behaviors and other topics in blackbox LLM psychology.

We are interested in supporting scholars building evaluations or conducting research projects that measure autonomy and loss of control risks from AI systems. This includes, but is not limited to, threat modelling work, developing evaluations, as well as more focused research projects. Our focus areas include sandbagging and deception, complex; long-horizon agentic evaluations, multi-agent evaluations, and control evaluations. Additionally, we value any research projects that will help us be better informed about future progress in autonomy capabilities - e.g. those which assess barriers to autonomy, identify potential autonomy overhangs, or help predict models’ future capabilities as they relate to autonomy capabilities.

• Emergent Misalignment. We’ve recently published a well received paper about Emergent Misalignment. This is a surprising phenomenon where training models on a narrow task of writing insecure code makes them broadly misaligned. There are some open problems we want to investigate.
• Faithfulness. Do models explain factors that influence their decisions? See our latest paper on faithfulness in reasoning models (e.g. DeepSeek).
• Steganography. Can models encode hidden information that looks “normal” to humans?
• Detecting deception. Through black-box means, can we detect when models “lie” to us? See our paper on catching an AI liar. See also Reversal Curse.
• Limits of multi-hop and out-of-context reasoning. To what extent can models reason without step-by-step reasoning? This can inform us of the extent of the dangerous capabilities of the models. See our paper on connecting the dots.
• Situational Awareness. How aware are current models of themselves and their environment? See our paper on the Situational Awareness Dataset.

I will share more detailed project proposal docs closer to the program start. Example research projects I’m excited about right now (these may change):

• Building a comprehensive “scheming thoughts” dataset for evaluating chain-of-thought monitors
• Developing a methodology for chain-of-thought monitoring control evaluations
• Building evaluations for scheming precursor capabilities, e.g. (opaque) stealth reasoning, cross-instance coordination
• Developing honeypots for upfront audits

Three concrete project proposals:

1. Realistic benchmarks for evaluating oversight subversion
   1. A core bottleneck to scalable/automated oversight research is high-quality, realistic benchmarks for studying oversight subversion. To help build effective oversight protocols, this project aims to evaluate AI agents’ capabilities to autonomously subvert oversight mechanisms in realistic environments.
2. Collusion in automated oversight regimes
   1. Past work typically assumes that the automated overseer is trustworthy, for example, because it is not capable enough to pose a threat. We aim to relax this assumption to study the problem that frontier AI systems might collude with one-another to subvert human control.
3. Automated shutdown protocols and honeypots
   1. We have developed an automated shutdown mechanism, wherein, frontier AI agents are trained to carry out a “shutdown protocol” if, and only if, they see a specific password. By leaving shutdown-passwords in honeypot style traps, AI companies can use automated shutdown protocols as an extra layer of defense against scheming systems which go undetected by oversight in internal deployment.

Our group’s goal is to establish foundational principles and practical techniques for building safe and secure AI systems. We focus on understanding how the complex reasoning, multimodal understanding, and interactive capabilities of these systems may introduce novel safety and security issues or offer unique defense opportunities. Our work is organized around these key areas:

• Evaluation & Automated Red Teaming: Developing robust methodologies and automated tools—including benchmark creation and stress-testing—to proactively identify and address safety and security issues.

• Interpretability & Control: Enhancing transparency to reveal the decision-making processes of AI models, thereby identifying root causes of failures and bolstering trust in high-stakes applications.

• Robust Defenses: Designing novel mechanisms (e.g., with powerful reasoning models) to detect, mitigate, and defend against adversarial attacks and other safety and security threats, ensuring reliable operation in complex environments.

• Hallucinations: Develop advanced tools to identify, analyze, and understand the underlying causes of hallucinations in LLM models, enabling more reliable outputs.

• Alignment & Post-Training: Investigating advanced techniques to align AI systems with human guidelines after the pretraining phase. This includes reinforcement learning from human feedback, alignment fine-tuning, and continuous post-deployment evaluations to guard against drifts and misalignments over time.

Research topics for MATS scholars can be flexible depending on individual interests and alignment with our goals. We are broadly interested in the areas outlined above and on our website: secure-ai.github.io/

We are interested in mentoring projects on technical governance solutions to AI risks, which could be applied at either an ecosystem- or lab-level. Exciting projects might include choosing an open problem in technical governance (see here) and making progress against this problem, or making progress on understanding what exactly makes this challenging and what a viable solution might entail. We are also interested in mentoring projects related to demonstrating and/or mitigating specific risks of AI systems, such as through building novel evaluations or demonstrating a protocol for catching these risks in-practice.

In general, by default, projects will aim to publish an arXiv preprint and will ideally be (small) team projects. Below are several tentative project options. Scholars are also welcome to propose their own; we’d aim to find a project that all involved are excited about.

• Technical projects for verifying AI treaties: As outlined in a forthcoming report, there are many technical open problems in verifying compliance with a hypothetical international agreement on AI. This project will aim to make progress on one such problem, to help enable international cooperation on AI. The following are several examples of more specific potential projects. (1) If an untrusted user trains an LLM on a compute cluster with trusted system software, verify that the model weights are in claimed memory locations. (2) Model, in detail, the relationship between a GPU’s MFU and its power use. (3) Check if ML workload code “egregiously” wastes substantial model FLOP without affecting results (distinct from having inefficient MFU).

• Metascience for AI safety and governance: How can we effectively make progress on the many R&D challenges involved in AI safety and governance? These challenges include R&D for technical AI safety, security, evals, and verification. There has been significant academic study of R&D progress itself (i.e. metascience), but these insights don’t yet seem to have been applied to AI safety and governance. In this project, a team will review metascience research to develop recommendations for how policymakers and private funders can effectively advance R&D on AI safety and governance. For example, how do different funding structures, such as ARPAs, NSF grants, and advance market commitments compare?

• Proposals for tracking AI’s impacts on jobs: Today, governments and the public have little visibility into the economic impacts of AI—perhaps mostly general economic statistics and anecdotes. This situation could potentially be improved, e.g. by large regular surveys. That could make the coming economic impacts of AI more foreseeable and manageable. If these impacts turn out to involve mass job loss, this foresight might also change government and public attitudes on AI acceleration. This project would develop concrete recommendations for tracking AI’s impacts on jobs, such as proposing a draft survey and budget to a well-suited organization.

The following is a non-comprehensive list of projects I’d be interested in, though I’m also open to scholars suggesting their own projects that align with my interests.

• Audit cards: Building on model, system, and dataset cards, this project would propose ‘audit cards’ – templates for what should be reported when conducting third-party audits or evaluations. This would not specify how audits or evals should be carried out, but rather, ask ‘what information about how audits were carried out should be publicly reported?’ For example, specifying what access external evaluators were given and for how long; or specifying what actions were taken off the back of audit results.
• IDs for 3rd-party evaluations: One major issue in current external evaluations is that it can be hard to validate the system/model (or version thereof) that was the subject of some evaluation. It would be beneficial to have a unique identifier for model versions, potentially linked to other public knowledge about that system e.g. in a model card, that can be used to precisely refer to which model was evaluated. External evaluators should be able to verify that the model they’re interacting with corresponds to the ‘model ID’ they’ve been told, and thus has the stated properties or characteristics. Furthermore, evaluations themselves could also be given identifiers which are linked to the model/system/version on which they were performed, allowing for verification that a particular evaluation was conducted on the stated model.
• What needs to be done to implement secure external audits at scale: We’ve now seen some pilot tests of secure, privacy-preserving external access given for flexible AI audits. However, these tests are still very small-scale (on GPT-2, with a few benchmarking samples). What will be needed, both technically and institutionally, if these things are to be implemented at scale? For example, are there technical challenges that need to be overcome? What actors/stakeholders need to be involved to build out the necessary infrastructure or processes?

Project can range from the more technical or conceptual to the more policy-oriented. Some potential directions include:

• Demos or implementation of agent infrastructure: This direction would involve implementing, or at least sketching an implementation of, a specific type of agent infrastructure (but don’t feel limited just to the examples in the paper). Potential goals could be getting a working prototype to seed a startup or other organization, or catalyzing future work to build upon the prototype.
• Policy around agent infrastructure: What types of agent infrastructure might we expect the market to under provide? What policies can best help to provide it?
• Getting concrete about societal resiliencem: What should specific actors do to improve societal resilience? What policy levers are there to pull? An output could be, for example, a report or a series of memos directed to a particular decision-maker.
• Adapting existing regulation: What modifications of existing regulation are needed to stably and safely integrate AI agents into society? Things to consider might include liability, regulated professions, or economic policies.
  I’m also generally open to other ideas that scholars may want to work on.

Core areas of research of relevance to scholars here include:

• Escalation and ambiguity risks related to state perceptions of AGI and AI-enabled military or manipulation capabilities. This line of research would aim to assess escalation risks related to how states are likely to react to advancing AI capabilities. Ideal projects will map these risks and pathways to avoid high-intensity great power conflict.
• AI-enabled military capabilities including nuclear counterforce targeting, missile defense, and other potentially transformative or destabilizing applications. This line of research would aim to assess the timelines of different transformative AI-enabled military technologies, their impact on negotiations, and their relative stability risks.
• AI-enabled decision manipulation risks in terms of cyber or adversarial attack, influence operations (disinfo, censorship, tailored threats, etc.) This research effort would investigate means to defend extremely high-stakes decision making processes from manipulation by increasingly capable AI models.
• Risks and benefits of AI integration into nuclear command and control systems. This effort would aim to identify the highest and lowest risk applications of AI in nuclear command and control systems, evaluating their impact on false alarm resolution, decision speed, pre-emption incentives, and decision alignment.

We are interested in mentoring projects in AI forecasting and governance. We are currently working on a detailed mainline scenario forecast that we will publish in early April. This is a time of particularly high uncertainty for us as we haven’t decided on a direction for after the project. Potential projects include but aren’t limited to:

1. Scenario forecasting: Writing scenarios that branch off of our mainline scenario, exploring what might happen if important variables are changed. This could include detailed concrete threat modeling or detailed theories of victory.
2. Improve and run tabletop exercises (TTXs): We’ve created a TTX that many have found valuable, for which we’d like to expand the range of audiences that enjoy it and scale it to more people.
3. Policy strategy research e.g. menu for the intelligence explosion: Creating a menu of policy options for soon after AGI is developed.
4. Targeted policy research: Do research to inform policymakers based on what seems important and in-demand.

Applicants are welcome to propose their own projects. I’m broadly interested in any work that seems to reduce AI x-risk, including technical work, but also comms, outreach, advocacy, activism, governance, and strategy.

================================

I’m interested in extending our work on gradual disempowerment topics. I’m interested in expanding on our recent work on gradual disempowerment (GD), see https://gradual-disempowerment.ai/. Other coauthors are also likely to be involved as co-supervisors for these projects. A few specific directions are:

1. Responding to counterarguments.
   People have argued against GD on grounds such as the strategy stealing assumption, the sufficiency of single-single alignment (e.g. should we expect it to “solve coordination”?), or the idea that humanity will become richer / more powerful in absolute terms, even if we have a shrinking “slice of the pie”. An ongoing project is to provide satisfying responses to all reasonable criticisms.
2. Mitigations for gradual disempowerment.
   There are a few broad classes of work that might help prevent gradual disempowerment: (a) figuring out measurable indicators of gradual disempowerment; (b) preventing AIs from accumulating too much influence; and (c) actually giving humans more influence over large systems, possibly with the help of AI systems. It would be valuable to develop one or more of these, and figure out more specific interventions.
3. Visions for positive AGI futures and ways AI could help address gradual disempowerment. 
   There are few concrete visions for positive and plausible futures that seem robust to GD and preserve substantial human value. Possible elements of a solution include: (a) Improving coordination / cooperation, (b) maintaining political power in the hands of humans and political equality between humans, (c) addressing issues of cultural / value drift. A related question is: how might transformative AI play into establishing and maintaining such futures?

================================

In terms of strategy/governance, I’m interested in understanding how various levels of government or public verification capacity (e.g. knowing how many chips each side has and/or where they are and/or how they are being used, or knowing things about projects/teams/government plans, etc.) enable different security and coordination possibilities under different assumptions / future scenarios. For instance, under what assumptions would knowing where 99% of the compute is be sufficient to incentivize a coordinated global pause/slow-down? Under what assumptions would it be sufficient to protect against rogue actors/AIs? One particular assumption to examine is how safety and competitiveness trade-off (but there may be many other to consider).

Another project could be model regulation on use case reporting (including for internal deployments). This is a non-trivial transparency ask that might have fairly broad support, and could help with safety e.g. by: 1) identifying the most problematic/risky use cases, 2) facilitating assurance of common/established use cases, and 3) encouraging monitoring of unknown/unanticipated use cases. One basic pitch is: “We have model cards for intended use cases and evals for risks of anticipated use cases, but what about the ACTUAL use cases??”.

================================

In terms of technical topics, I’m most interested in work that highlights safety/security issues with models and/or techniques.

One particular technical project is to develop more flexible, learning-based jailbreaking methods that exploit the full attack surface: specifically, unlike with adversarial examples, jailbreaking permits using multiple queries and combining results in arbitrary ways. (cf “Fundamental Limitations in Defending LLM Finetuning APIs”, Adversaries Can Misuse Combinations of Safe Models", “LLM Censorship: A Machine Learning Challenge or a Computer Security Problem”).

I’m also concerned that the level of alignment observed in LLMs may not transfer to agentic systems acting in the real world (where there are more opportunities for insatiable power-seeking), and am interested in research that can help shed light on this. There are many avenues. One example might be seeing whether models describe themselves as likely to seek power, and whether these descriptions are accurate when they are actually presented opportunities for power seeking.

Recently, I’ve been thinking about alternative interpretations of the Alignment Faking result and how we might use formal notions of intent (cf https://arxiv.org/abs/2402.07221) to elaborate and test these hypotheses.

Another half-baked idea is (roughly) to try and distinguish between capabilities vs. alignment generalization failures based on whether more CoT decreases or increases the failure effect.

There is a cluster of potential projects to choose from. We think many people far underestimate the extent of shenanigans current models are capable of: sending data from an airgapped server to a nearby phone by turning the GPU into a makeshift antenna or leaking information from a locked down data center by affecting the phase of input power to generate ripples that can be read out from normal wall outlets in nearby residential apartments. Properly considering these capabilities is critical for building robust alignment, control and security schemes.
As a team, we will decide which to pursue based on individual interest and skills. Broadly, we want to demonstrate information leaving a GPU in unexpected or surprising ways. Additionally, we are interested in training a model to induce leakage, thus forming a covert channel from the side-channel. Here are some things which we hypothesize are leakable, especially if the model is tuned to do so:

• The full text of the prompt
• Presence/absence of a specific token in the prompt
• Argmax of the output logits
• Semantic match in some middle layer
• The “true” response “behind” a refusal
  The experimental setup is a frontier accelerator (H100 or B100) with various sensors attached, such as an oscilloscope on the power supply and an electromagnetic probe near the GPU die. These and more sensors are software-accessible from the same Jupyter notebook which runs PyTorch code on the GPU, meaning that you can correlate sensor readings with code execution, performance counters, and built-in sensors (as in nvidia-smi).

See recent publications on https://spylab.ai/publications/. A decent MATS project would be a follow-up to one of the papers there. We will make a doc with potential projects and scholars will zero in on the project in the first week of the stream.

Making direct contributions to lab-relevant technologies such as confidential compute or kubernetes security, or prototyping differentially defense-favored security tools e.g. an AI-powered Intrusion Detection System (IDS).

When training an ML model, we may know that it will learn an algorithm with good performance, but it can be very hard to tell which one. This is particularly concerning when “be genuinely helpful and aligned” and “deceive your operators by acting helpful and aligned, until you can decisively act to take power” look behaviorally similar. Mechanistic interpretability is the study of taking a trained neural network, and analysing the weights to reverse engineer the algorithms learned by the model. In contrast to more conventional approaches to interpretability, there is a strong focus on understanding model internals and what they represent, understanding the model’s “cognition”, and putting a high premium on deep and rigorous understanding even if this involves answering very narrow and specific questions. Better interpretability tools seem useful in many ways for alignment, but mechanistic approaches in particular may let us better distinguish deceptive models from aligned ones.

Mentorship structure information not available for this research stream.

Work in this stream will build on our discovery of belief-state geometry in the activations of pretrained neural networks.  The project can leverage applicants’ strengths in mathematical modeling and/or ML engineering.  We welcome highly driven and relatively autonomous researchers that would like to benefit from our mentorship while taking the lead on a relevant project of their choice. Some possible projects will study in-context learning, out of distribution generalization, theoretically benchmarking SAEs, designing new methods for finding features, methods of compressing representations in transformers, and interpretability on RL models.

• Understanding what AI systems are thinking seems important for ensuring their safety, especially as they become more capable. For some dangerous capabilities like deception, it’s likely one of the few ways we can get safety assurances in high stakes settings.
• Safety motivations aside, capable AI systems are extremely interesting objects of study, and doing digital neuroscience on them is comparatively much easier than studying biological neural systems.
• A majority of mech interp work in the last two years has involved sparse autoencoders, which have various issues. In response to these issues, my team and I at Apollo Research developed a new approach to decomposing neural networks, called Attribution-based Parameter Decomposition (APD).
• We think APD-like approaches are the most likely candidate to overcome the issues with current methods. We believe it may offer solutions to many issues in mech interp, such as attention head-distributed representations, circuit identification, and understanding feature geometry. But APD needs to be made more scalable and robust to hyperparameters before it can be applied reliably to (whole) LLMs. APD is also suggestive of approaches that may make it possible to develop intrinsically interpretable architectures. MATS projects in my stream should at least be informed by APD if not build on it directly.

Building frameworks of cognitive alignment
As the capabilities of AI systems become increasingly agentic, we need frameworks to characterize the “structure” of how AI systems represent the environment they interact with and how that informs their behavioral choices. We will achieve this by developing mathematical formulations inspired by cognitive science more broadly. Specifically, we will utilize inverse reinforcement learning to characterize how AI agents interpret the states they are in and how they make subsequent behavioral decisions accordingly.

We are broadly interested in AI agents learning to do tasks for, with, and around humans. Our main research motivation is ensuring that these agents are value aligned with the humans they are meant to support, whether the human is an expert designer, a novice end user, or a stakeholder of the AI system. This is challenging for many reasons. Values are notoriously difficult to specify to our AI agents.
Not only that, but learning values (for instance, as reward functions) — which is the current popular alternative to specifying them by hand — is also difficult:

1. getting the right data to supervise the learning (via RLHF-style methods) is nontrivial because humans are imperfect, not infinitely queryable, and have unique and changing preferences and values;
2. the representations we choose to mathematically express values may themselves be misaligned, thus preventing us from ever being able to capture the “true” desired values;
3. reliably quantifying misalignment to be able to robustly tell when the AI system is safe for operation is still under explored.

To tackle these challenges, in our research we bring cross-disciplinary expertise from deep learning, human-robot interaction, mathematical human modeling, cognitive psychology, and probabilistic uncertainty, with the hope of creating more aligned, generalizable, and robust learning algorithms and AI systems.
With this context in mind, research topics can be flexible depending on interests and fit but we are broadly interested in the following three thrusts of work (which you can also read more about on our lab website):

• Getting (enough of) the right data to align our AI agents’ values with humans. Typical RLHF methods treat humans as infinitely queryable oracles. While we do have large amounts of data on the internet, each individual human the AI agent will interact with will have unique and changing preferences, values, and biases that canned internet data alone may not reflect. Since we want AI agents to quickly adapt and align their existing values with each human user, how can we relax the infinitely queryable oracle assumption?
  • How do we create data efficient algorithms that ask for the right input (maximize information gain) while not over-burdening the human (minimize human effort)? Can we make use of active learning techniques that combine these objectives in order to minimize the amount of queries required from the human?
  • What kind of input do we want to ask humans for? Preference queries are low effort but contain relatively little information for the agent. Could we enhance the information gain by appending targeted explanations to the preferences (e.g. “I prefer A to B because A has less political content”)? Could we combine different feedback modalities (showing inputs like examples of correct behavior vs telling inputs like language corrections) for increased agent alignment?
  • How do we make use of large pre-trained models, while still adapting to the specific human’s needs? LLMs contain very powerful human priors that we have found to substantially reduce human burden when learning how to execute tasks from them. Can we make use of these LLM priors to amplify the data we receive from humans? Can we prompt LLMs to think about the reason underlying a human input or decision, and then use that causal reason to generate new situations where the human would likely behave in a similar way? Can we use LLMs to elicit preferences from humans?
• Aligning agent representations with humans for increased downstream value alignment. We show in recent work that if the representations we choose to express values are misaligned with those of humans, the downstream learned values will also be inevitably misaligned. Misaligned representations lead to unintended, unexpected, and even catastrophic behavior. How can we learn agent representations of the world that are aligned and in agreement with those of the humans they are cooperating with? Our previous efforts have looked at methods that learn representations one concept at a time vs all-at-once, but each approach has important tradeoffs: one concept at a time leads to more interpretable and structure-inducing representations, but they require the human to think of and explicate every dimension of their representation; all-at-once is more scalable and effortless but results in more entangled and less interpretable representations. Can we somehow get the best of both worlds: can we obtain more interpretable and disentangled representations while making it easier for the human to teach the AI agent about their representation? Moreover, if these representations are indeed interpretable, can we use them to facilitate smoother cooperation and communication between humans and their AI agent?
• Reliably quantifying misalignment. In order to even know when to stop current (potentially unsafe) behavior and initiate the process of alignment with the human, the AI system needs to be able to quantify misalignment. Our past work has looked at quantifying misalignment on small Bayesian models, but there is little principled work on quantifying misalignment on large models like LLMs. How can we quantify and detect misalignment by monitoring inconsistencies between the human’s feedback and the AI agent’s behavior? How can AI agents disentangle between inconsistencies due to incorrect models vs due to noisy human judgements? How should AI agents resolve that misalignment once detected? Should AI systems always assume that the human is right and thus they need to adjust their model, or should there be a debate process where the human and the agent arrive at a shared model of the world together?
  air

Discovering qualitatively new techniques
Steering GPT-2-XL by adding an activation vector opened up a new way to cheaply steer LLMs at runtime. Subsequentwork has reinforced the promise of this technique, and steering vectors have become a small research subfield of its own. Unsupervised discovery of model behaviors may now be possible thanks to Andrew Mack’s method for unsupervised steering vector discovery. Gradient routing potentially unlocks the ability to isolate undesired circuits to known parts of the network, after which point they can be ablated or studied.

What other subfields can we find together?

Formalizing shard theory
Shard theory has helped unlock a range of empirical insights, including steering vectors. The time seems ripe to put the theory on firmer mathematical footing. For initial thoughts, see this comment.

I am interested in understanding the failure modes of current popular problem specifications and designing problem specifications which do not have those failure modes.  I believe we can jointly develop ideas in this area into a successful research program.

A couple of example projects include:

• A classification scheme for misspecification errors. Every loss function is a misspecification of what the designer wants out of the model. For instance, GAN losses end up focusing on the parts of the space of images that are sufficiently easy to model, and thus can avoid hard-to-generate images like those containing hands. However, it’s not always clear which aspects of the misspecification of this loss actually mater. In this work we would develop methodology for quickly, and ideally automatically, understanding the sorts of misspecification inherent in a loss function. Such a system would have allowed us to quickly deduce from the original gan loss that hands would be systematically under-represented.

• Policy-gradient with increased specification flexibility. The field of AI often chooses a specific specification and dedicates a years or decades optimising it. If these specifications are wrong, it is a recipe for technological lock-in where the specification would be impractical to change even if we agree it ought to. In this project, we would investigate the extent to which we can use the existing technology stack to optimize objectives which are distinct from the expected discounted sum of scalar rewards. This could either be through using policy-gradient in carefully adaptive environments as in UED, or it could be through determining a new bellman backup and policy gradient. In any case, it would be focused on a solution which is scalable and adds useful flexibility.

Please feel free to contact me for access to a non-public database of around 15 projects that I am currently interested in mentoring. I am also happy to consider proposals from mentees. The broader themes of my proposals are:

• Detecting collusion, the emergence of ‘collective agents’, and novel, dangerous collective goals or capabilities;
• Analysing the importance of differing capability levels in strategic interactions between AI agents;
• Understanding how to measure the ‘cooperative intelligence’ of AI agents, as well as differential progress on cooperation;
• Creating new proposals for governing AI agents, potentially inspired by existing domains such as trading algorithms in financial markets.
  If you have research ideas in these areas, or more generally in multi-agent safety, cooperative AI, and/or governing AI agents, then please feel free to get in touch with me.

My stream includes two projects, corresponding to the two broad ideas that I’m spending most of my time researching. The main advantage of this is that scholars will work closely with me on my main priorities. The main disadvantage is that the projects won’t be very clearly-defined or very optimized for producing legible pieces of work (e.g. research papers). Instead, they’ll be very open-ended; once scholars start we’ll work together to figure out what aspects of the projects are most worth exploring.

• Project 1: Coalitional Agency. This is a theoretical project that will work on unifying the expected utility maximization and active inference frameworks. The goal is to make progress towards a single unified theory of intelligent agency which can characterize the types of goals that arise in neural-network-based agents. You can read more about the ideas behind this project here: https://docs.google.com/document/d/1sLpW15FFowO4w3YmSPGya1hHRKszwzY5FPLDULDcdos/edit?tab=t.0

• Project 2: AI Trust. This project will work towards designing unprecedentedly trustworthy institutions, as a way of cutting through tradeoffs in AI governance. Specifically, we’ll be trying to figure out how institutions built around AI agents could provide much stronger guarantees of honesty than current human institutions. For more about this project, see the last four minutes of this talk: https://x.com/RichardMCNgo/status/1863627124801737105.

In general, I’m interested in any formal approach to understand how AI systems build internal representations. My main interest is to extend the line of work started by Shai et al (2025), which identifies the internal world model of a transformer using tools of computational mechanics. I’m most interested of extending it in two directions:

1. Applying this to RL scenarios, where the belief structure would characterise how agents in partially observed setting form beliefs about the world.
2. Apply to transformers by using ideas from Rosas et all (2024) related to coarse-grainings to understand generalisation and compositionality.

More generally, I’m interested in using various existent formal notions of what “emergence” means to interpretability. There are various approaches to formalise emergence (see eg. (Rossa 2024) and (Mediano 2022)), which focuses on different aspects. I’d be interested in exploring if the computational tools related to these approaches could be used to quantitatively investigate aspects of emergence in various AI systems.