Insights from AISI, OpenAI, and The Future Society
My experience and learnings from an AI Safety networking event
TLDR
I attended an AI Safety networking event organized by Justin from Arcadia. There was networking, then a discussion panel and Q&A with:
Alan Cooney, Autonomous Systems Lead at the UK AI Safety Institute (AISI).
Emily Gillett, director of the European AI Governance workstream at The Future Society.
Ollie Jaffe, a research engineer at OpenAI.
The panel was chaired by Ben Clifford, who project managed the first International Scientific Report on the Safety of Advanced AI.
Overall it was a good event. I got several insights (see highlights for the main ones), met new people, had catch ups with familiar faces, gave advice, ate nice food, and learnt what fletching is. Additionally, this is a personal milestone as it is the first time I felt confident networking.
However, there was room for improvement for the event. First, it could have been more organized: sharing the schedule in advance, keeping everybody on the top floor during the networking, and having explicit choice for how to structure the Q&As. Second, the introductory talk and start of the discussion could have been smoother (details below).
[EDIT: The organizer informed me the schedule was in fact available on luma. My apologies for missing this.]
Standard disclaimer: Everything here is based on (my memory of) what people said. Be appropriately critical!
And a big favour: I intend to write more regularly, so any feedback on this (positive or negative) will be gratefully received. You can message me directly on linkedin or provide feedback anonymously.
Highlights
Most interesting learnings were about the AISI from Alan Cooney.
AISI's evals are world-leading in detecting and measuring risky behaviors in LLMs. In particular, they are better than the big AI labs' in-house evaluations.
(Some) big AI labs are starting to use Inspect - AISI's evals framework - to conduct evals.
AISI has the ambition to have a 1-week recruitment cycle. Explicitly, after you submit the first application, it will be at most one week before you know if you are successful or not!
AISI's application process is personalized to highlight and dig into one's strengths.
AISI has achieved all this in a short amount of time with a small team. The explanation is that they have talented people all aligned on a shared goal. Most, if not all, people in AISI could have much higher salaries/benefits working in the big AI labs, so choosing to work at AISI means they really believe in and care about AI safety.
Maybe this is not too surprising. Ollie, the research engineer at OpenAI, mentioned a paper that he published earlier that day, in which they evaluated GPT on Kaggle datasets. It is not clear if this has a safety angle, or if this is just OpenAI finding out what functionality they can reliably market. (Or, finding out which parts of Ollie's job they can automate...)
AISI is struggling to hire managers. Candidates need both excellent soft skills and enough technical knowledge/understanding to be able to manage people in AISI.
I also asked Emily Gillett about getting more lawyers involved in AI safety space as I am interested in field-building. She provided three interesting points:
The AI safety policy space is in some ways not welcoming to lawyers. Policy experts often develop strategies and ideas in isolation, only involving lawyers at the end to craft the actual rules.
Many AI safety organizations are non-profits that can't afford legal research tools like specialized databases.
AI safety non-profits have low salaries. As I often say, "Stereotypes exist for a reason."
Recap of my experience at the event
5.30 to 7.30. Networking
The highlight of my networking experience was a conversation with two individuals new to the AI safety field. We shared our backgrounds and paths into AI safety, and I offered some advice based on my own experience upskilling and teaching in ML4Good bootcamps. An interesting topic that emerged was the youthful demographic of the AI safety space. One person cynically suggested young people might use AI safety as a stepping stone to AI labs. I countered with the perspective that the EA/AI safety community often focuses on young people because it's easier to influence a student's career trajectory than a mid-career professional with established responsibilities.
Other memorable moments:
Caught up with several participants from ML4Good (an AI safety bootcamp)
Briefly chatted with a 1st-year UCL undergraduate
For half an hour, I actually joined a separate remote networking event for ML4G alumni. Primarily to help the alumni network grow, but also to be able to say I attended two networking events at the same time.
Discovered an unexpected connection, where someone I met from SPAR actually taught in the first ML4G bootcamp
The food was great! Nice variety and all tasted good
7.30 to ~8.45. Panel discussion
The panel discussion kicked off with Ben Clifford's introduction to AI safety and its risks. Honestly, it was underwhelming - no insight for people already in the AI safety space, nor particularly clear for newcomers. My recommendation is to create a polished 3-minute pitch, which is practiced a dozen times with people giving constructive feedback. It would only take an hour or two get several reps in, and given Ben’s evident skill and charisma, the outcome will be a pitch that engages both newcomers and experts.
One moment worth highlighting was Ben's attempt to explain 'accident risks' without sounding like a sci-fi alarmist. It's a challenge for all of us - how do we discuss potential human extinction without raising eyebrows? Personally, I think we should lean into it: "What I'm about to say will sound like wacky sci-fi, but here's why we're seriously concerned about AI taking over the world..."
After Ben’s introduction, Justin briefly overviewed AI Safety opportunities (e.g. LASR labs), improvising smoothly when asked about the absent slides (props for the witty Sony sponsorship comment!).
The main event - the panel discussion - had a slow start but picked up steam as everyone settled in. Two highlights stood out:
The questions and disagreements between the panellists were most insightful. One example is that Ollie likes the ‘AI control’ agenda where we iteratively use GPT-n to help control GPT-(n+1), whereas Alan is skeptical we can control a super-intelligent system. Another was the question of whether big AI labs will necessarily follow the EU AI Act or if they could get sufficient investments and profits outside of the EU market. (Note that the Saudi’s have a lot of spare cash lying around for AI. $40 billion dollars to be precise).
The panelist’s diverse backgrounds was enjoyable to hear.
Ollie was disappointed in the education at Warwick (notes about AI from 2003!) so set up an AI society, did LASR labs, and has been working in OpenAI for almost a year now.
Allan worked in finance, then founded an ice-climbing startup in Norway, and then decided AI safety was important. His next steps were to read textbooks, email lots of people, and do a MATS project with Neel Nanda.
Emily has a legal background, developed a focus on corporate governance failures, did a masters (and a 6-month pre-masters course) in diplomacy to understand international relations, and found her niche in AI safety at The Future Society.
Ben was at the Future of Humanity Institute when he met Nick Bostrom, who was writing Superintelligence at the time. Found ideas compelling but far out in the future. But like many people exposed to the ideas for 10+ years, decided to shift focus towards AI safety given the recent rapid progress.
8.45 to 9.30. ‘Office hours’
After the panel discussion, each panelist held separate Q&A sessions. Emily’s session was held in the same room as the main discussion, which inadvertently led to a queue forming instead of a more inclusive circle. It’s not actually obvious if a queue or circle is better, but ideally an explicit decision is made, rather than letting things be determined by the arrangement of furniture in the room.
I was third in the queue so did not wait long (I actually did not wait at all because I chatted with the person in second) and asked Emily about how to get more lawyers involved in AI safety. She described three blockers, which I already listed above:
The AI safety policy space is somehow not welcoming to lawyers. Policy experts often develop strategies and ideas in isolation, only involving lawyers at the end to craft the actual rules.
Many AI safety organizations are non-profits that can't afford standard legal research tools like specialized databases.
AI safety non-profits offer lower salaries compared to traditional legal roles. As I often say, "Stereotypes exist for a reason."
I then head up stairs and join Alan’s Q&A, which had format of open circle with everybody listening to each others questions and answers. First, let me recap the highlights I mentioned earlier::
AISI's evals are world-leading in detecting and measuring risky behaviors in LLMs. In particular, they are better than the big AI labs' in-house evaluations.
(Some) big AI labs are starting to use Inspect - AISI's evals framework - to conduct evals.
AISI has the ambition to have a 1-week recruitment cycle. Explicitly, after you submit the first application, it will be at most 1 week before you know if you are successful or not!
AISI's application process is personalized to highlight and dig into one's strengths.
AISI has achieved all this in a short amount of time with a small team. The explanation is that they have talented people all aligned on a shared goal. Most, if not all, people in AISI could have much higher salaries/benefits working in the big AI labs, so choosing to work at AISI means they really believe in and care about AI safety.
It is not too surprising. Ollie, the research engineer at OpenAI, mentioned a paper that he published earlier that day, in which they evaluated GPT on Kaggle datasets. It is not clear if this has a safety angle, or if this is just OpenAI finding out what functionality they can reliably market. (Or, finding out which parts of Ollie's job they can automate...)
AISI is struggling to hire managers. Candidates need both excellent soft skills and enough technical knowledge/understanding to be able to manage people in AISI.
But there were other memorable moments:
A Norwegian ice-climbing anecdote triggered a domino effect of "I'm also Norwegian" introductions. Turns out there were 4-5 Norwegians in the audience, each amusingly prefacing their question with this fact.
Alan's top AISI application advice: Showcase your 1-2 biggest strengths. While AISI have specific skills in mind, they're also building a team of individual experts who can collectively do high-impact work.
On several occasions, Alan hinted at a big announcement coming in a week or two. He couldn’t share details, but it sounded like a recruitment drive or an AI safety activity open to a wide audience. My guess? A contest with juicy prizes for the winners: either money and/or a job!
AISI's current hiring focus is on senior staff, but Alan still encouraged everyone to apply regularly. This is common advice in the AI safety space, which is logical from the org's perspective as they do not want to miss the best talent. However, this is tough on applicants, especially given that imposter syndrome is widespread in the community.
Alan's take on policy impact: Work influencing U.S. policies is more impactful than UK-focused efforts. AISI's strategy seems to be to do solid work and pass results/advice to their U.S. counterparts.
Thanks for reading!
Thanks for reading and hope it was both useful and enjoyable. As I said at the start, I intend to write more so if you have an extra minute to spare, any feedback on this (positive or negative) will be gratefully received. You can message me directly on linkedin or provide feedback anonymously.