With a number of
apparent AI breakthroughs in recent weeks, it’s a good time to revisit AI safety (see
previous coverage) - efforts to ensure that AI progress benefits humans and avoids catastrophic consequences. Nick Bostrom’s 2014 book
Superintelligence - which warned of superintelligent artificial agents with wildly different values from us - has played a large role in framing the debate so far.
This week’s
ChinAI podcast sees Jeff Ding interview
Eric Drexler, one of the fathers of nanotechnology, on his paper from earlier this year,
Reframing Superintelligence. Drexler argues that the Bostrom-ian framing of superintelligent agents doesn’t fit what we know about AI progress so far. Drexler says a more likely outcome is “Comprehensive AI Services” (CAIS) - i.e. a collection of superintelligent
tools, rather than
agents. Drexler is optimistic that this will give us more options for solving the “
control problem”, as the tools won’t have goals and values of their own.
Scott Alexander at Slate Star Codex is
broadly positive on Drexler (and makes good metapoints about why he didn’t think of this before) Rohin Shah has
a great synopsis, but thinks CAIS will eventually be overtaken by AI agents.
Gwern’s critique - that “tool AIs want to be agent AIs” - is perhaps the best, despite predating Drexler by two years. Nevertheless, it’s valuable to have a framing of AI risk that’s harder to parody than a
paperclip maximiser - which hopefully means more people will (rightly) take it seriously.