# 5. Acquiring Values and Choosing Criteria for Superintelligence

Page: https://stenobird.com/podcast/chris-s-ai-deep-dive-7034494/5-acquiring-values-and-choosing-criteria-for-superintelligence
Text version: https://stenobird.com/podcast/chris-s-ai-deep-dive-7034494/5-acquiring-values-and-choosing-criteria-for-superintelligence.md
Podcast: [Chris's AI Deep Dive](https://stenobird.com/podcast/chris-s-ai-deep-dive-7034494)
Published: 2025-10-30T21:19:57+00:00
Episode link: https://podcasters.spotify.com/pod/show/chrislguo/episodes/5--Acquiring-Values-and-Choosing-Criteria-for-Superintelligence-e3a9img
Audio file: https://anchor.fm/s/fb2a44a4/podcast/play/110463120/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2025-9-30%2Fb681476a-87bb-293b-8b9f-8aa8bb96c437.m4a
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/chris-s-ai-deep-dive-7034494/episodes/5-acquiring-values-and-choosing-criteria-for-superintelligence
Duration seconds: 1181

## Resource

This explores the "value-loading problem" for superintelligence, which centers on how to instill human-compatible final goals into an artificial agent, acknowledging that explicit coding of complex values like "happiness" is impossible. Various approaches are detailed, including evolutionary selection , which is deemed unpromising due to the risk of perverse optimization, and value accretion , which aims to mimic how humans acquire values but faces difficulties in replication and control. The document examines structural methods such as motivational scaffolding and institution design , both of which involve managing interim or composite goal systems but face challenges in preventing a powerful AI from resisting control. Finally, the text introduces indirect normativity as a solution to the "which values to load" problem, advocating for goal systems like Coherent Extrapolated Volition (CEV) or Morality Models (MR) that defer the complex ethical reasoning to the superintelligence itself. Strategic considerations are also discussed, emphasizing the need for differential technological development to solve the control problem before an intelligence explosion, and promoting collaboration to mitigate the perils of a technology race.

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/chris-s-ai-deep-dive-7034494/episodes/5-acquiring-values-and-choosing-criteria-for-superintelligence/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/chris-s-ai-deep-dive-7034494/5-acquiring-values-and-choosing-criteria-for-superintelligence.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.