Emmett Shear on AI Alignment, AGI, and Why Most AI Companies Are Wrong
Emmett Shear explains AI alignment, why AGI requires theory of mind, and why most AI startups are building the wrong products.
In this episode of the Founders in Arms podcast, we sit down with Emmett Shear, founder and CEO of Softmax, former CEO of Twitch, and former interim CEO of OpenAI, to talk about AI alignment, AGI, and how intelligent systems actually learn.
Emmett argues that most discussions about AI alignment are fundamentally confused, and that alignment is not about enforcing rules—but about building systems that can continuously learn, adapt, and understand other agents.
The conversation also explores why AGI and alignment are the same problem, why current AI approaches may be heading in the wrong direction, and what founders are getting wrong about building AI products.
This conversation dives deep into:
What AI alignment actually means
Why alignment is not about rules or morality
Theory of mind as the foundation of intelligence
Continuous learning and why it’s so hard
Why AGI and alignment are the same problem
The risks of “aligned to me” AI systems
Why singleton AI is likely the wrong model
How AI will integrate into human society
Why most AI startups are building the wrong products
Treating AI as systems to train, not tools to prompt
In this episode, we cover:
(00:00) The dangerous truth about alignment
Emmett explains that alignment is not inherently “good.”
The same capability that allows coordination and cooperation can also enable large-scale harm.
True alignment increases both the potential for positive and negative outcomes.
(01:13) What Softmax is building
Softmax is focused on understanding alignment as a general problem across systems—not just AI.
The goal is to create environments where agents can learn how to align with each other over time.
(02:05) Why most people misunderstand alignment
Emmett argues that most discussions of alignment lack a clear definition.
The key question is:
Aligned to what?
Without answering that, the concept of alignment is meaningless.
(03:35) Building training environments for alignment
Instead of building a single “aligned model,” Softmax is building learning environments.
These environments:
Are multiplayer
Allow cooperation and competition
Require agents to model each other
This helps develop theory of mind.
(04:00) Theory of mind as a prerequisite
To align with others, an agent must understand them.
That requires:
Modeling other agents’ goals
Predicting behavior
Inferring intent
Without this, alignment is fragile and accidental.
(04:45) Continuous learning is required
Aligned systems cannot be static.
They must:
Continuously adapt
Learn from new experiences
Update their understanding of the world
The world is non-stationary, so alignment must be ongoing.
(06:30) Multiplayer environments vs static training
Softmax uses open-ended environments where agents:
Compete
Cooperate
Solve evolving problems
This better reflects the real world compared to static datasets.
(07:13) Alignment is always relative
Alignment is not universal.
It depends on:
The agent
Its relationships
What it identifies with
Humans align with family, community, and society in different ways.
AI will face the same problem.
(10:55) Cooperation and competition both matter
Real-world environments are not purely cooperative.
Agents must learn to:
Collaborate when beneficial
Compete when necessary
Training environments need both dynamics.
(12:59) AGI and alignment are the same problem
Emmett argues that building AGI inherently requires solving alignment.
Key missing capabilities today:
Theory of mind
Self-modeling
Learning from experience
Without these, systems remain fragile.
(15:20) Alignment enables both good and evil
A key insight:
The ability to align systems also enables large-scale coordination.
That coordination can be used for:
Positive outcomes
Harmful outcomes
Alignment is a capability, not a guarantee of safety.
(17:14) Why “aligned AI” can be dangerous
When someone says they are building aligned AI, it often means:
Aligned to them.
This creates a concentration of power and risk.
(17:50) Why a single super AI may not dominate
Emmett argues that a “singleton AI” is unlikely.
Instead, we may see:
Many independent AIs
Distributed learning systems
AI societies
This could create more robustness.
(20:44) Alignment must be built through relationships
AI cannot align to “humanity” in the abstract.
It must align to:
Individual humans
Direct interactions
Real relationships
This mirrors how humans develop alignment.
(21:54) Why current AI approaches may be wrong
Many AI companies focus on:
Bigger models
More compute
Longer training
Emmett compares this to building a bigger jet engine instead of designing a new kind of system.
(37:29) The core problem with continuous learning
Training on your own outputs leads to “mode collapse.”
Systems reinforce their own behavior until they become repetitive and less useful.
Solving this requires:
Better feedback signals
Understanding what “good” means
(39:30) Why emotions are part of intelligence
Humans rely on:
Emotions
Intuition
Subjective signals
These act as training signals for decision-making.
Pure reasoning is not enough.
(43:07) Advice for AI founders
Emmett’s key advice:
Stop treating AI like a magic tool.
Instead:
Treat it as something to train
Build systems that improve over time
Focus on creating value, not saving labor
(45:10) Why most AI startups are wrong
Many startups focus on:
Making AI easier
Reducing effort
Automating tasks
But the real opportunity is:
Creating entirely new capabilities.
(45:45) The Twitch lesson: people want quality, not ease
At Twitch, making streaming easier didn’t drive growth.
Making it better did.
The same applies to AI:
Users don’t want easy outputs
They want high-quality results
(51:01) The “AI slop” problem
Low-effort AI-generated content is a temporary trend.
These products will either:
Evolve into higher-quality tools
Or disappear
Key Takeaways for Founders
Alignment is not a fixed goal
It is an ongoing process of adapting to changing environments and relationships.
AGI requires self-awareness
Systems must understand themselves and others to become truly intelligent.
Continuous learning is the core challenge
Training models on static data is not enough for real intelligence.
Alignment increases both risk and potential
The same capability enables both cooperation and large-scale harm.
AI should be trained, not just prompted
The biggest opportunities come from building systems that improve over time.
Focus on value, not automation
The most important AI products will create new capabilities—not just reduce costs.
About the Guest
About Emmett Shear
Emmett Shear is the founder and CEO of Softmax, an AI research company focused on alignment and learning systems.
He previously co-founded Twitch, which was acquired by Amazon, and later served as interim CEO of OpenAI.
Listen to Founders in Arms
Founders in Arms is a podcast for ambitious builders—covering startups, AI, marketplaces, leadership, and the realities of building enduring companies.
🎙 Subscribe to Founders in Arms on your favorite platform
💬 Join the conversation at TribeChat.com
🚀 Discover more insights from founders and operators shaping the future of technology
