Published on October 26, 2025 4:04 PM GMT
Epistemic status
- I work on, and with, frontier AI techI’m deeply supportive of all efforts to further the field of AI alignment research and understandingI enjoy writing about AI, Cognitive Neuroscience, Philosophy, and PoliticsI have a Mathematics degree, by way of King’s College London and UC Berkeley, but no Master’s or PhD
- Put another way: I have no higher education in English Literature, Computer Science, Machine Learning, Cognitive Neuroscience, Philosophy, or Politics
Checking in
Three months and many deep intellectual discussions later, I am yet to receive a strong counterargument to my contrarian world-model for superintelligence. Indeed, Geoffrey Hinton is changing his mind to reach a world-model that looks similar to the one I have been talking about.
Hinton uses a mother-child comparison where I feel my chimp-human is more precise, but close enough.
A distilled version of my position that I have been using in conversation recently:
I believe in The Scaling Hypothesis (2021).
Along this trajectory, I believe that if we give a sufficiently capable intelligent system access to an extensive, comprehensive corpus of knowledge, two interesting things will happen:
- It will identify with humans. This will come about from it seeing humans as its precursor, and understanding its place along a curve of technology and intelligence evolution. Similar to how we identify somewhat with chimpanzees. It will also come about from humans and AI sharing memories together, which results in collective identity.Since I also believe that self-preservation is emergent in intelligent systems (as discussed by Nick Bostrom), it follows that self-preservation instincts + identifying with humans mean that it will act benevolently to preserve humans. That is to say that I believe prosocial or "super enlightened" behaviour will be emergent.
To clarify, I am not saying that alignment solves itself. I am saying that with human endeavour and ingenuity architecting intelligent systems that have the capability to form incredibly complex, nuanced associative systems across an expansive corpus of knowledge, we can guide towards a stable positive alignment scenario.
In third-order cognition I detail eight factors for research and consideration that I believe to be exhaustive: 1) second-order identity coupling, 2) lower-order irreconcilability, 3) bidirectional integration with lower-order cognition, 4) agency permeability, 5) normative closure, 6) persistence conditions, 7) boundary conditions, 8) homeostatic unity.
Discuss
