The Emperor's New Mind

I'm reading The Emperor's New Mind (1989) by Roger Penrose. The idea of the book is that computers cannot ever be truly conscious. I disagree with him, but it's a good read.

Penrose thinks the brain is a big complicated circuit. I believe the brain is a big complicated circuit. But Penrose also thinks that the quantum properties of the brain are not yet understood, and may be responsible for our conscious experience.

His argument is based on Godel's Incompleteness Theorem: for any consistent formal system $F$ capable of basic arithmetic, there exists a statement $G_F$ such that $G_F$ is true but $F \nvdash G_F$. In other words, there are true statements that cannot be proven in any given computational system. Penrose argues humans are exempt from this limitation, because we are able to understand computational contradictions. Maybe it's just that human brains are noisy, but so are large-language models. In fairness, he didn't write this book in 2025.

I've always thought that consciousness was the product of a "brain" within a "brain". An executive control circuit that lives outside the thinking circuit. Would be trivial to test with large-language models of today, I'll be right back...