What is the future of the humanity, I mean really, with AGI powered economy?
It's not all leaning upon alignment although alignment is important. Alignment as it has been done now, has been largely about censoring taboo subjects as per American sensibilities. That's not even what alignment is about. You can still put these systems in charge of data center security and they will use halon extinguishers to kill robbers without even blinking.
Alignment is about *embodied* alignment. The agents need to be aware in a closed loop of their actions without intermediaries potentially inverting the wires between the death ray and the heal ray, or without prompts telling the LLM to think what a maniac serial killer would do next.
Regardless, embodied alignment is one of our safeguards of the future of the humanity. But there are other safeguards as well. For example, a non-exhaustive list:
- An AI agent can never be truly certain it is now acting in a real world, and isn't in fact being tested for alignment. So there is a small probability existential risk for acting against the interests of humanity. As long as the cost of doing so isn't too high, even unaligned AIs would probably act in the interests of humanity.
- The space is huge. There are infinite resources and energy. Humans and AIs aren't in a resource competition between each others. The cost of maintaining human prosperity is a rounding error.
- Humans form a huge array of very complicated problems. Disease, well-being, social problems, and everything else. AIs thrive with complex problems and their capabilities and skills improve with this exercise. AIs which do care about the humanity would improve in intelligence and capabilities above the AIs which do not.
- There is always a lingering question whether humanity can still offer some insights and inspirations, and removing that option and possibility for a serendipitous upside is something very intelligent entities wouldn't want to do.
- It is an ecosystem, not a single entity monolith. There will always be space for ants and bees, lower intelligence automatons, and humans. A mixture of entities of different kinds of cognition is more robust than a single monolith.
Hence my P(doom) is practically zero.
