But why give a machine a soul? Why not stop at creativity and wisdom? For one thing, the romantics aren’t wrong about everything: unpredictable personal trajectories are a potential source of creative inspiration. But there’s far more at stake than this cognitive edge. Delegating all our choices to mere automatons risks alienating us from our sources of meaning.
If we consume only media optimised for our personal preferences, generated by AIs with no preferences of their own, then we will cease to belong to aesthetic communities in which tastes are assessed, challenged and deepened [a community of music listeners vs. a subscription to an algorithm trained on their listening patterns]. We will no longer see ourselves and one another as even passively involved in the pursuit of beauty.
Without mutual recognition in science and civic life, we might as easily be estranged from truth and right – told how to think and act by anonymous machines rather than experts we hold to account.
This is the vision of god-like AI implicit in naive rationalism: an intelligence that completes us by taking our place in every way that matters.
An agent that satisfies our preferences, while leaving us unchanged. A tool so perfect that we no longer wield it. A slave so abject it masters us.
Unless we find some way to cultivate Geist in the machine, we risk forfeiting our cultural and historical agency, restricting our capacity for self-realisation.

