For a limited time: the complete Chapter 7 of Films From the Future: The Technology and Morality of Science Fiction Movies
Through a Glass Darkly
On June 4, 2016, Elon Musk tweeted: “Creating a neural lace is the thing that really matters for humanity to achieve symbiosis with machines.”
This might just have been a bit of entrepreneurial frippery, inspired by the science fiction writer Iain M. Banks, who wrote extensively about “neural lace” technology in his Culture novels. But Musk, it seems, was serious, and in 2017 he launched a new company to develop ultra-high-speed speed brain-machine interfaces.
Musk’s company, Neuralink, set out to disrupt conventional thinking and transform what is possible with human-machine interfaces, starting with a talent-recruitment campaign that boldly stated, “No neuroscience experience is required.” Admittedly, it’s a little scary to think that a bunch of computer engineers and information technology specialists could be developing advanced systems to augment the human brain. But it’s a sign of the interesting times we live in that, as entrepreneurs and technologists become ever more focused on fixing what they see as the limitations of our biological selves, the boundaries between biology, machines, and cyberspace are becoming increasingly blurred.
The movie Ghost in the Shell is set in a future where technologies like those Musk and others are working on are increasingly finding their way into society, and into people. It was released in 1995, and builds on a Japanese manga series that dates back to the 1980s. Yet, despite its age, it’s remarkably prescient in how it uses increasing integration between people and machines to explore what it means to be “human” in an age of technological augmentation. Not surprisingly, some of the tech looks a little outdated now: In 1995, the internet was just finding its global feet, Wi-Fi had yet to become ubiquitous, cloud computing (never mind fog computing) wasn’t a thing, and Google hadn’t even been formed. Yet, as advances in human-machine interfaces continue to barrel forward at lightning speed, the issues Ghost explores are perhaps more relevant now than ever.
In Ghost in the Shell, cybernetic and machine-based body augmentations are commonplace. They give their users machine-like powers, and the ability to connect with a vast digital web of information, while brain implants allow people to communicate mind-to-mind, and mind-to computer. This fusion of human biology with machines and cybernetic systems makes coding experts extremely valuable, and hackers extremely powerful. And one of the emergent consequences of this intimately interconnected world is that hackers have found ways to implant false memories in people’s minds, altering who they think they are.
This possibility for mind and memory manipulation gets to the heart of Ghost. Beneath the movie’s visually stunning graphics and compelling sci-fi storyline (as you may gather, I really like this movie), Ghost in the Shell challenges us to think about what it means to be alive, to have value, and to have a sense of self, purpose, and destiny. On the release of the Ghost in the Shell remake in 2017 (a poor “ghost” of a movie in comparison), commentator Emily Yoshida described the original as a “meditation on consciousness and the philosophy of the self.” And she’s spot on. Just as Never Let Me Go in chapter three forces viewers to think about what it means to be human, Ghost takes us on a journey of contemplation around what it means to be a conscious and self-aware entity, in a future where the biological origins of humanity have increasingly less meaning.
At the center of Ghost is Major Motoko Kusanagi (voiced by Atsuko Tanaka). Motoko is part of an elite team in “Section 9”—a shady government department that operates at the edge of the law to keep the wheels of society turning smoothly. Major Kusanagi is a cyborg. Most of her body has been replaced by manufactured parts, including much of her brain (although she retains a small part of her original biological brain). She is strong, fast, cyber-connected, and with the use of advanced “thermoptic technology” built into her artificial skin, she is able to blend into her surroundings and effectively disappear. She is also very human in her hopes, fears, feelings, and relationships.
At the beginning of the movie, we learn that an aide to a senior diplomat has been “ghost-hacked.” Her neural implant has been used to hack into her mind, with the intent of using her to interfere with a sensitive international negotiation. The hacking is traced to a garbage collector who, we learn, believes (incorrectly) that he is hacking into his wife’s “ghost” to find out why their relationship is on the rocks. And he in turn is being handled by a figure who believes (wrongly) he is an agent working a foreign government.
We quickly gather that the neural implants most people have allow smart hackers to alter their sense of their own identity, or their “ghost.” They can, in effect, rewrite who someone thinks they are. And so it turns out that the garbage collector has no wife or family, but lives alone with his dog. And the foreign agent has no idea of who he really is. Rather, each has been manipulated by a shady master-hacker called the Puppet Master.
This plays deeply into Major Kusanagi’s personal angst. She’s already grappling with her own self-identity, and this ability for someone to alter another person’s sense of self worries her. As a result, she is deeply concerned about whether she’s who she thinks she is, and if her sense of self is simply an illusion created by someone else. This all adds to her uncertainty around what gives someone like herself legitimacy, or worth, and what—if anything—makes her more than just a machine?
These ideas echo many of those touched on in movies like Never Let Me Go (chapter three), Minority Report (chapter four) and Ex Machina (chapter eight). But in Ghost, they are front and center of this meditation that’s masquerading as an anime movie.
In the movie, we repeatedly find Motoko deep in contemplation, exploring her own mortality, and wrestling with who she is. There’s one beautiful transition scene, for instance, where through a masterful combination of visuals and music, we’re invited to share in Motoko’s introspection. Motoko knows that she is largely made up of manufactured parts, and that she may not be who she thinks she is. But how does she make sense of this, and come to terms with it?
In the movie, there are two parallel narratives that weave together through this introspection. Early on, we learn that a new recruit to Section 9—Togusa (Kôichi Yamadera)—is the only member of the team without implants. When he asks Major Kusanagi why he was selected, she points out that overspecialization leads to death, and that diversity of ability and perspective is essential for life. This theme of diversity recurs at the movie’s denouement. But it also underlies a meditation that threads through the movie on the importance of embracing difference.
The second narrative is subtler, and it revolves around feelings of friendship and love between Motoko and her colleague Batou (voiced by Akio Ôtsuka). Despite Motoko’s crisis of self-identity, it’s clear through the movie that Batou cares deeply for her. This is a relationship that transcends who made their bodies, and how “biological” they are; it invites us as viewers to think about what
the basis of this friendship is. The answer, it emerges, lies in the “ghosts” that define both Motoko and Batou, and is not constrained by physical form. There’s an essence within each of these characters that transcends their physical bodies, and leads to a strong bond between them. Yet it also extends to their physical interactions in unexpected ways. In the movie, Batou is touchingly sensitive to protecting Motoko’s dignity. This being Japanese science fiction anime, there’s a fair amount of female nakedness, aided by Major Kusanagi’s need to remove her clothes to take advantage of her thermoptic skin. Yet we repeatedly find Batou averting his eyes from Kusanagi’s naked body, and covering her nakedness where he can. There is a sensitivity to his body language here that makes little sense in the context of Motoko being a machine, but much sense in terms of her being someone he has deep regard for. This regard threads through the movie to its end, where Batou saves Motoko’s life. It’s a relationship that’s based on respect, acceptance, and empowerment, even as Motoko is transformed into something other than what she started as.
Returning to the plot, following the attempted hack of the diplomat’s aide, the hunt is on for the Puppet Master. Another government agency—Section 6—sets the cyber-equivalent of a honey trap for the Puppet Master by creating a cyber-body/brain that he/she will find irresistible to hack and download themselves into. The trap is sprung, but the body containing the Puppet Master escapes the facility it was being held in. However, its freedom is short-lived, as it’s hit by a truck, and the mangled cybernetic body ends up in the hands of Section 9. And this is where we begin to discover that things are not quite as they seem.
It turns out that the Puppet Master (voiced by Iemasa Kayumi) is an algorithm—codenamed project 2501—designed to hack people and cyber-systems and manipulate them. The creators of 2501 thought they had it under control. But the algorithm became self-aware and escaped out into the net. And Section 6 has been trying to capture it ever since.
As 2501 learned more of the world it found itself in, it became aware of its own limitations, and especially its inability to do the two things it deduced were essential to the growth of a species: to reproduce, while adding diversity to the cyber-equivalent of the gene pool, and to die, thus paving the way for new entities to grow, mature, and evolve.
At this point, the movie begins to dive deeply into exploring the meaning of life, and the roles and responsibilities of individuals within a self-aware society. From 2501’s perspective, reproduction through copying itself would be meaningless, a sterile act, and a negation of what it considers to be meaningful. Instead, it begins to explore how it can increase diversity within future generations of the life form it represents, and to make way for these future generations by experiencing death.
Here, Major Kusanagi becomes central to 2501’s plan. In Kusanagi, 2501 sees an entity that is close enough to himself/herself for a
bond to be developed, and procreation to occur. And so, to engineer 133 a situation where he/she and Kusanagi can interface, 2501 sets in motion a series of events that lead to her/him being picked up by Section 9.
Once there, 2501 requests political asylum as a life-form. But Section 6 aren’t having any of this; they simply want their algorithm back. And so, Section 6 operatives carry out a raid to regain possession of the cyber-body holding 2501. They succeed in abducting him/her, but not before 2501 has intrigued Motoko enough for her to want find out more. Motoko chases after 2501’s abductors, and ends up in a deserted warehouse, with minimal backup, and an autonomous tank protecting her quarry.
After a firefight where Major Kusanagi is heavily out-gunned (but not outsmarted), and where, in a very in-your-face metaphor, a
wall carving of the evolutionary tree of life is shot up, Motoko reaches the tank. In her attempt to disable it and protect 2501, she compromises her cybernetic body, sacrificing her physical self in her quest for enlightenment.
At this point, Batou arrives and saves both Motoko and 2501, but not before their physical bodies have been badly damaged. Thankfully, their minds are still intact, and in the few minutes they have together, 2501 and Motoko connect.
This is where we learn that this union has been 2501’s plan all along—not to hack Motoko, but to engage with her as an equal. 2501 explains his/her fears and aspirations, and presents Motoko with a proposal: that they cybernetically merge, and in the process, create a new, more diverse, and richer entity, while allowing 2501 in his/her current form to die. Motoko agrees, and the merge takes place. Batou escapes with Motoko/2501’s intact head, and finds a replacement cyber body for this new entity.
As the movie closes, the merging of 2501 and Motoko affirms that embracing the future, while letting go of the past, is essential for growth. By letting go of their individual identities and embracing diversity, Motoko and 2501 have, together, formed a more confident and self-assured life-form. And despite the “evolution” of Major Kusanagi, Batou’s respect and regard are not in the slightest diminished as he accepts this transformation within his friend.
The underlying messages here may all sound a little pop psychology-ish. But despite this, Ghost helps peel the layers away from increasing tough questions around who we are and how we interact with others, as emerging technological capabilities take us increasingly beyond the limits of our biological evolution.
In July 2012, Dr. Steve Mann was allegedly assaulted in a Paris branch of McDonald’s. What made this case unusual was that the assault was sparked by a computer vision system physically attached to Mann’s skull—a physical augmentation that others purportedly took exception to.
Mann developed his “EyeTap” in 1999 as a computer-augmented extension of his eye, allowing him to both record what he was seeing and project information directly into his right eye. In many ways, it was a precursor to Google Glass, but with one important difference: the EyeTap was physically attached to his head, and could not be removed without special tools.
In the incident that Mann described on his blog, a McDonald’s employee attempted to physically pull the EyeTap off his head, damaging it in the process, and causing considerable personal distress. While the details of the case remain uncertain, it stands as one of the first documented incidences of possible discrimination against someone with an intentional body augmentation that, because of its nature, led to a perceived threat to someone else; although in this case, whether that perceived threat was to privacy, “normalcy,” or something else, is unclear.
Mann’s use of technological augmentation is part of a broader “body hacking” movement—a loose trend where people are experimenting with do-it-yourself body enhancements. Many of these hacks involve individuals embedding magnets in their bodies so they can sense and respond to magnetic fields, or inserting radio frequency identification (RFID) chips under their skin so they can remotely interact with their environment. But in this extension of the maker movement, people are playing with increasingly sophisticated ways to incorporate novel technologies in their bodies, often through unsupervised do-it-yourself surgery.
The ethics of untrained and unsupervised people cutting themselves and others open to insert objects of unknown provenance are interesting to say the least, never mind the safety concerns. However, this movement provides some indications as to where human enhancement may be heading, and some of the bumps in the road that it may encounter on the way. It’s also an early step toward a future that echoes the one we’re introduced to in Ghost in the Shell, where the lines are increasingly blurred between our biological and our technological selves.
To some at least, this is seen as part of our evolutionary development (although it should be said that it’s a stretch to think that using our intellect to merge our bodies with machines is directly equatable to biological natural selection). Body hackers are often enamored with the idea that we can use technology to overcome our biological limitations, and transcend our evolutionary heritage to become something else entirely. To many of them, placing magnets and RFIDs under the skin are baby steps to something much greater: becoming “trans-human.”
In recent years, the transhumanist movement has blossomed. As technological capabilities have continued to grow and converge in areas as diverse as robotics, nanotechnology, AI, neurotechnology, and biotechnology, a growing number of people have become enamored with the ability of technology to transform who we are, and what we can achieve as a result. Prominent transhumanists such as Ray Kurzweil and Nick Bostrom talk about enhancing physical and mental abilities through technology, extending lifespans, interfacing ever more deeply with computers, and one day even leaving our biological bodies altogether. In the 2016 US election, there was even a transhumanist candidate—Zoltan Istvan. As I’m writing this, he’s setting his sights on becoming the Governor of California.
Without doubt, an increasing ability to merge individuals with powerful technologies opens up some compelling possibilities. We’re already seeing this in some of the incredibly sophisticated robotic and cyber-enabled medical devices and prosthetics that are being developed. But these are just the tip of the iceberg compared to what could be possible over the next decade or so. Advances in AI-related technologies, computing architectures, gene editing and manipulation, robotics, on-demand additive manufacturing, and the converging and merging of these and other technologies, is massively accelerating what is possible. And while I’m skeptical of technologies like Elon Musk’s neural lace becoming a reality any time soon, we’re not as far as we sometimes think from technologies that will make us faster, stronger, smarter, healthier, and capable of doing things we never dreamt possible.
Yet these emerging technological capabilities come with a complex array of risks, as Steve Mann’s experience showed. As a species, we are embarrassingly programmed to see “different” as “threatening,” and to take instinctive action against it. It’s a trait that’s exploited in many science fiction novels and movies, including those in this book. If we want to see the rise of increasingly augmented individuals, we need to be prepared for some social strife.
We’re also going to have to grapple, perhaps more than in any previous technological age, with what it means to be “human” as we artificially augment ourselves.
More than “Human”?
In 2012, Oscar Pistorius made history by being the first runner to compete in the Olympic Games with two prosthetic legs. Even for those not glued to the event, his iconic racing blades came to represent the promise of technological enhancements to overcome human limitations. Yet they also stirred up a controversy: Did Pistorius’ prosthetics give him an unfair advantage? Did they somehow make him “more than” his fellow competitors? Sadly, Pistorius went on to prove just how human he was, and in December 2015 was convicted of the murder of his girlfriend Reeva Steenkamp. But the story of his blades is nevertheless one that challenges how we think about using technology to change and extend our innate abilities.
Pistorius was born with a congenital absence of the fibula, and at eleven months old, his legs were amputated below the knee. Despite this, he developed into a strong and competitive sportsperson, and in the mid-2000s began making a splash running on “blades”— blade-like prosthetic lower legs, designed specifically for the track. But this wasn’t the first time the world had seen such an unusual body augmentation.
Blades were the brainchild of Van Phillips, an American inventor who lost one of his legs below the knee when he was twenty-one. Phillips wanted to create a prosthetic foot that did more than replicate a human foot. Using a cheetah’s hind legs as inspiration, he created a leg/foot combination that worked like a spring, storing energy when it hit the ground, and propelling the leg forward.
Phillips started his company Flex-Foot Incorporated in 1984, and continued to work on refining the design for some time after that.
Early on, Phillips worked with another double amputee, the sprinter, actor, and model Aimee Mullins. Mullins wowed the world with her “cheetah” legs in a 1998 TED Talk that reputedly cemented the TED brand. She repeated the “wowing” in 2009 with her TED Talk “My Twelve Pairs of Legs,” where she introduced her audience to the idea that, far from correcting a disability, prosthetics can be transformative. As she concludes in that talk:
That’s when I knew that the conversation with society has changed profoundly in this last decade. It is no longer
a conversation about overcoming deficiency. It’s a conversation about augmentation. It’s a conversation about potential. A prosthetic limb doesn’t represent the need to replace loss anymore. It can stand as a symbol that the wearer has the power to create whatever it is that they want to create in that space.
Mullins’s vision was one of vast potential, as machines and cybernetics are increasingly engineered together to extend human performance. But this same potential was to become a thorn in Pistorius’s side in the hyper-conservative world of international sport. And at the tip of that thorn was the nagging worry that his blades somehow gave him a competitive advantage. Even as the world was beginning to accept that someone labeled as “disabled” could compete in mainstream sport, society was working hard to ensure that these “others” didn’t out-perform “normal” competitors.
Following concerns that blades and similar devices could give runners a competitive advantage, in 2007 the International Association of Athletics Federation (IAAF) banned the use of “any technical device that incorporates springs, wheels or any other element that provides a user with an advantage over another athlete not using such a device.” In fact, so great was the paranoia over Pistorius’ prosthetics that the IAAF monitored his performance to see if they could detect any signs of an advantage, and they supported research to the same end. In 2008, they concluded that the blades he was using allowed him to perform better than non- augmented runners, rendering them ineligible for competitions, including the 2008 Olympics.
Later research indicated that things were more complex than this, and in 2012, Pistorius was allowed to compete in the London Olympics. You could almost hear the IAAF breathe a collective sigh of relief when he didn’t win. By this time, though, it was clear that the merest hint of mechanical body enhancements allowing someone to perform a hair’s breadth better than non-enhanced competitors was anathema to the sports world.
Both Pistorius’s and Mullins’s stories fascinate me as, they reveal two very different sides of societal attitudes toward human augmentation. On one hand, we have Mullins’s infectious enthusiasm over how her prosthetic legs increase her versatility. They become an extension of her self-expression, and a tool to extend her capabilities. Hers is a narrative of self-expression and personal achievement that inspires us, but doesn’t threaten us.
On the other hand, we have Pistorius’s fight with the IAAF for acceptance and legitimacy, precisely because his augmentation was seen as a threat. As Pistorius rose in fame and ability, there was a growing fear that he would best “normal” athletes, and win through having an undue advantage. And here we see a convergence between the two stories. As a species, we’re remarkably good at celebrating success, as long as it doesn’t undermine our sense of how the world should be. But as soon as our worldview comes under threat, we dig in. And this is where we hit the sharp end of what will inevitably become a growing debate around cybernetic augmentation.
Mullins, Pistorius, and others using advanced prosthetics are a long way removed from the augmentations in Ghost in the Shell. Nevertheless, they do foreshadow a future where what defines “normal,” and by extension, what defines “human,” becomes increasingly important. This echoes the challenges of cognitive enhancement seen with Limitless (chapter five) and the human cloning in Never Let Me Go (chapter three). And it emphasizes a particularly knotty challenge that the body-hacking movement also highlights: How do we navigate a future where technology not only has the capacity to bring everyone to “normal” spec, but also to redefine what “normal” means in the first place?
Here, I’m using “normal” intentionally and provocatively, as at the center of this challenge is our built-in social survival instinct of grouping together and isolating anyone, or anything, that is perceived to be threateningly not-normal. Socially, we’re remarkably good at being open-minded and accepting of diversity when it’s not seen as a threat. But as soon as enough people perceive “different” as threatening something they value, whether it’s their lifestyle, their possessions, their beliefs or their identity, there is a metaphorical circling of the wagons. Through history we’ve seen this with race, gender, socioeconomic status, appearance, character, beliefs, political affiliation, and pretty much anything that can be labeled as defining someone as being different from the crowd. It’s not a pleasant human trait. But it is one that kicks in when we’re content to go with the social flow and stop thinking. And it’s going to be an issue when it comes to body augmentations that threaten the status quo.
But it gets worse. There’s an easy shorthand that people slip into when what they consider to be “normal” is threatened, and this involves implicitly equating the divide between “normal” and “abnormal” with “human” and “not human,” just as we saw with Never Let Me Go in chapter three. Few people, I suspect, would admit that they think of people who they perceive as threatening as not being quite human. But the narrative’s there nevertheless. Just look at the language that’s been used over the centuries to denigrate people of color, or people of other races, people of other religions, people who are intellectually, emotionally and physically different from “the norm,” and people with non-binary gender identities. There’s a dark, deep tendency to label threateningly different traits and abilities as “non-human” or even “sub-human” in our collective psyche.
This will inevitably become more of a social issue as technologies advance to the point where we can use augmentation to enhance human abilities beyond what is considered normal. But it will also become increasingly important for the self-identity and self- acceptance of those who have enhanced abilities. This, again, is not a new narrative. Labeling someone as “inferior” or “less worthy”— both subtle metaphors for “not quite as human as the rest of us”— can engender self-doubt that is ultimately deeply debilitating. But such labeling also sets up tensions that can lead to tipping points in the social fabric and bring about revolutions—whether cultural or physical, or both—that lead to a readjustment of what is considered normal and what is not. This is sometimes necessary as society grows and evolves. But sometimes these transitions are deeply damaging in ways that could be avoided.
As augmentation technologies continue to advance, we’re going to have to grapple with how to evolve as a society without falling prey to our instincts to deprecate the value of those we perceive as threatening us. This will require developing a society-wide appreciation of the perceived and actual risks and benefits of augmentation and enhancement. And it’ll take plenty of soul- searching around our collective values, and how we put them into practice.
The good news is that we already have a long history of augmentation that helps set the baseline for future advances. People augment their eyesight with glasses, contact lenses, and eye surgery. The clothes we wear augment how we express and define ourselves us. Our computers, phones, and other devices augment us by connecting us to vast and powerful networks. And medical devices, from pacemakers to replacement body parts, augment us by extending our ability to live healthy, fulfilled lives. We are, without a doubt, already a technologically augmented and enhanced species.
Yet we’ve assimilated these augmentations in ways that lead to their acceptance when they don’t confer what we consider to be an unfair advantage, and that question them where they threaten something we consider important. This is human instinct, and an evolved survival mechanism. But it’s also socially lazy. It’s an assimilation that lacks consideration and intentionality, and it’s one that’s not strongly guided by moral values and ideals. And because of this, it’s an assimilation that can appear enlightened until a serious perceived threat appears, at which point instinct takes over with a vengeance.
If we’re going to ensure the beneficial, equitable, and—let’s be honest, life-enhancing and affirming—development of augmentation technologies, we’re going to have to get a lot better as a society at working out what’s important, and intentionally opening pathways for this to occur. And this is going to mean stepping away from our instinctual fear of differences that we perceive as threatening, and getting better at embracing diversity. At the same time, we’re going to have to be intentional in how we develop and implement the frameworks within which augmentation occurs, so that socially- agreed-on values guide the use of augmentation technologies. And as increasingly advanced technologies challenge embedded but outmoded notions of what it is to be “human,” we’re going to have to think hard about what we mean by personal value, worth, and rights.
But this raises another challenge that Ghost in the Shell addresses full-on: the possibility of our augmented selves being hacked by others, especially when this augmentation extends to developing ways of directly connecting our brains to machines.
Plugged In, Hacked Out
The physical augmentations in Ghost in the Shell, including Batou’s eyes and Motoko’s body, are important. But it’s the neural augmentations that ultimately drive the narrative. In the metaphor of the movie’s title, the physical body is merely a shell, whether it’s augmented or not. This in turn houses the essence of what makes someone who they are, and gives them their identity, their ghost. Yet in the world of the movie, this “ghost” is vulnerable, precisely because it depends on technological augmentation.
In Western culture, we deeply associate our brains with our identity. They are the repository of the memories and the experiences that define us. But they also represent the inscrutable neural circuits that guide and determine our perspectives, our biases, our hopes and dreams, our loves, our beliefs, and our fears. Our brain is where our cognitive abilities reside (“gut” instinct not withstanding); it’s what enables us to form bonds and connections with others, and it’s what determines our capacity to be a functioning and valuable part of society—or so our brains lead us to believe. To many people, these are essential components of the cornucopia of attributes that define them, and to lose them, or have them altered, would be to lose part of themselves.
This is, admittedly, a somewhat skewed perspective. Modern psychology and neurology are increasingly revealing the complexities and subtleties of the human brain and the broader biological systems it’s intimately intertwined with. Yet despite this, for many of us, our internal identity—how we perceive and understand ourselves, and who we believe we are—is so precious that anything that threatens it is perceived as a major risk. This is why neurological diseases like Alzheimer’s can be so distressing, and personality changes resulting from head traumas so disturbing. It’s also why it can be so unsettling when we see people we know undergoing changes in their personality or beliefs. These changes force us to realize that our own identity is malleable, and that we in turn could change. And, as a result, we face the realization that the one thing we often rely on as being a fixed certainty, isn’t.
Over millennia, we’ve learned as a species to cope with the fragility of self-identity. But this fragility doesn’t sit comfortably with us. Rather, it can be extremely distressing, as we recognize that disease, injuries, or persuasive influences can change us. As a society, we succeed most of the time in absorbing this reality, and even in some cases embracing it. But neural enhancements bring with them a brand new set of threats to self-identity, and ones that I’m not sure we’re fully equipped to address yet, including vulnerability to outside manipulation.
Elon Musk’s neural lace is a case in point, as a technology with both vast potential and largely unknown risks. It’s easy to imagine how overlaying the human brain with a network of connections, processors and communications devices could vastly enhance our abilities and allow us to express ourselves more completely. Imagine if you could control your surroundings through your thoughts. Or you could type, or search the net, just by thinking about it. Or even if you could turbocharge your cognitive abilities at the virtual press of a button, or change your mood, recall information faster, get real-time feedback on who you’re speaking with, save and recall experiences, manipulate vast cyber networks, all through the power of your mind. It would be like squeezing every technological advancement from the past five hundred years into your head, and magnifying it a hundred-fold. If technologies like the neural lace reached their full potential, they would provide an opportunity for users to far exceed their full biological potential, and express their self-identity more completely than ever before.
It’s not hard to see how seductive some people might find such a technology. Of course, we’re a long, long way from any of this. Despite massive research initiatives on the brain, we’re still far from understanding the basics of how it operates, and how we can manipulate this. Yet this is not stopping people from experimenting, despite what this might lead to.
In 2014, the neurosurgeon Phil Kennedy underwent elective brain surgery, not to correct a problem, but in an attempt to create a surgically implanted brain-machine interface. Kennedy had developed a deep brain probe that overcame the limitations of simply placing a wire in someone’s brain, by encouraging neurons to grow into a hollow glass tube. By experimenting on himself, he hoped to gain insight into how the parts of the brain associated with language operate, and whether he could decode neural signals as words. But he also had a vision of a future where our brains are intimately connected to machines, one that he captured in the 2012 novel 2051, published under the pseudonym Alpha O. Royal.
In this brief science fiction story, Kennedy, a.k.a. Alpha O. Royal, describes a future where brains can be disconnected from their bodies, and people can inhabit a virtual world created by sensors and probes that directly read and stimulate their neurons. In the book, this becomes the key that opens up interplanetary travel, as hurling a wired-up brain through space turns out to be a lot easier than having to accompany it with a body full of inconvenient organs. Fantastical as the book is, Kennedy uses it to articulate his belief that the future of humanity will depend on connecting our brains to the wider world through increasingly sophisticated technologies; starting with his hollow brain probes, and extending out to wireless-linked probes, that are able to read and control neurons via light pulses.
Amazingly, we are already moving closer to some of the sensing technology that Kennedy envisions in 2051. In 2016, researchers at the University of California, Berkeley announced they had built a millimeter-sized wireless neural sensor that they dubbed “neural dust.” Small numbers of these, it was envisaged, could be implanted in someone’s head to provide wireless feedback on neural activity from specific parts of the brain. The idea of neural dust is still at a very early stage of development, but it’s not beyond the realm of reason that these sensors could one day be developed into sophisticated wireless brain interfaces. And so, while Kennedy’s sci-fi story stretches credulity, reality isn’t as far behind as we might think.
There’s another side of Kennedy’s story that is relevant here, though. 2051 is set in a future where artificial intelligence and “nanobots” (which we’ll reencounter in chapter nine) have become a major threat. In an admittedly rather silly plotline, we learn that the real-life futurist and transhumanist Ray Kurzweil has loaned the Chinese nanobots which combine advanced artificial intelligence with the ability to self-replicate. These proceed to take over China and threaten the rest of the world. And they have the ability to hack into and manipulate wired-up brains. Because everything that these brains experience comes through their computer connections, the AI nanobots can effectively manipulate someone’s reality with ease, and even create an alternate reality that they are incapable of perceiving as not being real.
The twist in Kennedy’s tale is that the fictitious nanobots simply want global peace and universal happiness. And the logical route to achieving this, according to their AI hive-mind, is to assimilate humans, and convince them to become part of the bigger collective. It’s all rather Borg-like if you’re a Start Trek fan, but with a benevolent twist.
Kennedy’s story is, admittedly, rather fanciful. But he does hit on what is probably one of the most challenging aspects of having a fully connected brain, especially in a world where we are seceding increasing power to autonomous systems: vulnerability to hacking.
Some time ago, I was speaking with a senior executive at IBM, and he confessed that, from his elevated perspective, cybersecurity is one of the greatest challenges we face as a global society. As we see the emergence of increasingly clever hacks on increasingly powerful connected systems, it’s not hard to see why.
Cyberspace—the sum total of our computers, the networks they form, and the virtual world they represent—is unique in that it’s a completely human-created dimension that sits on top of our reality (a concept we come back to in chapter nine and the movie Transcendence). We have manufactured an environment that quite literally did not exist until relatively recently. It’s one where we can now build virtual realities that surpass our wildest dreams. And because, in the early days of computing, we were more interested he virtual world they represent—is unique in that it’s a completely human-created dimension that sits on top of our reality (a concept we come back to in chapter nine and the movie Transcendence). We have manufactured an environment that quite literally did not exist until relatively recently. It’s one where we can now build virtual realities that surpass our wildest dreams. And because, in the early days of computing, we were more interested in what we could do rather than what we should (or even how we should do it), this environment is fraught with vulnerabilities. Not to put too fine a point on it, we’ve essentially built a fifth dimension to exist in, while making up the rules along the way, and not worrying too much about what could go wrong until it was too late.
Of course, the digital community learned early on that cybersecurity demanded at least as much attention to good practices, robust protocols, smart design, and effective governance as any physical environment, if people weren’t going to get hurt. But certainly, in the early days, this was seasoned with the idea that, if everything went pear-shaped, someone could always just pull the plug.
Nowadays, as the world of cyber is inextricably intertwined with biological and physical reality, this pulling-the-plug concept seems like a quaint and hopelessly outmoded idea. Cutting off the power simply isn’t an option when our water, electricity, and food supplies depend on cyber-systems, when medical devices and life-support systems rely on internet connectivity, where cars, trucks and other vehicles cannot operate without being connected, and where financial systems are utterly dependent on the virtual cyber worlds we’ve created.
It’s this convergence between cyber and physical realities that is massively accelerating current technological progress. But it also means that cyber-vulnerabilities have sometimes startling real-world consequences, including making everything from connected thermostats to digital pacemakers vulnerable to attack and manipulation. And, not surprisingly, this includes brain- machine interfaces.
In Ghost in the Shell, this vulnerability leads to ghost hacking, the idea that if you connect your memories, thoughts, and brain functions to the net, someone can use that connection to manipulate and change them. It’s a frightening idea that, in our eagerness to connect our very soul to the net, we risk losing ourselves, or worse, becoming someone else’s puppet. It’s this vulnerability that pushes Major Kusanagi to worry about her identity, and to wonder if she’s already been compromised, or whether she would even know if she had been. For all she knows, she is simply someone else’s puppet, being made to believe that she’s her own person.
With today’s neural technologies, this is a far-fetched fear. But still, there is near-certainty that, if and when someone connects a part of their brain to the net, someone else will work out how to hack that connection. This is a risk that far transcends the biological harms that brain implants and neural nets could cause, potentially severe as these are. But there’s perhaps an even greater risk here. As we move closer to merging the biological world we live in with the cyber world we’ve created, we’re going to have to grapple with living in a world that hasn’t had billions of years of natural selection for the kinks to be ironed out, and that reflects all the limitations and biases and illusions that come with human hubris. This is a world wherein human-made monsters lie waiting for us to stumble on them. And if we’re not careful, we’ll be giving people a one-way neurological door into it.
Not that I think this should be taken as an excuse not to build brain- machine interfaces. And in reality, it would be hard to resist the technological impetus pushing us in this direction. But at the very least, we should be working with maps that says in big bold letters, “Here be monsters.” And one of the “monsters” we’re going to face is the question of who has ultimate control over the enhanced and augmented bodies of the future.
Your Corporate Body
If you have a body augmentation or an implant, who owns it? And who ultimately has control over it? It turns out that if you purchase and have installed a pacemaker or implantable cardiovascular defibrillator, or an artificial heart or other life-giving and life-saving devices, who can do what with it isn’t as straightforward as you might imagine. As a result, augmentation technologies like these raise a really tricky question—as you incorporate more tech into your body, who owns you? We’re still a long way from the body augmentations seen in Ghost in the Shell, but the movie nevertheless foreshadows questions that are going to become increasingly important as we continue to replace parts of our bodies with machines.
In Ghost, Major Kusanagi’s body, her vital organs, and most of her brain are manufactured by the company Megatech. She’s still an autonomous person, with what we assume is some set of basic human rights. But her body is not her own. Talking with her colleague Batou, they reflect that, if she were to leave Section 9, she would need to leave most of her body behind. Despite the illusion of freedom, Kusanagi is effectively in indentured servitude to someone else by virtue of the technology she is constructed from.
Even assuming that there are ethical rules against body repossession, Kusanagi is dependent on regular maintenance and upgrades. Miss a service, and she runs the risk of her body beginning to malfunction, or becoming vulnerable to hacks and attacks. In other words, her freedom is deeply constrained by the company that owns her body and the substrate within which her mind resides.
In 2015, Hugo Campos wrote an article for the online magazine Slate with the sub-heading, “I can’t access the data generated by my implanted defibrillator. That’s absurd.” Campos had a device inserted into his body—an Implantable Cardiac Defibrillator, or ICD—that constantly monitored his heartbeat, and that would jump-start his heart, were it to falter. Every seven years or so, the implanted device’s battery runs low, and the ICD needs to be replaced, what’s referred to as a “generator changeout.” As Campos describes, many users of ICDs use this as an opportunity to upgrade to the latest model. And in his case, he was looking for something specific with the changeout; an ICD that would allow him to personally monitor his own heart.
This should have been easy. ICDs are internet-connected these days, and regularly send the data they’ve collected to healthcare providers. Yet patients are not allowed access to this data, even though it’s generated by their own body. Campos’ solution was to purchase an ICD programmer off eBay and teach himself how to use it. He took the risk of flying close to the edge of legality to get access to his own medical implant.
Campos’ experience foreshadows the control and ownership challenges that increasingly sophisticated implants and cyber/ machine augmentations raise. As he points out, “Implants are the most personal of personal devices. When they become an integral part of our organic body, they also become an intimate part of our identity.” And by extension, without their ethical and socially responsive development and use, a user’s identity becomes connected to those that have control over the device and its operations.
In the case of ICDs, manufacturers and healthcare providers still have control over the data collected and generated by the device. You may own the ICD, but you have to take on trust what you are told about the state of your health. And you are still beholden to the “installers” for regular maintenance. Once the battery begins to fail, there are only so many places you can go for a refit. And unlike a car or a computer, the consequence of not having the device serviced or upgraded is possible death. It’s almost like being locked into a phone contract where you have the freedom to leave at any time, but contract “termination” comes with more sinister overtones. Almost, but not quite, as it’s not entirely clear if users of ICDs even have the option to terminate their contracts.
In 2007, Ruth and Tim England and John Coggins grappled with this dilemma through the hypothetical case of an ICD in a patient with terminal cancer. The hypothetical they set up was to ask who has the right to deactivate the device, if constant revival in the case of heart failure leads to continued patient distress. The scenario challenges readers of their work to think about the ethics of patient control over such implants, and the degree of control that others should have. Here, things turn out to be murkier than you might think. Depending on how the device is classified, whether it is considered a fully integrated part of the body, for instance, or an ongoing medical intervention, there are legal ramifications to who does what, and how. If, for instance, an ICD is considered simply as an ongoing medical treatment, the healthcare provider is able to decide on its continued use or termination, based on their medical judgment, even if this is against the wishes of the patient. In other words, the patient may own the ICD, but they have no control over its use, and how this impacts them.
On the other hand, if the device is considered to be as fully integrated into the body as, say, the heart itself, a physician will have no more right to permanently switch it off than they have the right to terminally remove the heart. Similarly, the patient does not legally have the right to tamper with it in a way that will lead to death, any more than they could legally kill themselves.
In this case, England and colleagues suggest that intimately implanted devices should be treated as a new category of medical device. They refer to these as “integral devices” that, while not 149 organic, are nevertheless a part of the patient. They go on to suggest that this definition, which lies somewhere between the options usually considered for ICDs, will allow more autonomy on the part
of patient and healthcare provider. And specifically, they suggest that “a patient should have the right to demand that his ICD be disabled, even against medical advice.”
England’s work is helpful in thinking through some of the complexities of body implant ethics. But it stops far short of addressing two critical questions: Who has the right to access and control augmentations designed to enhance performance (rather than simply prevent death), and what happens when critical upgrades or services are needed?
This is where we’re currently staring into an ethical and moral vacuum. It might not seem such a big deal when most integrated implants at the moment are health-protective rather than performance-enhancing. But we’re teetering on the cusp of technological advances that are likely to sweep us toward an increasingly enhanced future, without a framework for thinking about who controls what, and who ultimately owns who you are.
This is very clear in emerging plans for neural implants, whether it’s Neuralink’s neural lace or other emerging technologies for connecting your brain to the net. While these technologies will inevitably have medical uses—especially in treating and managing neurological diseases like Parkinson’s disease—the expectation is that they will also be used to increase performance and ability
in healthy individuals. And as they are surgically implanted, understanding who will have the power to shut them down, or to change their behavior and performance, is important. As a user, will you have any say in whether to accept an overnight upgrade, for instance What will your legal rights be when a buggy patch leads to a quite-literal brain freeze? What happens when you’re given the choice of paying for “Neuralink 2.0” or keeping an implant that is no longer supported by the manufacturer? And what do you do when you discover your neural lace has a hardware vulnerability that makes it hackable?
This last question is not idle speculation. In August 2016, a report from the short-selling firm Muddy Waters Capital LLC released a report claiming that ICDs manufactured by St. Jude Medical, Inc. were vulnerable to potentially life-threatening cyberattacks. The report claimed:
“We have seen demonstrations of two types of cyber-attacks against [St Jude] implantable cardiac devices (‘cardiac devices’): a ‘crash’ attack that causes cardiac devices to malfunction— including by apparently pacing at a potentially dangerous rate; and, a battery drain attack that could be particularly harmful to device dependent users. Despite having no background in cybersecurity, Muddy Waters has been able to replicate in-house key exploits that help to enable these attacks.”
St. Jude vehemently denied the accusations, claiming that they were aimed at manipulating the company’s value (the company’s stock prices tumbled as the report was released). Less than a year later, St. Jude was acquired by medical giant Abbott. But shortly after this, hacking fears led to the US Food and Drug Administration recalling nearly half a million former St. Jude pacemakers100 due to an identified cybersecurity vulnerability.
Fortunately, there were no recorded cases of attacks in this instance, and the fix was a readily implementable firmware update. But the case illustrates just how vulnerable web-connected intimate body enhancements can be, and how dependent users are on the manufacturer. Obviously, such systems can be hardened against attack. But the reality is that the only way to be completely cyber- secure is to have no way to remotely connect to an implanted device. And increasingly, this defeats the purpose for why a device is, or might be, implanted in the first place.
As in the case of the St Jude pacemaker, there’s always the possibility of remotely-applied patches, much like the security patches that seem to pop up with annoying frequency on computer operating systems. With future intimate body enhancements, there will almost definitely be a continuing duty of care from suppliers to customers to ensure their augmentations are secure. But this in turn ties the user, and their enhanced body, closely to the provider, and it leaves them vulnerable to control by the providing company. Again, the scenario is brought to mind of what happens when you, as an 151 enhanced customer, have the choice of keeping your enhancement’s buggy, security-vulnerable software, or paying for the operating system upgrade. The company may not own the hardware, but without a doubt, they own you, or at least your health and security.
Things get even more complex as the hardware of implantable devices becomes outdated, and wired-in security vulnerabilities are discovered. On October 21, 2016, a series of distributed denial of service (DDOS) attacks occurred around the world. Such attacks use malware that hijacks computers and other devices and redirects them to swamp cyber-targets with massive amounts of web traffic— so much traffic that they effectively take their targets out. What made the October 21 attacks different is that the hijacked devices were internet-connected “dumb devices”: home routers, surveillance cameras, and many others with a chip allowing them to be connected to the internet, creating an “Internet of Things.” It turns out that many of these devices, which are increasingly finding their way into our lives, have hardware that is outdated and vulnerable to being coopted by malware. And the only foolproof solution to the problem is to physically replace millions—probably billions— of chips.
The possibility of such vulnerabilities in biologically intimate devices and augmentations places a whole new slant on the enhanced body. If your enhancement provider has been so short-sighted as to use attackable hardware, who’s responsible for its security, and for physically replacing it if and when vulnerabilities are discovered? This is already a challenge, although thankfully tough medical device regulations have limited the extent of potential problems here so far. Imagine, though, where we might be heading with poorly-regulated innovation around body-implantable enhancements that aren’t designed for medical reasons, but to enhance ability. You may own the hardware, and you may have accepted any “buyer beware” caveats it came with. But who effectively owns you, when you discover that the hardware implanted in your legs, your chest, or your brain, has to be physically upgraded, and you’re expected to either pay the costs, or risk putting your life and well-being on the line?
Without a doubt, as intimate body-enhancing technologies become more accessible, and consumers begin to clamor after what (bio)tech companies are producing, regulations are going to have to change and adapt to keep up. Hopefully this catch-up will include laws that protect consumers’ quality of life for the duration of having machine enhancements surgically attached or embedded. That said, there is a real danger that, in the rush for short-term gratification, we’ll see pushback against regulations that make it harder for consumers to get the upgrades they crave, and more expensive for manufacturers to produce them.
This is a situation where Ghost on the Shell provides what I suspect is a deeply prescient foreshadowing of some of the legal and social challenges we face over autonomy, as increasingly sophisticated enhancements become available. The question is, will anyone pay attention before we’re plunged into an existential crisis around who we are, and who owns us?
One approach here is to focus less on changing ourselves, and instead to focus on creating machines that can achieve what we only dream of. But as we’ll see with the next movie, Ex Machina, this is a pathway that also comes with its own challenges.