VR's long, weird history

Time, place, even a sense of self can lose meaning in these creations weaved from the gossamer threads of fact found in fiction, the illusional fancies of magic realism, the mind-expanding nature of synthetic hallucination.

But unlike the evanescent nature of dreams, worlds made in virtual reality stick around. They await your return; they know you are there and control all that you see and hear.

Today, just about anyone can, as Timothy Leary once urged, "turn on, boot up, jack in." Virtual reality machines start at less than $100 and are powered by phones, computers and game consoles.

But that wasn’t always the case.

The long, slow evolution of virtual reality was marked by fits and starts, by staggering success and abysmal failure. It was a journey of secretive labs, bold experiments, game consoles and robot dogs. Its conceivers came from all schools of thought. They were writers, actors, philosophers, mathematicians, musicians, tinkerers and scientists.

Some were drawn to the budding technology through a desire to improve movie theaters; others saw its innate ability to help treat the afflicted. It was to some the ultimate display, to others the perfect medium for expression, or a tool for astronauts, or a way to be in two places at once, or a game engine, or a mind expander.

Over the course of more than 50 years, the innovators of virtual reality plugged away at this machinery of the imagination, adapting, adopting, evolving, pushing forward.

There is a direct line that can be drawn from Mort Heilig’s 1958 Sensorama to Palmer Luckey’s 2016 Oculus Rift. It’s a line that passes through four eras of virtual reality, that swims through the think tanks of MIT, Atari, NASA and the University of Southern California. It is through this baton pass of knowledge from mentor to student, seen over and over again across decades of research, that virtual reality finally got a name, finally came into its own and inevitably found a marketplace among gamers, gadget owners and, perhaps one day, everyone else.

This story lightly touches on the deep history of virtual reality and the many people who helped shape it over time. It includes 25 of the top innovators in the field dating back to Mort Heilig, considered by many to be the father of the technology.

While this is by no means meant to be an all-encompassing examination of the people who impacted virtual reality’s development, it offers a snapshot of the sheer variety and creativity of some of those luminaries of virtual science and art.

1966 // Sword of Damocles

As Mort Heilig toiled away in virtual obscurity, working to find funding for the commercialization of his virtual reality machine, Ivan Sutherland was working on his bachelor’s degree in electrical engineering at Carnegie Institute of Technology. Then he moved on to his master’s from Caltech and finally, his PhD from MIT.

While Sutherland would eventually go on to have such a deep impact on computer graphics that he would become known as the father of that field, in 1966 he created the first working virtual reality head-mounted display.

His work on what he called the Ultimate Display started in 1965, when he published a paper on the topic that imagined a room within which a computer could control the existence of matter.

"A chair displayed in such a room would be good enough to sit in," he wrote. "Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked."

With the help of student Bob Sproull, Sutherland began integrating a variety of devices into a single head-mounted display. The end result used a stereoscopic display and a complicated mechanical head-tracking system that allowed users to change their point of view inside wireframe rooms.

To track movement and help alleviate the weight of the headset, the entire thing was attached to a mechanical arm suspended from the lab's ceiling. The somewhat dangerous-looking device had to be strapped to a user’s head, earning it the nickname "The Sword of Damocles."

1975 // Videoplace

As Todd Richmond, director of USC’s Mixed Reality Lab, puts it, artists have a way of cutting through the BS. They don’t have to worry about selling something or making it profitable.

Among the early innovators exploring the noncommercial and nonscientific uses of virtual reality was Myron Krueger. As with many of the early innovators, it’s hard to place a single label on Krueger. After earning a PhD in computer science, he went on to explore the notion of interactions within art and with art, chiefly powered by computers and often in the context of augmented or virtual reality.

Most influential among his work is likely Videoplace, which evolved over time starting in 1975. The first iteration of Videoplace was computer-free and had two people interacting through a video screen and projected images. The current version includes 25 different programs and is more of a virtual and artificial reality lab. This early work had a deep impact on both virtual reality and the room-sized virtual reality technology that came to be known as CAVEs.

1978 // LEEP

MIT graduate, inventor and perpetual tinkerer Eric Howlett spent nearly 30 years after college in a number of wildly different jobs, from researcher to engineer to marketing manager to optics consultant. It was that last job that, in 1978, led to his invention of an extreme wide-angle stereoscopic optics system.

The creation of this special optics system helped define the way most people view virtual reality today, providing the wide field of view so important to immersion. "His key insight was that you could achieve a very wide-angle view by pre-distorting images to neutralize the distortion introduced by the viewing lenses," his son Alex Howlett tells Polygon. "This pre-distortion technique, which my father originally invented for the purpose of wide-angle color photography, is now one of the fundamental building blocks of VR technology, no matter which optics are being used."

Scott Fisher, working on the Virtual Interactive Environment Workstation for astronauts at NASA’s Ames Research Center, was one of the first to use Howlett’s Large Expanse, Extra Perspective (or LEEP) system. The system was also used in theme parks in the ’80s. It was a secondhand LEEP system that later led to Palmer Luckey meeting with USC Mixed Reality Lab founder Mark Bolas and landing a part-time job at the lab, which in turn fueled Luckey’s early work on the Oculus Rift. In fact, the Oculus Rift prototype used LEEP lenses.

1979 // Aspen Movie Map

Another innovator who straddled the line between art and science, Michael Naimark’s early notable work in the field of virtual reality was the hypermedia project Aspen Movie Map. The idea, born out of some early work done at MIT’s Architecture Machine Group, involved students filming the hallways of the school with a stop-motion camera.

Lab founder (and future Wired magazine founder) Nicholas Negroponte found money for a much bigger version from the Defense Advanced Research Projects Agency, which needed to create a program that could familiarize soldiers with the terrain of a town they’d never visited before.

The project involved a number of people, with Naimark, then a grad student, overseeing the cinematography and production. A team descended upon Aspen, Colorado, to film the streets of the town, 10 feet at a time. The completed project was viewed in an immersive "media room" and enabled users to interactively direct which way they wanted to go as if they were in the city. It was an early version of what Google Maps Street View does today. Last year, Naimark was Google's first-ever Resident Artist in its new VR division.

While not something as sophisticated as today’s head-mounted displays, it delivered the feeling of putting a person in a real place and letting them control what they saw and where they traveled.

1980 // Telexistence

After graduating from the University of Tokyo with degrees in mathematical engineering and information physics — and spending time in a government position as the director of the biorobotics division — Susumu Tachi was awarded a visiting scientist position at MIT in the late ’70s. In 1980, he proposed the concept of telexistence, which is meant to give a person the realistic sensation of being in another place without actually going there. Telexistence can be created with remote hardware and robot interfaces or through virtual reality.

Tachi went on to lead research on a number of projects that explored this idea and is currently heading up a group researching embodied media. The Tachi Lab at Keio University studies virtual reality, augmented reality and telexistence.

1982 // Wired glove

In 1982, Thomas Zimmerman filed a patent for an optical flex sensor that could be used to, among other things, detect finger bending in a glove. His initial idea was to use the creation as an interface for music, allowing people to essentially play air guitar to deliver the sound of a real guitar.

At the time, Zimmerman was working at the Atari Research Center in California with a host of other big names in VR, including Jaron Lanier. When Atari started to cave in on itself in 1984, Lanier and Zimmerman left to found VPL Research, a company specializing in virtual reality tools. At VPL, Zimmerman created the Data Glove, the technology of which Nintendo and others later used.

1982 // Atari Lab

Alan Kay started his graduate work at the University of Utah College of Engineering, where he worked with, among other people, Ivan Sutherland. In the early ’70s, Kay joined Xerox Corporation’s Palo Alto Research Center (PARC) — famed for its massive impact on modern technology, including everything from laser printers to the rise of personal computers — where he helped create the programming language Smalltalk and established his name as a gifted researcher and scientist.

In 1982, Atari brought on Kay to build his own lab and fill it with people who could help think of what computers would be used for in five to 10 years. That group, plucked mainly from the ArcMac Lab at MIT, would go on to contain some of the greatest influencers in virtual reality. When Atari closed the lab, quite a few left for NASA while others created their own startups, and almost all of them had a meaningful impact on the way virtual reality is created and viewed today.

1987 // Virtual reality

Computer scientist, author and musician Jaron Lanier spent the early ’80s as a game developer, creating the experimental games Moondust for the Commodore 64 and Alien Garden for the Atari 800, before joining Atari’s research lab in 1983.

Lanier left the lab in 1984 to found VPL Research with Tom Zimmerman. While there, Lanier continued work on a visual programming language and focused on commercializing some of the hardware technology that got its start at Atari, including Zimmerman’s data glove, early data suits and a head-mounted display called the EyePhone. After famed mind-expanding psychologist Timothy Leary became fascinated with computers and virtual reality as a non-psychedelic-drug way of expanding one’s consciousness, Leary began to work with some of the group at VPL.

Lanier’s research included work on multi-person VR simulators and the creation of avatars. He is also often credited with coining or popularizing the term "virtual reality." Lanier currently works at Microsoft Research.

1989 // Crystal River Engineering

Working with spatial sound innovator Beth Wenzel at the NASA Ames Research Center, Scott Foster was on the cutting edge of sound in virtual reality. Foster founded Crystal River Engineering in 1989 after receiving a contract from Scott Fisher at NASA to help create the audio for his VIEW. Foster's work on that and other projects helped to define the importance of 3D sound in virtual reality environments.

Crystal River Engineering’s innovations eventually started showing up in commercial products. In 1996, the company merged with Aureal Semiconductor, which used the technology to create A3D and Vortex sound cards. The company went bankrupt fighting off a patent infringement case brought by Creative Labs, which later bought up all of Aureal’s (and Crystal River Engineering’s) intellectual properties and technology.

1990 // Interior Body series

Canadian artist Char Davies became interested in computer graphics and imaging in the mid-’80s. In 1988, she joined 3D animation software company Softimage as vice president of virtual research. During her time there, Davies began exploring ways to adapt the software to become home to her art. Her first such project of note was the Interior Body Series in 1990, which used 3D digital images exhibited in lightboxes to show that computer graphics could be, and sometimes should be, painterly.

Her 1993 Osmose experience dropped virtual reality users into a landscape that had to be navigated by breathing and balance. In 1998, Davies premiered Ephemere, built onto the concepts of Osmose to layer in interactive gaze and exploration of the interior body. Davies’ push from the 2D realm of traditional art into the immersive 3D virtual worlds of her new art helped to both extend the use of virtual reality in new ways and broaden the impact of the medium by exposing a new and diverse group to it.

1992 // VR movie Angels

Conceived while Nicole Stenger was a research fellow at MIT in the late ’80s and early ’90s, Angels was the first immersive virtual reality movie. The movie had participants use a head-mounted display and VPL data glove to explore a variety of virtual worlds populated by angels. By interacting with the angels' hearts, participants could experience scenes that represented bliss, loss and fusion. Stenger went on to create a number of VR movies, most notably two other entries in a trilogy that started with Angels and ended with 2007’s Dynasty, which has users travel through time to meet their ancestors. After the trilogy, she went on to create The Isle That Was a Book and The Wish.

Her writing on the topic of art and virtual reality has also been deeply impactful, most notably "Mind is a Leaking Rainbow," in which she discusses virtual reality as a sort of state of grace in which all sensory output can be "pulsating in one harmony." Stenger is currently working to transfer her works to the Oculus Rift and then plans to test those creations on other VR headsets.

1995 // CAVE

After graduating from the Universidad Metropolitana in Venezuela and earning a master’s from the University of Illinois at Chicago, Carolina Cruz-Neira began work on her PhD dissertation at Chicago under computer graphics pioneer Thomas DeFanti. Cruz-Neira worked with a team of graduate students to design the first Cave Automatic Virtual Environment in 1995. The CAVE is a virtual reality room that projects interactive images onto its walls. It gets its name from Plato’s Republic and the Allegory of the Cave, which ponders the question of whether most of humanity is viewing reality as merely shadows cast on a wall and not the real objects.

This approach to VR differs from the sort gaining popularity today in that it doesn’t require the user to wear a headset. Instead, the room is built with an array of sensors, projectors and other devices. Cruz-Neira also designed the software used in CAVE. While a different approach to virtual reality than the head-mounted display units currently sold for home use, the room-sized systems remain popular and important for research and as testbeds for a variety of simulations.

1998 // COSMOS

Michitaka Hirose, a professor of human interface and systems engineering at the University of Tokyo, got his start in virtual reality while he was a visiting scholar at the University of California, Berkeley. Over time he began to focus his work on large-scale virtual reality projects. His works include the Scalable Virtual Reality Contents, which focused on using VR in education, and two five-screen immersive environments called CABIN and COSMOS. CABIN and COSMOS were later connected via a gigabit network and used for research in telepresence, haptics and cohabiting virtual reality worlds.

Hirose now works at a cyber interface lab with Tomohiro Tanikawa and Takuji Narumi at the University of Tokyo, focusing on developing user interfaces that combine humans and computers using virtual reality.

2005 // Bravemind

As the director for medical virtual reality at USC’s Institute for Creative Technologies, Albert "Skip" Rizzo researches the use of VR for rehabilitation and psychological resilience. Rizzo’s work on using virtual reality for treatment dates back to the early ’90s, though, and it weathered the "nuclear winter" of VR that saw the technology almost die out. His best-known work is Bravemind, which started in 2003 but wasn’t funded until 2005. The program uses virtual reality vignettes to help treat disorders, as well as software that allows patients to relive traumatic events in virtual reality as a way of opening the door to treatment. Rizzo received the American Psychological Association’s 2010 award for outstanding contributions to the treatment of trauma.

He also spent time working on VR game systems and hardware to help patients with autistic spectrum disorder, post-stroke rehabilitation, traumatic brain injuries and prosthetic use training. He’s currently working to expand Bravemind’s use to other forms of trauma and to help with nonmilitary victims of traumatic events, like last year’s terrorist attack in Paris.

2008 // EVENT Lab

Mel Slater founded his first VR lab in London in 1991. He has been Professor of Virtual Environments at UCL since 1997, and founded the University of Barcelona's Experimental Virtual Environments for Neuroscience and Technology (EVENT) Lab 10 years ago. His focus is on understanding presence in virtual reality with applications in social situations, and has reproduced classic social psychological results such as the Stanley Milgram Obedience experiments in VR. He also studied the bystander effect - how people respond to violent incidents. For the past few years he has been studying body representation, using VR to substitute people's bodies by life-sized virtual ones that move synchronously with real movements.  A 2013 study, for instance, looked at how this can be used to reduce implicit racial bias and racism.

VR journalism creator Nonny de la Peña cites the lab and Slater’s work as one of the reasons she got involved in virtual reality and immersive news coverage.

2009 // MxR

Mark Bolas’ early work in virtual reality started in the late ’80s as he worked on his master’s thesis at NASA under the guidance of Scott Fisher. Bolas went on to co-found Fakespace to provide hardware for labs researching virtual reality. But his greatest impact on VR came just after Scott Fisher, who had taken a job at USC, hired Bolas as an association professor at the university’s School of Cinematic Arts in 2004.

Fisher calls Bolas "brilliant" and says it wasn’t long before he was splitting his time at the media division and working at the Institute for Creative Technologies, which was basically trying to build a functioning holodeck.

In 2009, Bolas founded the Mixed Reality (MxR) Lab and Studio with an initial goal of making head-mounted displays cheaper to build and disrupting the current state of HMDs, says Todd Richmond, the current director of the Mixed Reality Lab and Studio. And over the course of the next seven or so years, that’s exactly what the lab managed to do, churning out creations like the fold-up viewer in 2012, open-sourcing designs for those viewers and helping out folks like Palmer Luckey and Nonny de le Peña.

Richmond says the lab "kind of declared victory" in the pursuit of affordable HMDs about a year ago, but continues to examine other things like haptics, audio in VR and light fields.

De la Peña tells Polygon that Bolas’ willingness to open up the lab to so many people was a big part of the success of VR during this generation. Richmond echoes that sentiment, adding that Bolas’ push to rapidly prototype ideas also led to a steady flow of new VR hardware.

Bolas left USC this summer to work as partner director of program management at Microsoft. He told Polygon in a recent interview that a design groove drives the rhythm of research in virtual reality, and commented on how his new job would impact VR.

"In my mind, virtual reality is simply one genre of how we can use algorithmic power to enable people to fluently connect with what drives them," he said. "I view my work at Microsoft as an evolution of that connection."

2010 // Kinect

While Alex Kipman has been at Microsoft for more than 15 years, he made the most noise in 2008 when he moved from the Windows division to Xbox. It was there that Kipman began his work on Microsoft’s Kinect peripheral. He saw the device as a "more natural way for people to interact with technology," he told Kotaku in 2010. "The Kinect is the start of that journey."

The device, an array of microphones and cameras that turned movement and the spoken word into action inside games and also controlled the Xbox, went on to sell more than 24 million units. In 2013, Microsoft rolled out an upgraded Kinect for the Xbox One, which, at the time, required the Kinect unit.

In January 2015, Microsoft unveiled HoloLens, a self-contained mixed reality headset that could offer a wearer a glimpse at their world turned into a video game, a how-to video, or a tour of the real Mars. Kipman both introduced the new gear on stage during a press event and led the project during its creation at Microsoft. The development edition of the device shipped in March for $3,000.

2012 // Oculus Rift’s big day

While Palmer Luckey was working away on lighter, less expensive, consumer-friendly head-mounted displays at his home and USC’s MxR Lab, id Software’s John Carmack was also tinkering with the tech. Taken by Luckey’s approach to several design and tech problems, Carmack embraced his HMD. Shortly thereafter, id Software announced that an updated version of Doom 3 BFG Edition would be compatible with the device. But the biggest moment for the tech was when Carmack brought the prototype to 2012’s E3 running his software and showed it off to developers and press, giving the concept a massive media push.

The following summer, Carmack joined Oculus VR as the company’s chief technology officer. Months later, Carmack resigned from id Software to focus entirely on Oculus Rift.

2014 // Chief scientist at Oculus

Michael Abrash started his career in the early ’80s, programming games for IBM PC before moving to Windows to work on graphics and assembly code. But it wasn't until 2012 that Abrash really began his work on researching augmented and virtual realities, along with what he termed wearable computing.

In a blog post on Valve’s website in 2012, Abrash explained how he first got interested in the pursuit: Neal Stephenson's science fiction book Snow Crash.

"I picked it up and started reading, decided to buy it, and wound up devouring it overnight," he wrote. "I also started thinking to myself that I had a pretty good idea how about 80 percent of it could work right then, and wanted to implement it as badly as I had ever wanted to do anything with a computer — I had read [science fiction] all my life, and this was a full-on chance to make [science fiction] real."

That was 1994, but it wasn't until 2012, after a stint at id, another at Microsoft and one at RAD Game Tools, that Abrash landed a job at Valve and began looking into wearable computing.

"By 'wearable computing' I mean mobile computing where both computer-generated graphics and the real world are seamlessly overlaid in your view; there is no separate display that you hold in your hands (think Terminator vision)," he wrote. "I’m pretty confident that platform shift will happen a lot sooner than 20 years — almost certainly within 10, but quite likely as little as 3-5, because the key areas — input, processing/power/size, and output — that need to evolve to enable wearable computing are shaping up nicely, although there’s a lot still to be figured out."

Abrash spent nearly three years at Valve researching AR and VR and writing about the work on his Valve blog. In 2014, Abrash left Valve to join Oculus as chief scientist.

2014 // Facebook deal

An avid collector of virtual reality head-mounted displays and the technology that drives them, Palmer Luckey got his big break after tracking down Mark Bolas at USC’s MxR Lab to talk to him about an HMD he picked up at an auction. Bolas ended up offering Luckey a part-time job, affording him a chance to use the technology in the center and spend time with other innovators in virtual reality.

During his time there, Luckey helped Nonny de la Peña's VR project by jury-rigging an HMD for her immersive journalism piece, which was being shown at a film festival.

Luckey regularly posted updates on his work and prototypes to a website forum dedicated to VR enthusiasts, including John Carmack. Carmack asked to see the early device in 2011 and was so impressed by it, he took it to E3 in 2012 to show it off.

Later that year, Luckey started a Kickstarter campaign to fund the creation of Oculus Rift development units. The first development kit hit in early 2013, and the company began shipping a second version of the dev kit in the summer of 2014.

In March 2014, Facebook purchased Oculus for $2 billion. Many VR innovators and developers say that injection of cash into a VR company almost single-handedly powered the sudden growth and business interest in virtual reality software and hardware.

In March 2016, the first consumer units of the Oculus Rift began shipping. The Rift's Touch controllers go on sale in December.

While Luckey has become the sort of de facto face of virtual reality, he has recently dropped from the limelight following a Daily Beast story in which he said that he donated money to Nimble America, an anti-Clinton/pro-Trump group that seeks to influence the presidential election through "shitposting" and billboard-sized memes.

Illustrator: Chris Kindred