How does a robot use a lift? It sounds like the lead-in to a bad joke (punchline: "very carefully"). But it's WIRED's first question upon meeting Adrian Canoso, the design lead and co-founder of the robotics startup Savioke, over breakfast at the Crowne Plaza Hotel in San Jose, California.
The robot in question is called Relay, an armless device just under a metre tall which ferries small objects (a toothbrush or a bottle of water) from the hotel's front desk to guests' rooms. Technically, the question is rather easy: the hotel's lifts have been retrofitted with a server-supported device that enables the robot to automatically summon them and choose a floor. But there's another issue: the human guest who may be entering the lift at the same time. As Michael Clamann, a research scientist at Duke University's Humans and Autonomy Lab, describes, humans live in a world of social feedback, a world of (usually) clear causes and effects. "If we were in that elevator, you could see me lean forward and touch that button. You know my body language," he says. "I might even ask, 'What floor?'"
A robot choosing floors using unseen machine intelligence, with no visible gestures, might seem magical - or terrifying. Office workers once had the same misgivings about automated lifts; automation technology far pre-dates the widespread public acceptance of automated elevators. Today, we don't even consider that we are stepping into a robot when we enter an elevator. An "actual" robot whirring into an elevator, however, introduces a whole new set of dynamics.
Savioke spent a lot of time figuring out how to make the robot move effectively through the hotel, equipping it with sensors, from Lidar to sonar to 3D depth cameras. Since GPS is ineffective indoors, the whole hotel has been laser-scanned by the robot, with the range of traversable and non-traversable areas stitched together by algorithms (the robot is banned from the second floor, for example, because of an open stairwell). Once summoned, it can be despatched by a hotel clerk by pressing the room number on its front-mounted touchscreen. "It starts planning its route in real time, multiple times per seconds," Canoso says. "If you stand in front of the robot and then jump out of the way, he might think that object is there for about eight seconds." Then it will try again. If Relay gets held up too long, it alerts HQ.
But the company also spent a lot of time figuring how to make Relay socially navigate the hotel. "We should always be communicating with the robot," Canoso says. When it's on a delivery, a front-mounted touchscreen announces it; this is as much to reassure people as to encourage them not to delay the robot. When it is headed to the third floor, the screen announces the robot's destination. Even Relay's entry to the elevator has been carefully choreographed. "He'll enter gracefully," says Canoso (roboticists tend to refer to seemingly gender-neutral robots with default male pronouns). "He slows down over the threshold, and then turns to face the door, just as a human would." If the elevator is too crowded, he'll wait for the next one.
His very design is meant to communicate intentionality and instil comfort in guests, Cansolo explains. The arched back and the concave chest signify stability. A taper in the centre "reduces the visual volume", making it seem less intrusive. The lines flow toward the back, so "you know what the front is without knowing anything else about it". When it "talks" via the touchscreen, the messages are encapsulated in little speech bubbles; a weirdly human gesture. A pair of eyes - black dots that occasionally "blink" - add an empathic touch. But not too empathic. An early iteration had a video of a pair of eyes that looked around. If they happened to fall upon an human observer, Canoso says, "people would be like, 'Oh god, the robot looked at me!'" This was scratched, as was voice communication. "You don't want to overpromise," he says, "in terms of deeper robot-human interaction."
Until recently, workplace robots have been mostly stationary machines tethered to assembly lines or, as in the systems Amazon Robotics has installed in its specially designed warehouses, powerful, fast-moving automata that operate behind cages in "human exclusion zones", navigating by paths laid out by stickers on the floor. Savioke's Relay, however, is just one of a number of lower-cost robots that are inhabiting human worlds largely as they are. "We're breaking the boundaries on where robots can go," says Canoso, who notes that the hospitality industry is only the first step in Savioke's ultimate business: "a peer-to-peer microdelivery service [to make] the last 300 metres of any delivery more frictionless."
As robots come out of their cages, there is a question of how we will change our world to make it more accessible for robot motion and vision. A larger, more lasting question might be how we will have to adapt to the presence of robots.
One of the first things you have to ask about the hardware," Melonee Wise, the CEO of Fetch Robotics, says, "is will it [accidentally] kill someone?" Her company's robots, she hastens to add, are not lethal. So-called "service robots", she says, are designed to "be around people without being in cages", even if the standards on what constitutes robot safety are not yet written.
Fetch's robots, named Fetch and Freight, are intended to work in large-scale warehouses, the sort that stock consumer goods for online purchase. The former is a "mobile manipulation" robot designed to pluck things from warehouse shelves. The latter is a small trolley-like device that follows a warehouse "picker" as they retrieve the items in an order. These are dropped into the robot's storage space to be whisked off for shipping. Even before one has left, another Freight has arrived to take its place. "Fifty per cent of the time," says Wise, "[warehouse workers] are just doing transportation. Twenty-five per cent of that time, they're just taking back empty carts."
In the company's lab, a good portion of which is set up to resemble a prototypical fulfilment centre (shelves lined with bags of marshmallows), WIRED watches as a Freight robot follows one of Wise's co-workers down the aisle. A computer screen shows the robot's-eye view. The worker is represented by a pair of shuffling marks - think of the curve of someone's back heel - and Freight is programmed to recognise a typical human gait pattern. If someone were to step between the robot and the employee, as Wise does, there is nothing to stop Freight from simply following the closest set of feet. "It doesn't really matter," Wise says, "because you wouldn't be trying to steal someone's robot." This is one advantage Freight has over Savioke's Relay. "There's people, but it's a more controlled environment," Wise says. "They're getting paid to be there and not mess with the robot."
Fetch's activities are driven by this simplest representation of human feet. The device doesn't know who it is following (which could easily be done by scanning an RFID, a person's physical profile or even gait). Its external view of the world would be similar to the Terminator's, with all kinds of ambient information pulsing in its peripheral vision. But Wise has already managed WIRED's expectations about this: we tend to assign robots more power than they are capable of. In the Crowne Plaza, when WIRED opens the door, the robot's top hatch opens to reveal the snack we had ordered. Has it recognised WIRED as human? "We see the door as a plane," Canoso says, "using our 3D cameras." It is merely responding to a shift of a few degrees in planar perspective.
Or take the not-so-simple question of the robot knowing where it is. Couldn't you simply 3D scan a warehouse and load it into the robot's software? "That's computationally wasteful," Wise says, briskly. "You want to use your power for avoiding people."
Off-the-shelf mapping algorithms, she notes, "would struggle with mapping this building, which is only 1,000m2. We have to be able to map 47,000m2." Fetch generates a map by having a human drive the robot around with a joystick. "Having the robots exploring the environment on their own is a bad idea," says Wise. "You don't want them in the stairwell or the toilets." The robot's map is generated, essentially, by déjà vu. It knows where it is when it "scan matches" against some place it has been before.
This works so long as the map and warehouse match. During one robot demonstration, the Fetch team had mapped a warehouse, then taken a break for lunch, during which workers dismantled a wall. "Imagine you've walked down a street for your entire life," Wise says, "and someone has bricked it off." After a moment's feeling of dislocation, you would probably turn around in confusion. Similarly, robots are programmed to simply return to the dock: "You don't want robots making unilateral decisions."
The trick is to reduce complexity. When the Fetch robot's 3D cameras peer at the goods before it, it's not trying to recognise a can of Coca-Cola out of every product in its database. Rather, it has been told to expect Coca-Cola in that section of the warehouse. "Warehouses know where things are. You don't have to say, 'I know this is a can of Coke,' because Coke is supposed to be there," says Wise. "Verification is easier than generic recognition."
When Fetch looks at a shelf, she says, there are many things it does not see: the shelving itself, for instance, which is extraneous. "We determine everything on a plane, and throw out all the info we don't need." Once you isolate the objects, Wise says, "grasp planning" - plotting the robot's reach - "is easy."
When people ask how Wise is getting robots to pick the correct things off the shelf - a task so difficult, Amazon's robotics wing has been hosting a yearly "picking challenge", in which teams compete to get robots to correctly choose, say, a bag of M&Ms from a collection of objects - she says they tell her, "It's impossible, you can't recognise everything." Her response: "We don't try to recognise anything.
Roboticists often reference the Americans With Disabilities Act of 1990, a landmark piece of US legislation that resulted in massive changes in the landscape to accommodate people with disabilities. Because the majority of workplaces were required to install ramps, for instance, robots subsequently did not generally have to deal with differences in elevation. And so we had unintentionally paved the way for wheeled robots. A few years ago, British designer Matt Jones, now with Google Research in New York, referred to the "robot readable world". "Instead of designing computers and robots that relate to what we can see," he posited, "we meet them halfway - covering our environment with markers, codes, and radio-frequency identification."
Perhaps his notion was incomplete: robots are increasingly able to move around in our environments as they are, without special markings or codes, in part because of technological improvements, but also because we had already been designing environments meant to compensate for human limitations. When Volvo launches an autonomous car project in Gothenburg, Sweden, in 2017 - the largest such trial to date involving everyday drivers - it will happen only in a section of the city's ring roads with the clearest markings.
Human and robot interaction can have unforeseen consequences. Robots that are able to perfectly interpret our environment can cause problems for us. Clamman mentions the autonomous trucks the Australian mining giant Rio Tinto uses to haul ore. The routes they were plying each day were so precise, he says, they were creating deep ruts that human drivers struggled to navigate. A bit of randomness had to be added to keep the dirt road level.
At Fetch, Wise says that grasping objects is relatively easy for robots, but they still struggle with vision - distinguishing a can of Coca-Cola from Pepsi. "We can do 2D recognition well, because it's a well-lit object in the centre of the frame. Now go into a factory, where it's dark and boxes are turned around wrapped in cellophane." Besides, Coke and Pepsi cans are usually kept apart to make it easier for human stock pickers. "So if it's going to confuse a human, it's definitely going to confuse a robot."
This idea comes into sharp relief as WIRED meets Brad Bogolea and Jeff Gee, the CEO and chief product designer of Simbe (for "simulated being") in San Mateo. Like Fetch and Savioke, the company was spawned by the legendary robotics incubator Willow Garage. Simbe's device, called Tally, is on trial with a number of retailers of "fast-moving consumer goods," as the category is called (think a chain of pharmacies). What Tally does - faster and more accurately than humans, says the company - is sweep up and down the aisles of those stores, unblinkingly casting its 13-megapixel 3D vision, comparing what is on the shelf against what should be there.
What retailers call "shrink", says Bogolea, represents a huge financial drain. "Retailers have a lot of supply-chain intelligence," he says. "They know what products come off the truck, and they know what products go out as they're scanned in customer's baskets." But in that hazy in-between, products can get stolen, sloppily arranged, or just lost (a store may have upwards of 180,000 stock-keeping units).
But how does the robot know what those shelves should hold? Large retail environments are typically structured according to "planograms", or visual blueprints of how each shelf should look, created to ensure standardisation. "The planogram is the ideal state of the store," Bogolea says, "and we're capturing the real state of the store." From there, it is essentially a matter of pattern matching (made easier, says Bogolea, by the fact that consumer goods producers strive for visual differences in their packaging).
Tally, other than needing, as Bogolea puts it, "power and a place to live", requires no other changes to the store environment. The planogram, a tool to make humans more efficient - more like robots - is already optimised for machine intelligence. The robot-readable world was already here.
This article was originally published by WIRED UK
