Amazon scraps secret AI recruiting tool that showed bias against women

But by 2015, Amazon realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. (AFP)
Updated 10 October 2018

Amazon scraps secret AI recruiting tool that showed bias against women

  • Automation has been key to Amazon’s e-commerce dominance, be it inside warehouses or driving pricing decisions
  • With the technology returning results almost at random, Amazon shut down the project

SAN FRANCISCO: Amazon.com’s machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.
The team had been building computer programs since 2014 to review job applicants’ resumes with the aim of mechanizing the search for top talent, five people familiar with the effort said.
Automation has been key to Amazon’s e-commerce dominance, be it inside warehouses or driving pricing decisions. The company’s experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars – much like shoppers rate products on Amazon, some of the people said.
“Everyone wanted this holy grail,” one of the people said. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”
But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.
That is because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.
In effect, Amazon’s system taught itself that male candidates were preferable. It penalized resumes that included the word “women’s,” as in “women’s chess club captain.” And it downgraded graduates of two all-women’s colleges, according to people familiar with the matter. They did not specify the names of the schools.
Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.
The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity. Amazon’s recruiters looked at the recommendations generated by the tool when searching for new hires, but never relied solely on those rankings, they said.
Amazon declined to comment on the recruiting engine or its challenges, but the company says it is committed to workplace diversity and equality.
The company’s experiment, which Reuters is first to report, offers a case study in the limitations of machine learning. It also serves as a lesson to the growing list of large companies including Hilton Worldwide Holdings and Goldman Sachs that are looking to automate portions of the hiring process.
Some 55 percent of US human resources managers said artificial intelligence, or AI, would be a regular part of their work within the next five years, according to a 2017 survey by talent software firm CareerBuilder.
Employers have long dreamed of harnessing technology to widen the hiring net and reduce reliance on subjective opinions of human recruiters. But computer scientists such as Nihar Shah, who teaches machine learning at Carnegie Mellon University, say there is still much work to do.
“How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable – that’s still quite far off,” he said.
Amazon’s experiment began at a pivotal moment for the world’s largest online retailer. Machine learning was gaining traction in the technology world, thanks to a surge in low-cost computing power. And Amazon’s Human Resources department was about to embark on a hiring spree: Since June 2015, the company’s global headcount has more than tripled to 575,700 workers, regulatory filings show.
So, it set up a team in Amazon’s Edinburgh engineering hub that grew to around a dozen people. Their goal was to develop AI that could rapidly crawl the web and spot candidates worth recruiting, the people familiar with the matter said.
The group created 500 computer models focused on specific job functions and locations. They taught each to recognize some 50,000 terms that showed up on past candidates’ resumes. The algorithms learned to assign little significance to skills that were common across IT applicants, such as the ability to write various computer codes, the people said.
Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured,” one person said.
Gender bias was not the only issue. Problems with the data that underpinned the models’ judgements meant that unqualified candidates were often recommended for all manner of jobs, the people said.
With the technology returning results almost at random, Amazon shut down the project, they said.
The company managed to salvage some of what it learned from its failed AI experiment. It now uses a “much-watered down version” of the recruiting engine to help with some rudimentary chores, including culling duplicate candidate profiles from databases, one of the people familiar with the project said.
Another said a new team in Edinburgh has been formed to give automated employment screening another try, this time with a focus on diversity.


Alexei Leonov, 1st human to walk in space, dies in Moscow

In this Tuesday, July 20, 2010 file photo, former Russian cosmonaut Alexei Leonov speaks to the media before a reception at the U.S. Ambassador's Spaso House residence in Moscow, Russia. (AP)
Updated 12 October 2019

Alexei Leonov, 1st human to walk in space, dies in Moscow

  • NASA on Friday offered its sympathies to Leonov’s family, saying it was saddened by his death

MOSCOW: Alexei Leonov, the legendary Soviet cosmonaut who became the first human to walk in space 54 years ago — and who nearly didn’t make it back into his space capsule — has died in Moscow at 85.
The Russian space agency Roscosmos made the announcement on its website Friday but gave no cause for his death. Leonov had health issues for several years, according to Russia media.
Showing just how much of a space pioneer Leonov was, NASA broke into its live televised coverage of a spacewalk by two Americans outside the International Space Station to report Leonov’s death.
“A tribute to Leonov as today is a spacewalk,” Mission Control in Houston said.
Leonov — described by the Russian Space Agency as Cosmonaut No. 11 — was an icon both in his country as well as in the US He was such a legend that the late science fiction writer Arthur C. Clarke named a Soviet spaceship after him in his “2010” sequel to “2001: A Space Odyssey.”
Russian President Vladimir Putin on Friday sent his condolences to Leonov’s family, calling him a “true pioneer, a strong and heroic person.”
“Infinitely committed to his vocation, he left a truly legendary mark in the history of space exploration and in the history of our country,” Putin said on the Kremlin’s website.
Leonov was born in 1934 into a large peasant family in western Siberia. Like countless Soviet peasants, his father was arrested and shipped off to Gulag prison camps under Soviet dictator Josef Stalin, but he managed to survive and reunite with his family.
The future cosmonaut had a strong artistic bent and even thought about going to art school before he enrolled in a pilot training course and, later, an aviation college. Leonov did not give up sketching even when he flew into space, and took colored pencils with him on the Apollo-Soyuz flight in 1975 to draw.
That mission was the first one between the Soviet Union and the United States and was carried out at the height of the Cold War. Apollo-Soyuz 19 was a prelude to the international cooperation seen aboard the current International Space Station.
But Leonov staked his place in space history ten years earlier, on March 18, 1965, when he exited his Voskhod 2 space capsule secured by a tether.
“I stepped into that void and I didn’t fall in,” the cosmonaut recalled years later. “I was mesmerized by the stars. They were everywhere — up above, down below, to the left, to the right. I can still hear my breath and my heartbeat in that silence.”
Spacewalking always carries a high risk but Leonov’s pioneering venture was particularly nerve-wracking, according to details of the exploit that only became public decades later.
His spacesuit had inflated so much in the vacuum of space that he could not get back into the spacecraft. He had to open a valve to vent oxygen from his suit to be able to fit through the hatch.
Leonov’s 12-minute spacewalk preceded the first US spacewalk, by Ed White, by less than three months.
Leonov might have become the Soviet Union’s first moonwalker, in fact, had his country’s lunar-landing effort not been canceled in the wake of Apollo 11’s triumphant moon landing by Neil Armstrong and Buzz Aldrin on July 20, 1969.
On his second trip into space ten years later, Leonov commanded the Soviet half of Apollo-Soyuz 19.
The cosmonaut was well known for his humor. Once the US Apollo and Soviet Soyuz capsules docked in orbit around Earth on July, 17, 1975, Leonov and his Russian crewmate, Valeri Kubasov, welcomed the three US astronauts — their Cold War rivals — with canned borscht disguised as Stolichnaya vodka and suggested a toast.
“When we sat at the table, they said: ‘Why, that’s not possible,’” Leonov recalled in 2005. “We insisted, saying that according to our tradition, we must drink before work. That worked, they opened it and drank (the borscht) and were caught by surprise.”
The cosmonaut turned 85 in May. Several days before that, two Russian crewmembers on the International Space Station ventured into open space on a planned spacewalk, carrying Leonov’s picture with them to pay tribute to the space legend. They said “Happy Birthday!” to Leonov before opening the hatch and venturing out.
Leonov’s modern-day successor, Oleg Kononenko, who was one of the two Russians on that spacewalk, told Rossiya-24 television on Friday that Leonov had tuned in to hear their congratulations from space.
“We were going to stop by Alexei Arkhipovich (Leonov) after our return and give him our space souvenirs, but you see it wasn’t meant to be,” Kononenko said.
When his crew returned to earth at the end of June, Leonov was already unwell.
Kononenko spoke fondly of the Soviet space pioneer, saying he was a frequent guest at send-off ceremonies for space crews in Star City and at the cosmodrome in Baikonur, Kazakhstan.
“We had this tradition that he would give cosmonauts pep talks before they board the spacecraft,” Kononenko said. “We all looked forward to that, always thought about it and always wanted Leonov to be the one to send us off into space.”
Messages of condolences poured from around the globe.
NASA on Friday offered its sympathies to Leonov’s family, saying it was saddened by his death.
“His venture into the vacuum of space began the history of extra-vehicular activity that makes today’s Space Station maintenance possible,” NASA said on Twitter.
“One of the finest people I have ever known,” former Canadian astronaut Chris Hadfield tweeted on Friday. “Alexei Arkhipovich Leonov, artist, leader, spacewalker and friend, I salute you.”
Russian space fans were bringing flowers to his monument Friday on the memorial alley in honor of Russia’s cosmonauts in Moscow.
Leonov, who will be buried on Tuesday at a military memorial cemetery outside Moscow, is survived by his wife, a daughter and two grandchildren.