Trailing by nine points with four minutes remaining, the Bucks constructed a 14-0 run and surged into a 3-2 lead in the best-of-seven first-round series.
Rookie point guard Brandon Jennings led the way with 25 points and John Salmons scored eight of his 19 in the pivotal fourth quarter run for the Bucks.
"This would have to be the best game of the whole year because it's the playoffs," Jennings told reporters. "I had got a little motivation before the game so that made me want to come out and really play. I knew I had to come out aggressive."
Milwaukee had not reached the playoffs since 2006 but after winning three games in a row following two losses at the start of the series, the sixth seeds can eliminate Atlanta with a home win on Friday.
The Hawks have struggled on the road and did not seriously threaten during back-to-back losses in Milwaukee but were expected to return to winning form at home.
Al Horford had 25 points and a game-high 11 rebounds for Atlanta while Marvin Williams had 22 points, but their teammates struggled to contribute and top scorer Joe Johnson fouled out in the fourth with 13 points.
Jamal Crawford, who earlier this week was named the National Basketball Association's best player in a reserve role, managed just 4-for-18 shooting to finish with 11 points.
After a tight first half, Atlanta built a 13-point lead in the third quarter and looked on course for victory before Milwaukee's late charge put the visitors in control of the series.
In Denver, the Nuggets kept their season alive with a 116-102 victory over the Utah Jazz.
Carmelo Anthony recorded 26 points and 11 rebounds as the Nuggets cut their best-of-seven series deficit to 3-2.
Utah point guard Deron Williams had 34 points and 10 assists in the loss, but the Jazz will have another chance to close out the series Friday at home.
