With the win, Berdych ended a streak of eight consecutive losses to Federer. The Czech also had lost 11 matches in a row against top-10 opponents.
"It was almost impossible to get it past the other guy with the serve," Federer said.
Others advancing to the quarterfinals included Andy Roddick and Rafael Nadal, who could meet in the semifinals.
Roddick beat German Benjamin Becker 7-6 (4), 6-3. Nadal defeated fellow Spaniard David Ferrer 7-6 (5), 6-4.
Three-time champion Venus Williams reached the semifinals by defeating No. 6-seeded Agnieszka Radwanska of Poland 6-3, 6-1.
The top-ranked Federer won a succession of spectacular rallies to salvage the second set, but converted only two of 10 break-point chances in the match and committed 62 unforced errors, many on an unreliable backhand.
"I fought as much as I could," Federer said. "My game has issues at the moment. I'm definitely lacking timing. I don't know where that comes from."
Federer held a match point at 6-5 in the second tiebreaker, but Berdych hit a forehand winner. Two points later, Federer sailed a forehand long to give Berdych the win at 12.16 a.m. local time.
"After a match like that, the feeling is great," Berdych said. "I'm really happy the way I finish it."
Federer had problems from the start, committing 18 errors to four by Berdych in the first nine games. Federer lost the final eight points of the opening set, double-faulting on set point.
He was twice two points from defeat in the second set before pulling it out, but his shot making became more erratic again in the final set. He rallied from a break down, but hit five errant groundstrokes in the decisive tiebreaker.
Fans in the packed stadium tried to help. Some shouted encouragement in Spanish, which is not one of the six languages Federer speaks. One man called in English, "I love you, Roger!" Another barked, "Get it together."
But Federer could never quite get over the hump.
"It fuels my desire to go back to the practice courts and come back even stronger," said Federer, who now begins his clay-court season. "I don't like to lose these type of the matches."
The No. 6-seeded Roddick rallied past Becker on the strength of his dominating serve. He dug out of a 1-4, love-40 hole in the first set and won 34 of his final 39 service points, including all six in the tiebreaker.
Roddick said experience helped with the turnaround.
"When things aren't going my way, I'm probably better now," he said. "Six years ago on the court my highs were a lot higher, and the lows were a lot lower. If I would have gotten down early, I don't know if I would have stayed the course."
Roddick next plays No. 33-seeded Nicolas Almagro of Spain, who eliminated No. 27 Thomaz Bellucci of Brazil 6-4, 3-6, 7-6 (3).
Serena Williams watched as her sister beat Radwanska. Top-ranked Serena has been sidelined by a knee injury since winning the Australian Open in January. In her absence, Venus has become the woman to beat as she bids for her fourth Key Biscayne title and her first since 2001.
"When I'm executing and playing my best, it's great," Venus said. "It feels good, and I feel like I'm definitely dictating the points and that I don't give my opponent as many chances to have a say."
The oldest women's quarterfinalist at 29, Venus has won 14 matches in a row, and two more would give her three consecutive tournament titles for the first time since 2002.
In the men's fourth round, American Mardy Fish retired with a sciatic nerve injury trailing Russia's Mikhail Youzhny 6-1, 1-0. The injury occurred when Fish took a tumble in the first set, causing discomfort from his back to his calf.
No. 8 Jo-Wilfried Tsonga of France earned a shot at Nadal on Wednesday by beating No. 12 Juan Carlos Ferrero of Spain 6-2, 6-2. The fourth-seeded Nadal, seeking his first Key Biscayne championship and his first title anywhere in 10 months, punctuated his win over Ferrer with a nifty leg kick-uppercut combination.
The No. 3-seeded Williams hit eight aces and lost only five points on her first serve. She broke five times, including in the pivotal eighth game, when Radwanska had consecutive double-faults and then hit a 62-mph knuckleball serve that Williams pounced on to whack a winner.
Williams' opponent Thursday will be No. 13 Marion Bartoli of France, who beat No. 12 Yanina Wickmayer of Belgium 6-4, 7-5.
