It was the second major surprise of the tournament after second seed Taufik Hidayat, a former world and Olympic champion, lost his first round match on Wednesday to Japan's Kazushi Yamada.
Chen looked to be fighting his way out of trouble against Zwiebler when he led 20-17 in the second game but the German battled back bravely to win five points in a row and secure a place in Friday's quarterfinals.
"It's one of the biggest wins in my career, that's for sure," the delighted winner told Reuters.
Chen, seeded six and an All England title winner in 2008, looked second best in the first game against the aggressive Zwiebler, world ranked 16.
But at 14-8 to Chen in the second it seemed order would be restored only for the German left-hander to fight his way back and then save three game points before sweeping to victory.
"I think playing in Europe made a difference and also in this hall," Zwiebler told reporters. "He made a few mistakes at 17-20 and as I got closer then he got nervous, I think, and I tried to play cool."
Zwiebler, who will be 27 on finals day on Sunday, now faces fellow giant-killer Yamada who booked his last-eight slot by beating China's Wang Zhengming 21-15, 19-21, 21-16.
While Chen struggled, teammate and Olympic champion Lin Dan, chasing his fifth All England title, suffered no such problems, easing past unseeded Indonesian Dionysius Rumbaka 21-11, 21-17.
Earlier, titleholder and world No.1 Lee Chong Wei of Malaysia survived a stiff second round task to oust seasoned battler Bao Chunlai of China 21-16 21-16.
Bao, a former world No.2, has dropped to eight in the world rankings and just missed out on the seedings but remains a valid threat to anyone at the top level.
Chong Wei held his nerve, however, to book his place in the quarters where he will play Thailand's Boonsak Ponsana, seeded eight.
Fourth seed Peter Gade of Denmark rolled back the years to see off last year's beaten finalist Kenichi Tago of Japan 21-9, 22-20.
Gade, 34, rallied from 16-20 down in the second game to take six points in succession and secure his last-eight slot where he will face China's fifth-seeded Chen Long.
Gade, All England champion as far back as 1999, told reporters: "Last year Kenichi reached the final and played a fantastic tournament so he's got plenty of confidence here.
"To win in two games against him was quite good. I was down 16-20 but I was able to step up a gear and put pressure on him so of course I'm very happy."
