Wesley Sneijder scored the winner in the 68th minute with a header following a corner kick.
Robinho had given Brazil the lead in the 10th, but the Netherlands equalized in the 53rd with an own goal from Felipe Melo.
It was a match of sharp contrasts for Felipe Melo, who set up the opening goal with a brilliant through pass but then was shown a direct red card in the 73rd for stepping on Netherlands winger Arjen Robben, leaving Brazil with 10 men.
The Netherlands reached the semifinals for the first time since losing to Brazil on penalties at the 1998 World Cup in France and will next face either Uruguay or Ghana.
Having won all five of their matches so far, the Netherlands extended its team-record unbeaten streak to 24 games, stretching back to a September 2008 loss to Australia.
At the final whistle, the Dutch players jumped on each other near midfield and lingered to savor the moment — the Netherlands' first win over Brazil since the 1974 World Cup, when Johan Cruyff was the team's star and "total football" reigned.
The Netherlands reached consecutive World Cup finals in 1974 and 1978 but lost both — to Germany and Argentina, respectively — and its only major tournament victory was the 1988 European Championship.
The Dutch will be favored against either Uruguay or Ghana in the semifinals.
Brazil also lost in the quarterfinals four years ago, falling to France 1-0. Former Brazil captain Dunga was hired to coach the team after that loss despite no previous managerial experience and his future is certain to come into question now.
On a warm and pleasant afternoon before a sellout crowd of 42,286 at the Nelson Mandela Bay Stadium, Brazil dominated the first 45 minutes, then fell apart.
Robinho's goal started with a long pass from Felipe Melo through the heart of the Netherlands defense, setting up an easy score for the Santos striker.
Two minutes before he scored, Robinho had a goal waved off for a close offside call on Brazil's first chance. Buzzing like a bee back and forth between the left wing and the center of Brazil's attack, Robinho started another effort in the 31st that nearly became the most spectacular goal of the tournament.
Robinho dribbled through three defenders up the left flank then gave the ball to Luis Fabiano, who flicked it back with his heel to Kaka. The playmaker then sent a searing shot to the top right corner of the goal, but Netherlands goalkeeper Maarten Stekelenburg timed his leap perfectly to deflect the ball away with one hand.
The one-goal lead wasn't enough, though, and Brazil began to unravel when Felipe Melo headed the ball into his own net in a mixup with Brazil goalkeeper Julio Cesar following a cross from Sneijder, who also led Inter Milan to the Champions League title in May.
Sneijder's goal followed a corner kick from Robben. Dirk Kuyt flicked the ball on with a header by the near post and Sneijder found the target from the center of the area.
Five minutes later, Felipe Melo walked off the field with his head hung low.
Brazil had one final chance in the 89th but a free kick from Daniel Alves smacked into a thick Dutch wall.
Felipe Melo missed Brazil's 3-0 victory over Chile but was rushed back into the lineup since fellow midfielders Elano and Ramires both sat out — Elano with a right ankle injury and Ramires due to yellow cards.
Maarten Stekelenburg, Gregory van der Wiel, John Heitinga, Joris Mathijsen, Giovanni van Bronckhorst, Dirk Kuyt, Mark van Bommel, Wesley Sneijder, Nigel de Jong, Arjen Robben, Robin van Persie (Klaas Jan Huntelaar, 85).
Julio Cesar, Maicon, Lucio, Juan, Michel Bastos (Gilberto Melo, 62), Gilberto Silva, Felipe Melo, Daniel Alves, Kaka, Robinho, Luis Fabiano (Nilmar, 77).
