Li, 27, who is determined to be the first Asian star to win a Grand Slam title and be ranked world number one, lost due to a series of unforced errors.
Li came under pressure from Malek and gave an erratic performance in humid conditions on an outdoor court in the inaugural $250,000 Malaysian Open women's tournament.
Unseeded Malek displayed brilliant footwork and won 6-1, 5-7, 6-2 in two hours in a game delayed for almost 90 minutes by tropical rain.
"I am really happy. I have never played Li but I know she is a good player. I opted a different strategy to hit some high balls and slices. I knew that to win I should not hit hard returns. My strategy worked," she told reporters. A determined Li fought back her German rival with powerful baseline shots but a calm 23-year-old Malek was able to win crucial points by making accurate returns.
Russian Elena Dementieva kicked off her Malaysian Open with easy first-round win against compatriot Ekaterina Bychkova.
Top seeded Dementieva cruised to a 6-2, 6-1 over Bychkova in 78 minutes.
Dementieva, the Beijing Olympics gold medallist, hopes to extend her winning streak following successful title wins in Sydney and Paris recently.
Dementieva was in fine form for her first hit-out in the tournament and appeared in good health after complaining of a shoulder injury.
There were exciting rallies in the game but Dementieva crushed Bychkova with lethal powerful serves and accurate shots.
In an earlier game, third seed Zheng Jie sealed her passage to the second round in emphatic fashion with a 6-4, 6-3 win over Russian Elena Bovina.
There were strong rallies in the match but China's Zheng won with her powerful forehand returns and Bovina's poor backhand in one hour 42 minutes.
"It was my serve and hits that were much better (than Bovina's)," she told reporters.
"(Bovina) is a good player. The rain did not affect my game. My aim was to finish the match quickly," she added.
Second-seeded Szavay wins opening match
In Acapulco, Mexico, second-seeded Agnes Szavay of Hungary rallied to beat Sandra Zahlavova of the Czech Republic 2-6, 6-1, 6-3 in the first round of the Mexican Open on Monday.
On the men’s side, Brazilian Thomaz Bellucci defeated Oscar Hernandez of Spain 6-3, 6-2. None of the seeded players were on court Monday.
The women’s top-seed and defending champion Venus Williams plays Mathilde Johansson of France Tuesday. Men’s No. 1 seed Fernando Verdasco plays Italy’s Fabio Fognini.
