Sri Lanka lost Tillakaratne Dilshan for a duck but recovered to 54-1 at the close. Captain Kumar Sangakkara struck seven boundaries for an unbeaten 33, while Tharanga Paranavitana was 10 not out.
Gayle hit 34 boundaries and nine sixes in 437 balls before being bowled by Mendis, who finished with 6-169 from 59 overs. Offspinner Suraj Randiv picked up three wickets.
Gayle fell short of Brian Lara's world record of 400 but joined Donald Bradman and Virender Sehwag of India as the only players to score two triple-hundreds.
It was the first triple-century by a West Indies player away from home, the highest by any player in Tests between the two teams, and the first triple hundred at Galle International Stadium.
West Indies also surpassed its highest total against Sri Lanka: 477-9 declared in St. Lucia in 2003.
Gayle was involved in three century partnerships, 110 with opener Adrian Barath, 196 with Darren Bravo and 167 with Brendan Nash.
Barath and Bravo scored 50 and 58 respectively.
Gayle was more sedate on Tuesday, having hit eight sixes in a brutal attack on the first day.
He survived a scare off the first ball on Tuesday when Sangakkara referred an lbw appeal to the television umpire Asad Rauf, who turned down the appeal.
Another drama unfolded in the second session when Gayle, on 287, was surprised by a rising delivery from seamer Dammika Prasad and fended it to short extra cover where Sangakkara took an easy catch.
But non-striker Nash convinced Gayle to refer the decision to the third umpire, who ruled the batsman not out because television replays showed the bowler had overstepped for a no-ball.
Gayle drove powerfully through long off for a boundary to reach his triple century and knelt on the pitch holding his bat and helmet high in celebration.
But after Mendis trapped Nash lbw with the score on 559, the visitors lost six wickets for 21 runs.
