It was the first time leaders from Hamas and the more secular Fatah movement of Palestinian President Mahmoud Abbas had shared the platform at a large public gathering since Hamas seized the Gaza Strip from Fatah in a 2007 civil war.
Palestinians mark "Naqba Day" on May 15, the day in 1948 when Israel declared statehood after which some 700,000 Arabs fled or were expelled in the war that ensued.
In the West Bank, sirens wailed through the Israeli-occupied territory as residents were asked to mark a minute's silence.
Speaking outside Gaza's UN headquarters, where the march terminated, Hamas official Ismail Radwan said "the right of return is sacred."
Meanwhile, Fatah official Zakaria Al-Agha said a letter from all the Palestinian movements to UN Secretary General Ban Ki-moon had been presented at the UN office.
It asked him to "act as soon as possible to lift the injustice against the Palestinian people."
The Gaza rally, which was organized by the much smaller Islamic Jihad group to commemorate the Naqba's 62nd anniversary, coincided with reports of serious talks between Hamas and Fatah to find ways to resolve their differences.
Top Palestinian businessman Munib Al-Masri who has been heavily involved in recent mediation efforts, expressed cautious optimism, telling Reuters that "the coming days may result in a positive outcome but we should not expect too much".
Masri has been mediating between the two groups' leaders in the Palestinian territories and in exile and has enlisted the support of Arab diplomats to help narrow the differences. Over two years of Egyptian mediation efforts have so far failed.
Masri's efforts have led to a phone discussion between senior Hamas leader Mahmoud Al-Zahar and Fatah official Azzam Al-Ahmed that has been well publicized among Palestinians.
Zahar said if discussions with Fatah were successful the two sides would bring a joint proposal to Egyptian officials who are leading talks, saying that "reconciliation has become an urgent necessity".
Al-Agha said the unity shown at the rally was "the start of a process to achieve reconciliation".
Hamas, which refuses to recognize agreements Fatah signed with Israel, has been locked in a power struggle with formerly dominant Fatah faction since it won a 2006 Palestinian election.
Palestinian residents in the Gaza Strip and the West Bank have said the enmity between Hamas and Fatah following the 2007 Gaza standoff has wrecked the social fabric of Palestinian society and has undermined peacemaking with Israel.
The two groups continue to carry out arrests of activists affiliated with their rivals in Gaza where Hamas rules, and in the West Bank where Fatah holds sway.
