Whaling has been a sticky issue between the two major trading partners, though both governments in the past have vowed not to let it affect ties. Rudd made his comments on the eve of a visit by Japanese Foreign Minister Katsuya Okada.
Rudd said that while Australia preferred to find a diplomatic solution to its standoff with Tokyo over the annual whale cull near Antarctica, it was serious about a threat made two years ago to challenge the hunt in an international court.
"If that fails, then we will initiate court action before the commencement of the whaling season in November 2010. That's the bottom line and we're very clear to the Japanese, that's what we intend to do," he told Australian television.
Environmentalists have accused Rudd of backpedaling on threats of an International Court of Justice whaling challenge to avoid damaging Australia's $58 billion trade ties with Japan, and so-far slow progress on a free trade deal.
"Prime Minister (Rudd) spoke carefully, saying 'only if it cannot be solved through dialogue' ... Solving this through dialogue is the basic line and I do not think we have major differences," Okada told a news conference in Tokyo. "Japan and Australia have very important friendly ties and I would like to discuss this calmly through those ties."
Some legal experts say the cull breaches international laws such as the Antarctic Treaty System. A court challenge would lead to so-called provisional orders for Japan to immediately halt whaling ahead of a full hearing.
Commercial whaling was banned under a 1986 moratorium, but Japan still culls whales for what is says is for research.
Tokyo has lodged a protest with New Zealand's government over a collision last month between an anti-whaling protest boat and a Japanese whaler, which caused the activist vessel to sink.
The hard-line Sea Shepherd Conservation Society's skipper is being held on board a whaling ship and may face charges in Japan after boarding it at sea to lodge a protest on Feb. 15.
Okada begins a two-day visit to Australia on Saturday and is to hold talks with Rudd and Australian Foreign Minister Stephen Smith on whaling, security and stalled free trade pact negotiations with Canberra.
