Traditionally, Japanese tell ghost stories in the middle of summer, perhaps as a chilling way to take their minds off the heat.

Prime Minister Shinzo Abe was no exception when he invited executives of his ruling Liberal Democratic Party to a dinner at the Prime Minister's Official Residence on July 30.

“Why don't we live here together? I am frightened," Abe is quoted as telling one participant. "I do not feel like living here because there are ghosts.”

Abe and his Cabinet have categorically denied that ghosts appear at the structure, associated with two bloody coup attempts by the military before World War II.

But he has stayed away from his official residence since he returned to office in December, choosing instead to commute to the prime minister's office next to the residence from his private home, 15 minutes away by car.

Ghost stories have swirled about that the property in Tokyo's Nagatacho district is haunted, for example, by spirits of Imperial military personnel.

At the dinner in a Japanese-style room, Abe was responding to Masashi Waki, the chairman of the LDP's Upper House Diet Affairs Committee, who said, “Some people are concerned that you have avoided moving into your official residence.”

A third member at the meeting joined in saying, “Another prime minister also said it is haunted,” sparking an animated conversation about ghosts for some time.

The official residence was originally built as the prime minister's office in 1929.

On May 15, 1932, Prime Minister Tsuyoshi Inukai was assassinated there by a group of naval officers. It was also the site of what is known as the Feb. 26 Incident in 1936, in which a radical Imperial Japanese Army faction tried to overthrow the government.

It is said that bullet holes from the Feb. 26 incident remain, as well as traces of a bonfire at the entrance made by young military personnel who breached the structure.

“I was told that there are many people clad in military uniform in the garden,” said Yasuko Hata, the wife of former Prime Minister Tsutomu Hata, speaking of the former official residence adjacent to the current one. Hata served as prime minister in 1994.

The former prime minister's office building has served as the Prime Minister's Official Residence since 2005.

Yoshihiko Noda, Abe's predecessor as prime minister who lived in the official residence from 2011-2012, said, “I have heard it is haunted.”

In May, a lawmaker of the opposition Democratic Party of Japan submitted an official inquiry asking why Abe has not moved into the building.

The document read: “There are rumors that the official residence is haunted by ghosts related to the Feb. 26 Incident and other affairs. Are they true?”

In response, the Abe Cabinet submitted a written reply saying, it “is not aware” of the rumors.

On a TV program in June, Abe said, “I have heard former Prime Minister Yoshiro Mori saw part of a ghost.” But he denied the ghost stories, saying they are "urban folklore."

The official residence, built in the style of U.S. architect Frank Lloyd Wright (1867-1959), is highly esteemed in terms of architectural value. But the interior is gloomy and silent even during the day, creating an eerie atmosphere for visitors.

“Ghosts likely come out here at night,” said a sixth-grade male elementary school student who toured the building on Aug. 3. “It was gloomy and spooky.”

Abe has not moved in, perhaps because the residence is too spacious and inconvenient to live in.

“I cannot feel comfortable because it is too large,” he has told his aides.

A senior government official said there is no problem with the prime minister living somewhere other than his official residence.

“If the worst happens, he can rush to (the prime minister’s office from his home) by helicopter,” the official said.