The Akatsuki (Dawn) was launched in May to observe the toxic atmosphere and volcanic surface of the planet, but it is thought that it failed to slow down enough to be pulled in by gravity.
"Unfortunately, it did not attain an orbit," BBC News
quoted Hitoshi Soeno of the space agency, JAXA (Japan Aerospace Exploration Agency) as saying.
"But it appears to be functioning and we may be able to try again when it passes by Venus six years from now."
Plans were for the box-shaped space craft, which is also called the Planet-C Venus Climate Orbiter, to orbit and observe Venus for two years.
reported that it is fitted with two paddle-shaped solar panels and five cameras to help it look through the sulphuric acid clouds and search for signs of lightning and active volcanoes.
Venus has large amounts of carbon dioxide and temperatures reaching as high as 460C (860F).
The probe was to work along with the European Space Agency's Venus Express, which began observing the planet in 2006.
The project has already cost Japan about 24.4 billion yen (£185 million - $293 million).
reported that getting the probe into orbit would have been a big success for Japan, as the Nozomi (hope), which was launched in 1998 to study Mars, had not been successful because of technical glitches.
The country’s Hayabusa probe, which returned to Earth in June, was in space for seven years and became the first to collect asteroid dust.