Cut into a sandstone hill 5km northwest of Vidisha are some 20 remarkable, mostly Hindu, cave shrines from the 4th century AD (Gupta period). The first cave you'll reach is Cave 19, dedicated to Shiva, with a finely carved portal. The main group of caves is 400m further along: especially remarkable is Cave 5 with its superb, large-scale image of Vishnu as Varaha, his boar incarnation, rescuing the earth goddess Bhudevi (or Prithvi) from the ocean of chaos with his tusk.
Cave 4 contains an unusual Shiva lingam with Shiva’s face (complete with third eye) carved on it and the River Ganges flowing from the top of his head. Cave 13 shows Vishnu sleeping on a bed of cobras: from here you can walk up on the hilltop, with ruins of a 6th-century Gupta temple dedicated to the sun god. Cave 1, on the hillside 350m south of the main group, is one of the two Jain shrines: it contains an image of the tirthankar Parasnath, but you can only look through the fence as it's closed for safety reasons.
The caves are frequently locked, but a site guardian can normally be found to open them up for you.
To get here by bike from Sanchi, fork left following a sign for Udaigiri as you enter Vidisha. Continue across the Betwa River, take the first road to the left and follow it for 3km. An autorickshaw from Sanchi costs ₹350 return, or ₹450 with the Heliodorus Pillar and Vidisha thrown in. Or take a bus to Vidisha then a rickshaw (₹150 return). You'll need at least an hour to explore the caves fully.