A short 3km round-trip hike down the road past Altai Oasis takes you to these small caves by the river where rural farmers used to sleep before market days in Sorata. The caves themselves aren't super impressive but if you continue 100m past them you'll reach a viewpoint over a beautiful canyon. Come in the morning to see squawking parrots flying through the valley.
The path isn't terribly well marked, but just keep left to follow the river and you should be able to find your way.