In addition to riding, I used to weight train and do 3 mile runs a few times a week to prepare for harescrambles races. Guess what? It's not enough, I would still get winded in the nasties and very tired towards the end of the race, and revert to survival mode. When I took up mountain biking all that changed. I can now exert myself at a much higher level and for a longer period of time without being winded. I fell so in love with the pain, suffering simplicity, and freedom of mountain biking that I started racing them. It's funny, at this point my bicycles cost more than my motorcycles.
When bicycling for fitness you should build a base first, and that starts with long slow road rides. Your heart rate should be around 60% of theoretical max (max=220 bpm-your age) for the duration. This builds up stamina. Yes, to be precise you need a heart rate monitor. When you go offroad it is harder to regulate your heart rate because you'll be conquering terrain, like hill climbs, boulder fields, log hops, etc. You should still try keep your heart rate under 85% max heart rate. When you do go above 85% of max for short periods, that will build up your anerobic threshhold and contribute to the intensity aspect of your training. A structured approach will help build a fitness base and prevent training injuries.