I have done BMX,DH, DS, CC, ST, Trials, and even uphill challenges.
Except for up-hill they ALL require skill in the handling department especialy if you have ever ridden Vultures KNob with all the bridges, switchbacks, rocks and roots. The trails around here require more effort than the down hill races only because you have to do repeatedly for 2 hours with out getting pulled back to the top by a tractor or ski lift. A lot of time can be made up by doubling and jumping on the downhill portions to miss rocks and roots. A few races I have done cross country have man made jumps in them just for spectators. And they do it on very little travel compared to our downhill bikes that are getting full 8" travel. That is why John Tomac was so good at both, he made up huge gaps on the downhill portions of the trail.
In the bang for the buck, downhill is a waste of money. It is fun but for all the more you get to ride during race day, I'd rather free ride with friends or go to the local trails to jump. Dual Slolom is a blast as long as you don't get bumped early.
I will continue to do them all till I am too old to pedal.