Stonyford is certainly farther than Elkins Flat from the south. If you do go to Stony, I would camp at an area like Letts Lake, which is farther drive but higher elevation and closer to woods trails than the main staging area.
There is a very good private campground (PiPi Valley) near the trails at Elkins, but it is probably already booked for anytime near July 4th, and you reportedly have to push your bike in and out of the facility. Camping at Elkins Flat itself is available, but primitive (does have an outhouse, but that's it).
FWIW, Elkins Flat/Gold Note trails gets VERY dusty in summer. Fine, powerdery, blinding dust. Some of the better trails are quite good, though.
From the south, it takes just as long to get to Georgetown/Mace Mill (also in the Eldorado Natl. Forest) as it does to Elkins Flat. Elkins is further south, but there is much more twisty road driving. Georgetown is about an hour east of Sacramento, between I-50 and I-80. I think there is more singletrack at Georgetown.