Rattlesnake Trail, from the Kern Plateau (Kennedy Meadows) is a 10-mile intermediate trail leading to Rattlesnake Creek; it's accessible from the San Joaquin Valley side for you. The creek is adjacent to but not in the Golden Trout Wilderness, so the limits/constraints do not apply; standard trout regulations do (I did my homework). Not many do this very thing, so I'd expect you'd have the creek to yourselves. It's been a dry year, so I wouldn't expect it to be as good as times past, but mixed with dirt biking, it's still quite an experience.
A friend & I did this very thing some time ago, staging off of the pavement @ the trail head and riding our dirt bikes down with gear in small backpacks, then carried our limit of them beauties on the way back out. The goldens lose some of their color @ lower elevations (8K' there), but they are still pretty fish - and I might add: tasty too.