I would at least slip the clutch a bit on down shifts, because there is already a load on the gears, then you downshift for even more load. If you upshift, just let off the throttle for a sec and bang it into the next gear. By letting off the throttle for that split second you take the load off the gears. ;)
Just make sure you use the clutch if you ever end up on some pavement though, no clutch-shifts on tarmac will chew your shifter forks/gears to sh&$ in no time. Good luck and have fun riding! :)