The '04s have really recalibrated suspensions. The forks are supposed to be more plush since they firmed up the midvalve, allowing softer full, probably explaining your observation. More to your problem, they also changed the triple clamps to better balance the bike, shifting some of the weight off the forks and to the shock. Shock valving was also improved to accomodate this. By putting your '03 triple clamps on the '04 you surely undid some of this engineering.
BTW, I also heard that Jeremy spent ~3 weeks in Austria so some of this '04 goodness is probably his doing...