I don't have any experience with the 250, but I can tell you on my XR650L ('93) it's a toss up as far as if it even runs better with the baffle out. The low to mid pick up is definitely better with the baffle in, and in theory removing the baffle should help top end, but riding the bike i'm not sure I feel it. It's nice sometimes to ride around a "stealth" vehicle that won't bother anyone with noise, no matter how hard I get on it.
To answer your question, are there two screws holding it in? Mine had two screws and a weld neat the bottom, the weld had to be ground away before it would come out. Also, there was a metal peice going crossways in the outlet of the baffle insert, I ground this out with a die grinder, so my baffle has been modified a bit.