It sounds like your pipe may have been bent. It should fit snugly when the whole assembly fits on the mounting brackets. You can bend them back, depending on how far out they are. Sometimes you can do it cold but it is better to apply some heat to avoid stress fractures setting in. The best way is to determine the angle of the bend and then clamp the section that fits into the exhaust port in a good bench vice. Firm but gentle pressure on the stinger section is usually enough to do the business. Don't get brutal with the pipe or you could crease/kink it and that will also stop it working as effectively as it should. Make sure the two neoprene sealing rings are renewed. If the seal is still not perfect even though the pipe fits well, you can smear silicone sealer around the sealing rings. this should prevent any fumes escaping and ensure the pipe maximises your exhaust gas flow - enhancing the scavenging process and promoting more efficient combustion - Yes, it will go better if it is fixed!
Sometimes when the exhaust flange gets really worn out its almost impossible to seal the cylinder/pipe joint. I have wrapped exhaust sealant tape around the flange to take up the slop and that worked pretty good. You can get it for under $10 US from DennisKirk.com. Hope this helps