The steel tank and steel rims also add weight. (Not sure if the rims are different form the dirt version). Typically the factory dual sports will have inferior suspension components to keep the price down, as the lights and other stuff cost $$$. Different jetting and maybe different carbs also.
Not too many emission controls were found on the '84 XL model, unless maybe there was a "California" version. My college roommate owned one. The bulk of the weight was the battery, steel swingarm, steel gas tank and lighting system. I thought it had aluminum rims, but I could be wrong... consumed more than my share of beer in those days.