My guess is the connector tubes Tom has circled are an either/or scenario. Your bike either has the longer top one, or the shorter bottom one. Probably varies by year, or whether it is an XL250R or XL350R.
I found some images on the web that suggest the XL350R has the carbs mounted very close together, with the short tube, which is how Tom described his bike.
IMO, the secondary carb is either getting no fuel, or it's not opening. In addition to examining the carb and fuel flow, make sure both throttle cables are working properly. If the cables are intact and the secondary carb has fuel in the float bowl, it must be something to do with the synchronization of the carbs.
The XL especially may have been jetted too lean, but the way he describes it the second carb is not working at all.