It sounds like the shift shaft return spring has popped out of place. There was an old thread about that a few months ago.
If that is the problem, drain the oil. Pull the RIGHT side engine cover off (you will probably have to remove the brake pedal). The shift shaft will be near the bottom rear of the engine. It is the same shaft your shifter is connected to on the left side of the bike - the shaft runs all the way through the tranny and the spring is near the end of the right side of the shaft. If one of the ends of the spring pops out of place (or the spring is broken) the bike will do as you say.
You can find a diagram for your tranny/shift mechanism at
www.buykawasaki.com It shows the spring. If this is your problem (I think it probably is) it is very minor surgery and does not require splittig the cases. Shimming the spring by putting some washers on the shift shaft might help it stay in place better. Thats the way is was on my old '84, anyways.