Try some silicone, if you don't ever want the speedo agai cut the end that go in the hole and plug the end with silicon, now the next guy can undo it easier. You could also machine a blank, shaped like the end of the cable.
I keep mine plugged with silicone works well. It was that way when I got it. Got a odometer and cable pulled the silicone right out installed the new cable rode it about 3 weeks something tore my cable off so I plugged it back with silicone.
There have been posts before where guys have used tooth paste tub caps, silicone tube caps, and other things of that nature to plut the hole, then covered it in silicone to seal it. This was you can pop the whole mess out pretty easily.
I machined a brass plug for mine with an o-ring on it. This way I could swap back and forth for dual sporting or when running the occasional MX or brush infested trail.