The time that the height of the bike matters most is when you come to a stop and need to put a foot down to keep the bike from falling over.
If you are fairly even terrain, such as a track or many trails, it isn't any problem at all. Even if you have to lean the bike over a bit to comfortably hold it there it really isn't a problem.
When you get to really uneven terrain, especially when you are forced to stop at a spot you didn't really want to, it can be a big issue.
I am 6 feet tall and I will occasionally come to a stop in a spot where there is a hole where I want to put a foot down. I have actually fallen over because there wasn't any ground within reach of my foot when I put it down.
As others have said, there are ways to lower the suspension in a manner that you can undo it later if you wish. Buy the bike that does what you want, then make the adjustments to fit your height.
Rod