Once the fast guys rescued you, you were far better off staying with them than riding alone and getting lost/stuck.
However, you might be responsible for losing the guys behind you, epsecially if you did not wait for them at a fork in the trail, or some other place where they could easily lose your tracks.
It's tough keeping a group with different skills together. At spots where it is easy to lose contact, you really need to wait for the next guy behind you. As much as it sucks waiting, it's even worse trying to find the lost guy. Worst of all is being the lost guy!
IMO, since your friend invited the slower rider, it was his responsibility to watch over the weakest link. When it was obvious the skills in the group were not compatible, he and the slow guy should have volunteered to seperate from the group, letting the fast guys go. At that point, you could have opted to stay with your friend or go with the fast guys.
If you are comparable skill to the guy who invited you, try to ride with just him next time. You both will probably have a blast and hopefully you can learn to trust each other. Of course, if you didn't wait for him at an intersection, he might bend your ear a little bit first.