some stones are pretty fragile, even the polished ones. I have kept them in little bags together, and it is possible that even the slightest bang up against each other can cause damage.
For the most part, I don't carry around any of the more valuable or any of my more treasured stones. They stay home. I find a smaller, less expensive of the same type and then put it in a padded or cushioned carrier, like a small change purse or something like that. I also keep them loose in my pocket sometimes - and if something happens or I lose it, no big deal.