Good that helps narrow it down...
First I would get a nice piece of MOONSTONE and put it in the main sitting area of your home....this will help ease the tension of the day and mark a nice transition from your world at work to your world at home.
There are a few stones that you can take with you and keep at work or keep on your person that will be of value to you in this situation.
If you sit at a desk a very affective tool to keep people around you from bugging you is to get a ROSE QUARTZ wand and an AMETHYST wand make sure they have at least one end with a point on them.
Turn the point of the rose quartz toward the person who irritates you...this will not harm them but act as a barrier...and turn the Amethyst point toward yourself....this soothes anger and impatience.
PURPLE FLORITE will absorb panic vibes..... so if this person thinks you are panicing, but you're not this might help equalize the opposing energies you are both feeling.
LEPIDOLITE one of my favorite stones to wear to give you a calm still center.
ANGELITE, now this I would definitly get a nice big chunk of it's an antidote for abrasive people....encourages them to be sensitive and kind ...
Two others that are worth mentioning are BLACK OBSIDIAN it dampens inner anger and frustration at others' incompetence or subborness.
And DANBURITE this one is a little hard to find but it is good for upping your tollerence levels if you work with difficult people.
So I hope this helps and that you might have a few of these stones right now so you can get a little relief from the situation....a glass of wine helps too.