That's an excellent question!!!
The first thing that needs to be made clear, is something that confused me for a long time, until I took up a traditional approach. And that is the use of the word 'ruler' I'm going to do an aside on Sign rulership first because the contrast is important
There are two distinct used of the word and they are quite different. The first relates to signs (not covered by your question). The sign rulerships are determined by 1) assigning the rulership of the Summer signs to Moon and Sun - so the Moon gets Cancer and the sun gets Leo 2) The remaining signs are allocated by applying the chaldean order of planets (Saturn inwards) working from Winter (Capricorn and Aquarius) through to the Spring (on the 'Lunar side' and Autumn (on the Solar side) ending with Mercury getting Gemini and Virgo.
I've laboured that side point because there's something vital about it for what follows. The rulership does not depend on affinity - Mercury is not ruler of Gemini because the two are similar Mercury is ruler of Gemini because to it's position in the zodiac circle relative to the Winter/Summer axis. Yes Gemini begins to take on an aura of it's ruler, and shows a side of Mercury but that is not the reason for it's rulership
Now when we come to plants, animals, gemstones, and other objects 'rulership' always has been given on the basis of affinity or similarity. So Saturn 'rulers' all things black, because the colour of Saturn is grey/black Mars rules all things red and sharp objects,
However problems begin to emerge with affinity as a basis - an object can have an affinity with the characteristics of more than one planet. This in turn means that some Astrologers will see one characteristic as being more important than another and so attribute rulership by affinity to one planet, whereas another Astrologer will see a different balance in the characteristics and attribute rulership to a different planet
For example, we might say that Saturn rules black things, and Mercury rules motor cars. So which one rules a black car? If you take black as an incidental property, compared to the car as a mode of transport, then you might go for Mercury. Now what about a black hearse? Its a black vehicle, but it's used for funerals and associated with death - so Saturn may rule black hearses or even dark red ones.
Now if you think about gemstones or plants you have all sorts of characteristics. Both can be used in healing - so we might classify them according to their healing properties or the type of illness they relieve . Both have colours as a property, plants may or may not have scents, plants can have culinary uses Gemstones can be 'hard' or 'soft' can be used in magical rituals.
Individual Astrologers may therefore assign rulership of a plant or a gemstone (or other object) to more than one planet based on the specific uses that they are concentrating on at any one time.
I've used the word 'affinity' but there's no real explanation extant of how the rulerships were awarded in the first place There's a good section in Lee Lehman's Book of Rulerships where she looks at the rulerships of plants and finds quite a disparity of rulerships given by traditional authors Some of that might be explained by my point on usage - for example Nicholas Culpepper was a herbalist, and it's no surprise that his rulerships are medicinally related but it's not always that clear. Lilly for example, attributes the rulership of eagles to, the Sun, Venus, Mars and Jupiter, within the space of about 10 pages, but doesn't explain why this is so. It seems almost as if he just copied it from someone else, but there's enough evidence elsewhere in his works that he thought carefully about his Astrology. So he must have had reasons - he just doesn't really share them.
So, I don't think there's any right or wrong answer to these rulerships (whereas there is for the signs) As long as you have clear reasons for assigning the rulerships, or even if you are accepting those of others. Personally, I go by te use or application I'm considering and the affinity or connection of a planet to that use but there's no text that says that's what I should do