Perhaps the cards can be wrong. But how do we know?
We don't. But what is the immediate guess, made by people?
There is a reader who judges a situation by appling different skills like background knowledge, intuition, maybe simply listening to to the sitter, general experience, whatever. And there is a card that turns up with an attached meaning that collides with the reader's judgement.
You may question the card being the "right" one.
You may question the interpretation of the card. In that case you can stretch, bend and creatively extend the meaning of the card to make it a better fit to your judgement of the situation. But that is just a compromise between common sense judgement and the divinational process by actually diluting the divination and making the reading more "intuitive".
You may question the reader's judgement of the situation and by doing so question all the skills the reader applied.
If it comes down to it, it's either the card (and maybe the reader failing to focus some sort of energies to make the right card appear) OR the ability of the reader to judge a situation by non-tarot skills. To me it seems a bit unfair to question the reader's judgemental skills in defence of a divination process that (as a process) will never by accurate in every single case, a fact that I actually thought would be undisputed.
If a person's common sense judgement stands against the outcome of a divinational tool of whatever kind, I would always blame the divinational tool first, if I have to have an opinion at all.