I accept that the horseback and flag-carrying allusion is strong. The 'problem' lies, I would suggest, in that St James is not (to my knowledge) ever depicted naked, nor, it would seem, with a two-toned banner or flag. These would be very specific if intended as St James - including the common red cross on his banner, I would suggest.
Still, further reflections on this card as intended St James may lead to new discoveries about the depiction of this Saint that we may have simply 'forgotten'... or of another very specific saint!
However there is also the further aspect that roppo mentions: the figure appears to be intended more as feminine than as masculine from the shape of the breasts.
The reference roppo makes to the Song of Songs 6:10 ("fair as the moon, clear as the sun, and terrible as an army with banners"), seems to me to be a far more apt fit of the Vieville card.
Perhaps it should be noted that Vieville would have used the (in this instance incorrect) Septuagint translation of the text, which does indeed render the last phrase as 'bannered hosts' (rather than the likely more correct reference to cluster of stars. Cf Bloch & Bloch The Song of Songs, 1995, p191 for the notes on translation).
... this is one aspect I really like about looking in detail at some of these early decks, both as single cards and in sequence: they lead to their own revelation and likely references but encouraging us to look at numerous aspects both broadly and meticulously, without in any manner taking away from our individual decisions or preference for using the image in still other ways!