My guess is that because they are the two opposing / contrasting elements, i.e. Hermit being lonesome, isolated, recluse, ascetic, free, wise, caring and idealistic figure, whereas the Devil is profane, materialistic, likely to be crowded or shackled, pleasure seeking, egotistic and addicted ... etc.
Regardless what sex they are, i.e. both male and female could be the Hermit or Devil depending on the situation and circumstances of their lives, I would imagine.
I have read somewhere that "every element has to be studied from the other side of coin, and in duality"- something like that, but cannot recall where it was from. Anyone?