Yes. The #4 is stability. But the Pents are ALSO stability. That makes 4/Pents Stable-Stable...which is too much and works against it. Best example: if you live in a earthquake area, building with give and sway will stay up. Those that don't, that are uber-ultra-stable will come crashing down.
So this is why the card is not about laying down stable foundations, but rather about being immovable, often to a damaging degree. What's been built here has much concrete and stone and glue that you're stuck with it exactly as it is. There's no way to adjust it if you change your mind, or need to expand, or need to build up that "empire" to a different height, etc. It also suggests that the person is saying, "we're done." No *more* building. No more expansion. Empires require solid foundations, yes, but they also need the ambition and energy and desire to keep building. That's not in this card. Pents are very slow moving, and 4/Pents indicates no movement at all. It's frozen in place.
This is also why--seemingly counter-intuitively--the 4/Wands is usually the card about the completion of a stable foundation and the promise of expansion (empire building) upon it. With Wands you're stabilizing energy. So it's like you put in the fireplace--you have all the heat, warmth and such you need to keep building. Go onto the 4/Cups, however, and you've a problem because standing water can go stale. You want water to flow. So stability for cups isn't so good. 4/Swords is good, however, because it stabilizes the turmoil, gives it a resting place. Air is quiet and still here rather than blowing all around.
Which is all to say, you're right about the meaning of 4's, but you have to take the suit into consideration. 4 = Stable. Pents = Stable. And Super-duper-not-going anywhere stable is not the foundation for an empire. It's a finished bunker locked up tight