Personally, would love to see at least two ladders is all. So I agree with OP.
I don't really like 7WD Vanilla. I find it needs pantheon to be a proper competitive game, and just personally don't really want to climb the ladder on vanilla. A ton of people like Vanilla, I get it. So it needs to stay. Just not for me.
Breaking down the other options:
Base + Agora: I just don't think this really has a right to be its own ladder. I enjoy it, but I enjoy it way more with Panethon included, because base + Agora takes a swingy game and makes it swingier. It's fun, but not really competitive enough to justify an arena ladder.
Base + Pantheon: Probably the single best competitive mode. Adds so much with the reveal tokens and the option to play Gods instead of drawing, the great control you have over the Gods, etc. But there's some pros to adding Agora.
Base + Agora + Pantheon: Pantheon greatly helps tone down Agora's swinginess. Agora offers a significant/needed buff to blue card, slight buff to military, interesting meta game in senate control, and more routes to economy that challenge the value of early browns/greys. Sure it adds a wild factor with conspiracies but it's not enough to completely throw off the integrity of the game with Pantheon included.
So I'm good with either Base/Pantheon or Base/Pantheon/Agora. If there's room for 3 ladders, that's cool too, but I understand keeping it to 2 ladders has benefits. Of the two, I'd just vote to do Base/Pantheon/Agora. I think it makes the most sense to have a ladder for all expansions and a ladder for none. It's also my personal favorite way to play so perhaps I'm a bit biased.