As with the topic, I also don't like when people decide not to accept starting of a game but however I also realised, that I do this myself aswell. Not simply because I wouldn't want to play the game, but how the game actually starts the way I was not prepared to play it, as I joined the game with the settings which I wanted.
Here is one of my personal reason, not to accept the game start
-I wanted to play a game of 4, because I like it that way for better gameplay, however the game started with more or less players for different reasons, for which sometimes includes player leaving a split second before the game starts or, when host decides to start the game with fewer players, or the host adds more joining spots.
In many games, the rules of the gameplay changes depending on the number of players. Sometimes even though a game allows it to be played with 2 players or 12 players, it's not always the optimal number for me. When I join a game with 4 players wanted, I'm looking forward to play it with 4 players. Personally because with 3 players feels less strategic and the rules might change and 5 players would be too much for me, because it makes the game less predictable and increases the play time, and with more people means there is an increased change of someone timing out and might get kicked off the game, which is very frustrating on a long game.
I use manual joining most of the time but it still happens, that a game gets started with fewer or more players than I personally wanted to play it and it is a reason for a lot of players to hit the disagree button instead of agreeing to play the game.