Don't click the "Open Table to other players" button or start the game until you're sure everyone you've invited has confirmed their participation.
The "Play with Friends" option apparently doesn't give you a chance to edit the configuration of the table before it automatically sends invitations. This is usually fine for real-time games, but BGA has an often-frustrating restriction on turn-based games that you can't change the table configuration after other players have joined.
The best way to prevent this from happening is to create the table manually without using the "Play with Friends" option, set the table up with both the minimum and maximum player counts set to the exact number of players you intend to invite, then invite everyone one by one. Once everybody joins, the game will start without ever being open to the public.
