When I follow the setup rules all of the cards start in the central display then each player that is part of the game takes a politics card and builds their starting deck from the central display. All remaining cards stay there. This leaves more cards in the display for less player counts. The current implementation has the same setup every time based on all 4 player hands being created.
For a 2 player game I believe the starting cards should be 16 survey, 14 warfare, 16 colonize, 12 produce/trade, and 16 research cards. Please check on this and confirm/update the setup on BGA.