You only receive points for
completed cards. The points are indicated at the
bottom of each card as NNNN, the first two digits are the points, the last two are the card number. So, there's a series of 1 point cards (01NN), of 2 point cards (02NN) and of 3 points (03NN). There is only one of the later (0301), but there is also a card incorrectly named 0001 that also scores 3 points (0302 maybe).
I doubt that the old cards come back up, as there's at least 16 of them, which is more than you can swap. I'm not sure about the exact total though.
I understand your confusion, it is hard to track what is going on since card completion is not logged nor are the points specified separately. On top of that, "spare" points seem to appear out of nowhere sometimes. Now I think it is due to cards
auto-completing between turns/swaps and since it isn't logged it happens too fast to notice. {Either it is the later or there is perhaps a bug.}
Anyone with more insights, please chime in. Thanks.