You only receive points for
completed cards. The points are indicated at the
bottom of each card as NNNN, the first two digits are the points, the last two are the card number. So, there's a series of 1 point cards (01NN), of 2 point cards (02NN) and of 3 points (03NN). {But I think there is only one of the later: 0301.}
I don't think the old cards come back up, as there's at least 16 of them, which is more than you can swap. {Not sure about the exact total.}
I understand your confusion though, it is hard to track what is going on since card completion is not logged nor are the points specified separately. On top of that, "spare" points seem to appear out of nowhere sometimes. Now I think it is due to cards
auto-completing between turns/swaps and since it isn't logged it happens too fast to notice. {Either the later or there is perhaps a bug.}
Anyone with more insights, please chime in. Thanks.