Roll 2 Water - Lose 5 points
Roll 2 Leaf - Lose 5 points
Roll 2 Fire - Lose 6 points
Roll 2 Gem - Lose 3 footprints
Roll 2 Cloud - Kill your most recent companion
Roll 3 of the same colour - Lose 7 points (there are 2 of these tokens)
A few things worth noting. These are all 1-time effects. If a player has 3 points for example because they have been re-rolling they will only lose 3 points, you can't go into negative points. There are certain spots on the map that let you kill a companion, they can also be used to remove these curses.
For those reasons Xar'Gok is a little underwhelming most of the time, but can be good if there's not much else.