I will assume you are talking about regular grow with room as urgent growing is generally a very strong action. This game is a wonderful mix of efficiency and tempo, so there is no easy answer, it depends on a lot of things, but here are some questions you might want to ask yourself when deciding on the 4th grow.
1) How empty is the growth queue?
As in: are multiple people with empty rooms fighting for the space with start players? The more players with empty rooms, the more likely you are scoring better with just efficient farming.
1.1) When are you expecting to grow next?
Related: if you are expecting your grows to delay much, it is possible you should try to set up other efficiencies.
1.2) When did growth flip?
Related: early grow (round 5) generally means emptier growth Qs down the line, while late growth (round 7) with busier growth Qs.
2) Can you handle the food increase?
Generally bigger families with more actions are easier to feed, but you do need a solid feeding plan still. If your plan is to bottom feed even more or just grab single animals to throw them into your cooking hearth, then growing might not be super efficient.
3) Do you have cheaper rooms / cards that make building easier?
Cards like Carpenter's Parlor, Clay Supports, Bricklayer, Recreational Carpenter, Cottager or Swagman (these last 2 better with a combo) can make additional rooms more feasible.
I am sure, I am forgetting other considerations, but overall, the answer is: it depends. Hope these pointers help though
