Rick is told that Glenn and Maggie are taken by a group while in the process of getting supplies which included baby formula. This implies 2 things: The group that took Glenn and Maggie are hostile, and the group knows that Glenn and Maggie were in a group with other people. One could logically conclude that this hostile group will try and find out from Glenn and Maggie where Rick's group is and attack it. I found it a little ridiculus that Rick only left Carl and maybe prisoner-who-could-be-a-pedophile to protect Herschal, the baby, and the two woman.