You will unite or you will fall

Because they were trying (and failing most of the time tbf) to keep their movements discreet, traipsing around with an army would have brought more attention than they wanted

On top of what has already been mentioned about wanting to lay low and not attract too much attention with a big army, I guess there's also the aspect of the Ring corrupting those around them and not wanting to risk an increase in these chances with more people, particularly warrior types who were less in the know, who could cause more problems if corrupted. Elrond knew first hand the dangers of the Ring's corrupting power. The Fellowship were seen as the right candidates to undertake the mission, in terms of size, trustworthiness and ability.