I do like the idea of more teams in the World Cup, if not for the same reasons as the OP.
If 32 teams was the case there would be a greater number of '3rd tier' teams and I think that would leave to a 'competition within a competition', with a greater number of close matches. I think it would do the lesser teams from the last world cup such as Japan and Namibia some good to have beatable competition in there, for example Madagascar, Spain. Because I think for entertainment purporses it would be great for each place to be contested, whereas now many group stage matches you feeel pretty apathetic about because it's usually two teams from different tiers playing one another with the likely outcome already know.
Not to mention it's a huge step towards making rugby a more universal sport at the professional level. We talk about wanting to close the gap between the top tier of teams and the 2nd tier, but if you want to make rugby a less elite sport you have to start from the bottom and give access to bottom tier sides.
I know its logistically more complicated than that though. Perhaps squads would need to be bigger to compensate for the extra games.