In any society, no matter what type or size, everybody is affected by everything in one way or another. So how would you decide that? (Not by voting, I'm sure )
Democracy can only really work in small communities and only when every voter's vote truly counts. Anything else, including choosing representatives, is basically a mechanism to reduce the influence voters have on the outcome...
Of course there have to be some restrictions on who can vote. But it's a slippery slope. A certain level of maturity/age seems sensible. Any other requirements or restrictions should be approached with great care...