One of the things you need to do when looking for a server to run Asterisk on is figure out how much bandwidth you need for the number of concurrent calls you’re expecting to have.
A great tool for this can be found here – http://www.asteriskguru.com/tools/bandwidth_calculator.php
Just set the codec you’re going to be using (check with your VOIP provider – g.711/ulaw is usual and the highest quality), the connection type (usually SIP or IAX2 with Asterisk) and the number of concurrent calls. It will then display the bandwidth required for that many calls.
One thing to watch out for if you’re planning on mixing codecs (say g.711 on one leg of the call and g.729 on the other) is that your server will have to transcode/convert the audio which is processing intensive. This may limit the number of concurrent calls your server can handle.
Also don’t forget that if you’re planning on running Asterisk at home your upload speed will normally be a lot slower than the download speed.