I have already mentioned that google will start penalizing your site in search if it is not fast enough. I have recently given a presentation at Drupal Con about Drupal performance and page speed. Below is a video of our session - its currently the fifth most watched session from the conference according to archive.org. Alternatively, there are several different formats there as well.