'Adding half a second to a search results page can decrease traffic and ad revenues by 20 percent, according to a Google study. The same article reports Amazon found that every additional 100 milliseconds of load time decreased sales by 1 percent. Users expect pages to load in two seconds—and after three seconds, up to 40 percent will simply leave.'
Source: Improving UX Through Front-End Performance - A List Apart article by Lara Swanson (2013)
I was just trying to work out how to optimise the crap out of my images and came across this good little article.
If you haven't thought about having a performance budget for you site yet this is why you should.