It is worth to speed up a website, because there is very high correlation between the speed of page load time and the amount of conversion rate. The conversion is a user action, which is desired by the owner of the site. Conversion for a blog is when someone displays an article or clicks on the ad banner, for an online store it is a product purchase.
Watch a transcription of Marcin Merchel speech at Joomla Day Poland 2015. The subject of the presentation was Speed up your website.