I've actually got two Nexus 5 phones. One is my work phone which has always been on KTU84Q, which is the special 4.4.4 image for a few carriers (including Telstra) in Australia, and the other was on KTU84P, for which I receved an over the air update to 5.0 and later 5.0.1. I have a 2013 Nexus 7 which is on Lollipop after an OTA upgrade.
As a result, I've had ample opportunity to compare 4.4.4 and 5.0.1, and it's clear to me that I not only prefer 4.4.4, but it's better than 5.0.1.
Problems with Lollipop
Audio on Lollipop over Bluetooth is problematic (at least it is with my Renault Megane RS 250). Waze and Google Maps voice prompts don't play - they just stop the music playing.
Bluetooth handover from car to phone is 5-8 seconds, rather than about 2 seconds under KitKat.
When listening to FM or AM radio, the Nexus would send the odd short noise to the radio head unit, like it was trying to force a Bluetooth connection, but not following through.
The quick access to WiFi and Bluetooth in Lollipop from a swipe down is different in behaviour to KitKat - in KitKat you can either toggle WiFi or Bluetooth, or choose a new WiFi network or Bluetooth device. That was taken away in Lollipop - you can only turn them on or off.
Battery life... this is a bit like talking about unicorns, but I had a suspicion that battery life dropped off with Lollipop, and now after a couple of days back on KitKat my battery life is restored. However, to be fair, I haven't put back every single app onto my phone since the downgrade, so it is possible that one of those apps is responsible (but I doubt it).
BeyondPod started behaving badly - the shortcut controls from a swipe down often didn't respond at all, and BeyondPod would become uncontrollable.
There were regular "pauses" on the phone generally, where nothing would happen for a few seconds.
The things I've mentioned above were clearly enough for me to downgrade, although I'll leave my Nexus 7 on Lollipop so I stay in touch with what's going on with the OS. However, I can add another reason - Lollipop didn't offer any substantial advantages that outweighed the inconveniences in the interface.
I should say that none of these issues arose on my work phone that had stayed on 4.4.4.
Despite the absolute rubbish you read on so-called tech news on the web, a lot of phones haven't had Lollipop pushed to them - I have a feeling Google knows that there are issues with it. Trust me, you don't really need the minor conveniences that Lollipop gives, and the "tech tips" that some sites give to fix Lollipop problems on a Nexus 5 really don't have a foundation in fact and are simply based on hope.
If you want to downgrade?
You should probably just search for the answer, but the short version is:
- you'll be completely wiping your handset, which shouldn't be a problem if you use it properly (i.e. let it back up photos etc and not have the only copy of anything on it), but...
- if you use Google Authenticator or any other two factor authenticator you should migrate them to another device or use a different second factor (SMS etc)
- you'll need to download some stuff - the image file, ADB drivers from Google, and the ADB Tools
- you'll need to understand how to boot the phone into recovery mode and it's a little more fiddly than you think
- you'll need to spend some time manually reinstalling your apps