I won't miss him, but I'll remember him fondly. Letterman was deliciously different even when he was an ass, at least for the first ~15 years, and somebody would inevitably bring up something from the show the following day, so...
I saw what I remember as his first week on the air doing his short-lived morning show - home sick from school - and loved it. All of the regular bits they did to a series of object the first however many years - running a steamroller over them, throwing them off a tower - they first did on the morning show as quick bits done to the then-ubiquitous NYC cardboard cup of coffee with blue greek-style artwork on the side.
Late Night was a fixture for me until something like the end of the 90s, I guess, even if it was only to tune in just long enough to catch the Top 10 list. I'm not sure when I stopped doing that. For that matter, I'm not sure when The Daily Show became the same sort of evening ritual...
But Letterman was just entertainment. Shockingly different for network television, and a trailblazer for those that would follow on cable especially. Would Dennis Miller or Bill Maher have gotten their first late night shows if Letterman hadn't gone ahead? But by contrast Stewart and Colbert had more of a message, with something current that deserved attention, delivered in an entertaining way. Seems more important to me now, and I wonder what Colbert's show will be like in September.