I love walking in this city. The hills can be hard work, but worth it for atmosphere, buildings and views. If you start at Union Square and walk on Powell towards Bay, you can take diversions off to ### Hill (Some of the big houses not destroyed in 1906) or California (Grace Cathedral) or Union Street (some good shops), or go a bit through Little Italy - which could include a walk up to the 1930s Coit Tower on Telegraph Hill for a panoramic view - last time we were up there the earthquake practice siren went off
I love the sea lions at Pier 39 - could watch them for ages - lazy, loud and lovely. Other than that, though, I don't know if I'd bother with Fisherman's Wharf any more - it's a great place to stay, but not to visit or shop if you don't like crowds (though my kids made me go to Ripley's Believe it or Don't, last time were were there)
This looks like a really good site, so there could be some eating recommendations worth following.
http://www.sftravel.com/sftodo2.htmlIf you like garlic, try the Stinking Rose

Oops - it didn't occur to me that the name of the Hill wouldn't be acceptable - It's on Clay, between Larkin and Leavenworth, and it's not at all rude