I can understand the conflict. If you can avoid the PhD topic (gently steer towards other chatter) I think this would be a benefit to you. Academia never forgets and often forgives, is my experience!
www.plurk.com/CeejayWrit... , knowing these fine folks, it would likely be no issue that the PhD attempt didn't work out. If I avoided the topic, it would be to avoid feelings of personal shame.
I think you would get a lot out of going. Don't give up the chance to reconnect with good folk. Your path has diverged from what you & they expected, but you're not a failure.