It's the idea of baggage. When you hear about people in their 40s boast about not having baggage. I think having no baggage is your baggage. That means that you haven't thrown yourself into the mess of life.
He talked a lot about the past and I gathered that he wanted to recover something, some idea of himself perhaps, that had gone into loving Daisy.