To me it sounds like he's talking about a tough childhood and growing up. And how he fell in love and was discussing with this person now (present) that he still thinks about the relationship and his childhood, it is very much a part of him and shaped who he is today. Even though it wasn't exactly easy, he still remembers his childhood fondly. He also talks about how it was tough not to want to grow up and to not have sex (innocence was hard to stand by) and that he probably took (asked for) too much from their girlfriends, i.e taking their virginity. All they wanted was the company. But life goes on... and they have all moved on and grown up.
To me it sounds like he's talking about a tough childhood and growing up. And how he fell in love and was discussing with this person now (present) that he still thinks about the relationship and his childhood, it is very much a part of him and shaped who he is today. Even though it wasn't exactly easy, he still remembers his childhood fondly. He also talks about how it was tough not to want to grow up and to not have sex (innocence was hard to stand by) and that he probably took (asked for) too much from their girlfriends, i.e taking their virginity. All they wanted was the company. But life goes on... and they have all moved on and grown up.