break up thjat blind sided me

I am in my early thirties, never married, no children. I was asked out by a consultant where I work 7 weeks ago. He is 39, was divorced 5 years ago and has a 7 year old girl. His divorce was very painful for him. I honestly was reluctant at first- saw him more as a brotherly type. Anyhow, I started to enjoy him and I was attracted to him. i have a healthy libido and sex is very important to me. This guy stated that it takes him a long time to get sexually comfortable. He was unable to sustain a hard on with a condom. This erectile dysfunction seemed to be happenning over and over again. I finally got so frustrated mainly because he would not talk about it or admit it was a problem. When I did confront him, he cried and got choked up and told me he did not want to lose me and that he was in love with me. I honestly felt this was too soon. But I definitely did not have that not in my stomach like I have had with guys in the past that I have been in love with. I told him I wasn't there yet but had feelings for him. Well, that was about three weeks ago. Since then, we have spent a lot of time together and I thought we were getting along pretty well. Last weekend, I got really overwhelmed. We spent so much time together, I felt I was being suffocated. I was very irritable with him. Then, this hostility seemed to come out in him and he was making a lot of passive aggressive comments and then denying he was doing it. I apologized last Monday for being cranky. He then said that he needed to think about things. I said that I couldn't wait for that and that we should settle everything that day. He came over that night and told me we weren't clicking and that he spent 8 years in an unhappy marriage and that he did not want to "get burnt" again. I asked if he would give me another chance and he said not right now. He texted me the next day and said he was sad all day. I didn't respond and then has to call him Thurs (really had no choice- work reasons). He said he felt "bad" and missed spending time with me. I suggested maybe we get together for a friendly dinner and he said he would call me when the weekend was over. He lives in my neighborhood and when I passed his street Thurs, I ended up calling him and asking him if he wanted to get a drink. He said he was in bed with a bad cold. I aked if he need anything and he said no. He sounded kind of cold- maybe just sick but I feel like a fool. I know I did what every woman shouldn't. I just got really vulnerable. He told me again that he would call after the weekend since he has his daughter this weekend.
I am confused- If I wasn't feeling like I was falling for him, why am I so hurt? Why was he in love with me one week and wanting to get rid of me the next? I don't know what to do now. I feel so embarrassed for calling him on Thurs.