Yes, I can understand that. The only thing I might be wary of is whether there's an added attraction in his unavailability that makes you want him more.

I used to get completely besotted with guys and go off them after a few weeks of dating! Of course, you already have a relationship, so it's not like that, but I often think back to those days and think, 'why did I get so obsessed only to get tired of them quickly?'

In truth, I usually went off them because they were annoying! But there's one other explanation beyond annoying, and that's the attraction of wanting what we can't have.

Whatever you do, I hope it feels like the right decision for you both in the long run.