He's not worried so much about the Fereldans; he's seen how informal they are and has got over himself in that regard. Orlais is his main concern; an affair between a prince and a noble would be fine, but anyone below that is his catamite, his whore. Especially an elf.