Even if it was as little as a few hundred, it makes sense to me that it would be enough to make Loghain nervous. People don't forget occupations quickly. I think he could be justified in his fears or not...

Woo hoo open-ended game!