It does say in most of their bio's and quest logs that they do have a little permanent skill damage roughly 10%  
what that means is this  when you lose to them whether you attack them or they attack you and they do so based on who you attacked if you attack say Knator commander  he is likely to attack you back 2 times to see if your defensive ritual can beat him 
which mine didnt lol    and you lose about .12 to all stats  roughly what you gain is more than you lose if you do beat him 
which i did lol 
98% to 98% lol so close but i barely won and he kicked my butt 2 times for that lol