I do have the same problem, but I think this is an issue Skype itself triggers. Since I have problems with the communication between Adium and Skype I also do have problems with Skype and Growl. Skype seem to lose the connection to growl and switches all the time (or some time...) back to the fallback built-in notifying-system. (Skype-Ticketing-System: https://jira.skype.com/browse/SCM-1017 - but the suggested starting-order doesn't solve this issue)

I started Adium in Debug-Mode today and saving the protocol to my log-folder, so I come back here, when I was able to log this issue.