I recently discovered an issue with WCF 3.5 SP1, specifically the error is “The caller was not authenticated by the service”
System.ServiceModel.Security.SecurityNegotiationException was unhandled
Message=”The caller was not authenticated by the service.”
My setup was using WSHttpBinding with message security, and it worked locally but not on a hosted machine in a domain. I had two services on same box, calling each other, and this issue cropped up. I was using the dns as the identity in the configuration file, however there was a change in 3.5 SP1 and now it requires either a SPN (service principal name) or a UPN (user principal name) to work correctly. The interesting thing is you don’t even need to know the spn since it is automatically created when the service is hosted. This is only needed for client configuration calling the services.
You can leave SPN blank it will work if default is kerberos, otherwise you will want to specify the specific spn for negotiate to fall back on.
So, if the host is running with user credentials, you should use its UPN:
<userPrincipalName value=“email@example.com” />
And if the host is running as s service, specify the SPN
<servicePrincipalName value=“Host/MACHINENAME” />
I hope this helps.