Stetenorve, you are welcome!
You are right, it is partially about the attitudes of other singers and, certainly in one case, the attitude of the choir director.
The more I think about it, the more complex my dilemma seems to become. I don't have a problem singing popular carols or traditional Christmas songs, maybe because I see them as part of the culture I grew up in. I am more comfortable with the Latin because I can regard it as nonsense syllables, even though I know what the words mean. I can reinterpret the spirituals in the light of the slaves' experiences and the coded messages they contain.
However, I am quite 'method' in my approach to performance (I do a lot of am dram) and I DO really get into feeling the part I am playing, so maybe that is why I feel so uncomfortable.