To me, it is more on how incongruous the word is in terms of world building. A lot of words are steeped in history and this history sometimes clash with the story's setting. A lot of expressions, especially cuss words, fall under this. "Okay" "f*ck" "shit" "down the drain" "when the shit hit the fan" and many more just doesn't fit given the history of how these expressions come to be.
Worse are the words that are heavily connected to modern technology. "Selfie," "online," "robot" and "cartoon" sometimes appears in works where they just don't make sense. Are there mobile phones, internet, intelligent machines or TV in the setting? The anachronism is just too glaring. Some aren't that distracting through like "trainwreck" "(TV) screen" and "balloon". They are just so invincible to me.
As for foreign words, the same thing. Some foreign words are actually very commonly used in English. "Et cetera (etc.)," "et alia (et al.)," "cum laude," "touché," "bon appétit," and the like are just so widely used. Many doesn't realize that many words they are using are foreign. "Alpha" "versus," "status quo," and the like are often thought of as English.
So, mostly no, especially if the words used are the common and familiar ones.