It's 異世界.
異 - different, strange, wonderful...
世界 - the world.
Isekai just means "another world". So anything where your character switches up worlds is isekai. The reason it's mostly our world to a fantasy one is simply for self-insert reasons. For the genre itself, the important thing is the transfer into another world. Isekai as a settings-genre is just as massive as "sci-fi" or "fantasy". In fact, it is so broad, that even works of people "travelling back in time to a 'different state of our modern world'" are often listed as isekai, because the 'new' world is completely different. Which is also the reason multiple sub-genres were created over time.
異世界転生 would be "another world reincarnation" which is the usual MC dies and gets reborn with all his memories intact.
異世界転移 would be "another world transfer/summon" which is the transfer of the human itself. More or less the western "portal fantasy".
異世界憑依 would be "another world possession". Examples for this more niche one would be those where the MC becomes an armor or a hotspring.
So in short, isekai really is just about the move between worlds. There is no need to have our world in it. Going by the usual rules for tag, the main requirement would be that the main/title character is the one switching worlds. So if your MC summons a girl into his own world, that wouldn't be an isekai. But if the MC is the girl summoned, then it would become isekai.
For the western audiences, isekai is a bit less well defined because a lot of people just throw their own labels on translated works and call it a day... oh... and also: Yes.