Ugh, world building. Generally, world building in manga is tolerable simply due to the constraints of the format. In prose there's nothing stopping the writer hitting you with 10k words of exposition right out of the starting gate (often labeled as a prologue). The moment I see that it's time for the back button. That's a problem that goes far beyond just isekai, it applies to fantasy and science fiction in general.
The best world building is where it's done as sparingly as possible and only where necessary and where it makes sense.
Ok, I think there's a confusion of terms here. What you're complaining about here is called Exposition. This is a lazy way to tell people your world building.
Actual competent writers won't do exposition dumps like this. They will show you their world building just by having the characters live in the world and discover it.
And also, seriously, exposition dumps in Isekai really are a sure sign that the author has no idea what they're doing at all. The biggest strength of Isekai is the "fish out of water" dynamic, which allows you to actually present the world to the audience via your protagonist as the protagonist is just as new to this world as the reader is. This means you actually can quite easily introduce them to the world a little at a time via exposure rather than dumping it all at once to catch the reader up with what your protagonist already knows.
Seriously, you're making the right decision backing out the moment you see an early exposition dump in an Isekai. It means the writer is tossing aside the single greatest strength of Isekai, which is a sure sign they have absolutely no idea what they're doing.
Also, if what they call "world building" amounts to nothing more than a list of races and country names and the relation of one country's political relations to another, that's another sign that the author doesn't know what they're doing, especially if they're dumping this info very early on in their story. That's not the actual interesting world building. Interesting world building would be things like what kind of environmental considerations the towns people take into account when they build their houses. And, telling the reader those considerations is also a no-go move at that. The author should just describe interesting features about the house and let the environmental reason for the structure be something you figure out by working backwards.
One group being prejudiced against another is also a sign of good world building if shown well. And, by shown well, I mean "show, don't tell." Actually show the prejudiced actions or the impact it has on people's lives rather than just saying "this group hates this other group."
So, yeah. World building = good. Showing the world building via exposition dumps = bad. So bad it can actually trash any good work you did with the world building.
Again, consider some of the greats in this. Mushoku Tensei, Overlord, and Shield Hero handle the presentation of their world building excellently, and in the case of MT it's actually some incredibly elaborate, in-depth, and very original world building. Re: Zero also has what seems to be some pretty good world building, but it actually goes to the opposite extreme in terms of how it presents the world building to the reader. Re: Zero's world building is a little inscruitable. It tells so little to the reader/viewer that you actually need to go to outside sources to understand how things like the magic system works or the various groups involved. And, really, not knowing that stuff really doesn't harm your ability to enjoy Re: Zero at all. The world building is there for Re: Zero, but it doesn't actually matter to Subaru's story at all, and as such the author decides not to tell it all that openly to the reader and just lets the super-fans figure it out by going to the extra effort if they really want to know.
And then, you have Ascendance of a Bookworm. OMFG, Ascendance of a Bookworm. This series is the absolute god-tier of world-building and also god-tier in the slow and perspective-driven presentation as well. It starts you off knowing nothing more than the inside of Mine's house. Then, it shows you the city streets. Gets you familiar with the city gate and the forest outside. The activities of the kids, and goes into detail about winter prep. And, the existence of magic is completely hidden from the reader for a huge chunk of the early story and is more dropped as a shocking reveal much later into the early parts of the series. The world building of Bookworm is incredibly elaborate, but it shows it to the reader a little at a time and never overwhelms them with it. It doesn't show anything until it matters, and it's ALWAYS "shown" rather than "told." There is very little outright exposition in Bookworm at all. In fact, I can't think of a single moment in Bookworm where there was even exposition at all.