Here's a quick rule of thumb to put you on track: you must decide whether you're writing the story to introduce the world, or writing a world that provides to the story. Is your fantasy story a Lord of the Rings, or a Berserk? Is the theme of your story centered around the world, or is it the other way around? It's better for your fantasy world to have a point in its creation, like the theme of destiny and breaking thereof in Berserk, or the classic fate in Tolkien's lore. Otherwise, it'll just be a pointless world with padded things that won't matter anyway besides the plot.
Sometimes, having a default template of a fantasy world is enough. If you keep adding more lore without any meaningful storytelling it'll just feel bloated and overimportant, making the story itself feel underdeveloped and poor.
You shouldn't put too much focus in creating the world your characters inhibit. A little bit of lore can be fun but it's mostly better to just treat it like a detailed background to further indulge your readers in. Unless your fantasy world IS the protagonist your story, then by all means, go for it.