Here's what I would do:
You have a gimmick, now you need 2-3 characters who will be the focus of your story. You don't need the story yet, you need to have an opening scene and a little bit of who these people are. At least one of them needs to have a close tie to your gimmick, and I recommend this scene be at least two of the characters meeting for the first time.
Now write out that opening scene.
Now ask yourself "What would these characters do from here?". Keep doing this. As you do this, you will be forced to answer questions about your world, like where they live, how they arrived at the place where they met, etc.
Somewhere in having characters talk about themselves or interact with the world, you should find out their individual motivations and goals, and hopefully you will find inspiration for a long-term challenge/antagonist that will have to be overcome. This should also avoid the tendency for people to make calamities the story's focus. You don't need that, all you need is some form of obstacle or threat for the protagonist that can not be overcome immediately but is also not a looming shadow. This gives you time and space for characters to grow to the point that they can overcome the obstacle.
That's where I started, I am now, 2 years later, I have over 450k words published and am becoming ever better as a writer as I go.
