in A silvertongue's tale, everyone is capable of magic, some more so than others. For instance, anju ( a character that's to be introduced next chapter), has very minimal magic. It only allows her to make and use spell cards. Each time she uses those, however, she feels a tad drained.
Then there's someone like Kerren who has an extraordinary amount of magic capability that Ice, so he wasn't more powerful than her, had runes etched into his skin to block his magical output. Kerren can still use magic, and he gets affected like a normal user. He gets more tired than normal. Magic takes your stamina first, then your lifespan if you deplete all your stamina. (Then there's blood magic which will, of course, take your lifespan, but it also requires a chant to keep it going).
The only ones who are technically 'free' from the limitations of it taking their lifespan are Deaths and Gaias, as they are made immortal by the High council. Though they do have rules about how they can use their magic. They must follow the way it's written, so to speak. But they can still get drained. Normally, they have one day where they sleep all day, kinda like a mini hibernation (and that reminds me of another post to make later about the Deaths and Gaias), but if they overuse their magic, they'll have more than the one sleep day per month.