I did a 500 word per chapter, and you can imagine just how irritating it can get...
Most often, my stamina can't keep up with my thoughts and schedule, so I'm stucked at 500 words per chapter. It's not just irritating to my readers, it's also very irritating to me.
Though initially, I opted for longer word count atleast 2000 per chapter, but the readers started 'suggesting' to put more chapters... In average, a novel have the minimum of 40k to 70k words. Imagine the number of chapters if you go in a 1000 word per chap, their would be 50 chapters right? But that doesn't how it really works. Controlling your word count like that seems forced and unproductive.
It's quite difficult to create an outline to follow for writing as an exact guideline on how to write a chapter. Sure, we have a prolouge, epilogue, interlude and the plot. But those stuff doesn't really tell you how much you should write in a particular chapter.
Yes, prologue, epilogue and interludes are shorter than the plot, that's a given. But that doesn't tell you how much you should write in the chapters to follow.
So my answer is... Chapter size relatively matters... It's a very safe answer, I must say. Yes, it matters because word counts give you a goal. No, it doesn't matter because their are no limitations to writing(this is just my opinion though.)