It's correct to say physics cannot be broken in a physical sense. Physics is a concept, and concepts do not break. They only cease to be applicable or interesting. (If one means "broken" as overpowered, then that's subjective.)
If real-world physics do not apply correctly to a given fantasy world, then that world has its own fantasy physics. This is fine.
However, claiming "this fantasy world doesn't have any physics at all" would be very odd. After all, it's one thing for an author to never want to define them, but it is quite another for such to not possibly exist in that world, regardless of character knowledge. After all, physics are defined by observation of the world, not any textbook. Science is descriptive, not prescriptive (unless someone wrote a really, really accurate theory perhaps). (Discworld for an explicit, intended example of fantasy novel with its own physics, much like our own but also quite different.)
On a related note, science and any fantasy phenomena (I can't think of a counterexample) are compatible. The most basic building block of science is the scientific method. That's the bit that starts with observation or recognizing a problem, studying the problem or observation, making a hypothesis, testing the hypothesis, analyzing the results, and drawing a conclusion (and potentially going back to the first step). This is how every scientific field is expanded, whether the scientific method was called as such at the time or not. Thus, any fantasy world can have someone who does a thing, records a result, and thinks of another thing they can do. This is an incredibly low bar to pass.
This requires some amount of causality, although it doesn't require an identical output every time, even if someone controls for every variable they can. The scientific method depends upon a shared understanding of "scientific evidence" and "empirical evidence", so perhaps in a setting where empirical knowledge cannot be attained, science would be trickier to obtain, though I expect such a setting not to be used in most stories.
Someone mentioned causality as part of physics. It would be trivially easy to violate physical causality (or has Wikipedia describes it "Causality(physics)") since that is a precise physics term. So, general causality, A causes B causes C, is not part of part of physics, particularly in that a "cause" can be anything and its effect need not be a physical outcome. So, generally, any understandable story should have basic causality unless someone is really trying to be different. At least, barring more specific definitions of any terms used herein, that's my perspective.
To summarize, fantasy worlds can have physics and science of their own and this shouldn't inconvenience an author or prevent them from writing the world they want.
Yes, I am a fan of "hard magic".
This took longer to write than I thought, so hopefully I didn't make a mistake anywhere.
Shared definitions are important for eliminating misunderstandings in communication and isolating the actual disagreements at hand.
