A flip-flop stores a bit in a feedback loop that has two stable resting points, 0 and 1. If D changes inside the setup and hold window, the loop can get an almost perfectly balanced push and end up teetering between them. That's metastability.
Picture a ball on a hilltop between two valleys. It will roll into one valley eventually, because any tiny bit of noise tips it, but you can't say which valley or exactly when. Meanwhile the flip-flop's output may sit at an in-between voltage or take much longer than tcq to settle.
The key fact: the chance that it's still unsettled falls off exponentially with waiting time. Waiting a little longer makes failure dramatically less likely.
Inside a synchronous design, the setup and hold checks stop D from ever changing in the window, so metastability can't happen. The trouble is an asynchronous input, such as a push button or a signal from another clock domain. It has no fixed timing relationship to the clock, so sooner or later it will change inside the window. You can't prevent that; you can only contain it, with a two flip flop synchronizer.