Does the density of a liquid and gas phase of a substance(nitrogen or water) converge when reaching critical point

I know that the there is no difference in density in a liquid and a gas phase of a substance after the critical point. But I do want to know if this difference in density is more continuous or if it is more like a big step when reaching the critical point? I.e if we are like 0.01 Kelvin beneath the critical point of water (with the critical pressure). Is the difference in density still big at this point or is it near zero?

Yes. This is exactly what happens. Let's take a look at a qualitative $P-T$ (pressure–temperature) phase diagram of an arbitrary substance:
As you can see, moving directly from gas to liquid involves crossing the phase boundary. However, if you're doing it in the supercritical, the change in density is continuous. Maybe you can get a better, more quantitative view with this example of $\ce{CO2}$:
This one is also a a $P-T$ phase diagram, with equal density $(\rho)$ lines. You can see that the lines radiating from the gas–liquid boundary lines are not equal on each side. For example, 0.1 and 0.9. However, as you get close to the critical point, $\rho$ on each side of the line converges to a single value: 0.2 and 0.7, 0.3 and 0.6 and so on.
What you see is a function of density - you can see the phase boundary between liquid and gas because they have differing densities. Notice in the videos how the phase slowly and gradually disappears as the density of both phases converges, effectively turning the $\ce{CO2}$ into a supercritical fluid.