When you write a total differential such as
$$dH = \left(\frac{\partial H}{\partial T}\right)_p dT + \left(\frac{\partial H}{\partial p}\right)_T dp \tag{1}$$
you are applying the methods of differential geometry, so in reality the answer to your question lies in the applicability of these methods in thermodynamics (the rest being "math"), something which statements such as "so-and-so is a state function" implicitly justify. One can write a total or exact differential of a state function, as for the enthalpy in the equation above. This equation can be interpreted as follows: small (differential) changes in p and T, which are orthogonal dimensions (in the sense that they can be varied independently), additively cause a linearly proportional differential change in the function H. In the differential limit, the surface of H looks like a plane. The partial derivatives describe the slope of the plane in the orthogonal dimensions.
The cyclic rule can be derived from the above equation by taking the partial derivative wrt one of the independent variables while holding H constant.
$$ 0 = \left(\frac{\partial H}{\partial T}\right)_p \left(\frac{\partial T}{\partial p}\right)_H + \left(\frac{\partial H}{\partial p}\right)_T \tag{2}$$
What does it mean here to hold H constant? It means we are looking for an isenthalpic path on the enthalpy surface, from the initial point at which we computed the partial differentials of the surface wrt T and p, in direction $\left(\left(\frac{\partial T}{\partial p}\right)_H dp, dp\right)$, where the partial differential $\left(\frac{\partial T}{\partial p}\right)_H$ also happens to be given (thanks to the geometry of the problem) by
$$ \left(\frac{\partial T}{\partial p}\right)_H = -\frac{\left(\frac{\partial H}{\partial p}\right)_T }{\left(\frac{\partial H}{\partial T}\right)_p} \tag{3}$$
Alternately, consider the line resulting from intersection of a horizontal isenthalpic plane $c(T,p)=H_0=H(T_0,p_0)$ and the plane $s(T,p)$ tangential to the surface $H$ at the point $(T_0,p_0,H_0)$, the tangential plane given by $$s(T,p) = H_0 + C_p \Delta T + \varphi \Delta p$$
where
$$C_p=\left[\left(\frac{\partial H}{\partial T}\right)_p\right]_{(T_0,p_0)}$$
$$\varphi=\left[\left(\frac{\partial H}{\partial p}\right)_T\right]_{(T_0,p_0)}$$
are the partial derivatives of $H$ evaluated at $(T_0,p_0)$, and $\Delta T = T-T_0,~ \Delta p = p-p_0$. Solving for the intersection line by setting $s(T,p)=c(T,p)$ gives
$$T = -\frac{\varphi}{C_p} p + T_0 + \frac{\varphi}{C_p} p_0 +\frac{c-H_0}{C_p}$$
The slope of the intersection line can be recognized to be the same as $\left(\frac{\partial T}{\partial p}\right)_H$ given by Eq. (3). The geometric nature of the problem and relation between the different derivatives should then be clear.
So how can we now use this variable when dH≠0?
A Joule-Thompson coefficient $\mu$, like any other thermodynamic (or state) properties, is strictly valid at the conditions under which it is determined (it may have a broader useful range depending on how much it varies with T and p and the tolerated error). This is different from the question of the mathematical accuracy of the relationships used to derive the properties. $\mu$ is derived at a specific state defined for a pure substance by a specific point (T,p) and as such is a fixed property of the substance at that point.
