It gets very complicated, there are many papers and books on the topic and this little table, from Jensen et al. 1, gives a glimpse of the anisotropic effects:
An extremely short answer is this: when linearly polarized light is incident on an optically active medium, it can be thought of as consisting of equal components of right circularly polarized (RCP) and left circularly polarized (LCP) light. The indices of refraction for these are unequal, so RCP and LCP propagate at different speeds through the medium. This is circular birefringence. Upon exiting, it is linearly polarized light that exits, not the RCP and LCP fictive components. But the differential transit speed results in a rotation of the plane of polarization of the exiting linearly polarized light. This is optical activity, so optical activity is circular birefringence. The longer the optical path, the more the rotation and so on.
This is a truly vast topic, so I recommend not jumping directly into the deep end. Shurcliff's wonderful old book 2, maybe still findable, is extremely readable and a delight: the Poincare sphere is genius from a true genius! The book by Kliger et al. 3 is a good starting point and not insanely hard to understand. The paper by Jensen et al. 1 is rewarding, but definitely not easy reading.
There are many more papers, books, animations, and so on from which to learn. But be aware that notations are all over the place and reconciling equations and terminology can be very frustrating.
References:
H.P. Jensen, J.A. Schellman, T. Troxell, “Modulation Techniques in Polarization Spectroscopy”, Applied Spectroscopy 32 (1978) 192-200 (doi.org/10.1366/000370278774331567).
W.A. Shurcliff, Polarized Light, Harvard University Press,
Cambridge, MA, 1962. For example on archive.org.
Kliger, D. S.; Lewis, J. W.; Randall, C. E. Polarized Light in Optics
and Spectroscopy, 1st ed.; Academic Press: Boston, 1990. To borrow for example from archive.org.
EDIT: In response to the OP's comment, I am adding two of my tables.
Circular birefringence (optical activity)
$$
\begin{align}
[φ] &≡ \text{molar optical activity}~(\pu{deg/cm M}) \\
\mathrm{CB} &≡ 2π(n_- - n_+)l/λ ≡ 2[φ]cl × π/180
\end{align}
$$
Circular dichroism
$$
\begin{align}
[θ] &≡ \text{molar ellipticity}~(\pu{deg/cm M}) \\
\mathrm{CD} &≡ \ln 10 (ε_- - ε_+)cl/2 ≡ 2[θ]cl × π/180 \\
g &≡ \text{dissymetry ratio} ≡ Δε/ε ≡ ΔA/A \\
ε &≡ \text{mean molar absorptivity} = (ε_- + ε_+)/2
\end{align}
$$
Complex angle of rotation, $χ$
$$
\begin{align}
χ &≡ χ' + iχ'' ≡ \mathrm{CB}/2 + i\mathrm{CD}/2 = φ + iθ \\
Δε &= 4π[θ]/180\ln 10 ≅ 0.03032[θ]
\end{align}
$$
Definitions of symbols and expressions
$$
\begin{align}
\mathrm{RCP} &≡ \text{right circularly polarized light (‘+’ subscript)} \\
\mathrm{LCP} &≡ \text{left circularly polarized light (‘−’ subscript)} \\
ε_± &≡ \text{molar absorptivities for RCP and LCP}~(\pu{L/mol cm}) \\
ε &≡ (ε_+ + ε_-)/2 ≡ \text{mean molar absorptivity}~(\pu{L/mol cm}) \\
A_± &≡ \text{absorbances for RCP and LCP} \\
A &≡ (A_+ + A_-)/2 ≡ \text{mean absorbance (true)} \\
n_± &≡ \text{(real) indices of refraction for RCP and LCP} \\
c &≡ \text{concentration}~(\pu{M}) \\
l &≡ \text{path length}~(\pu{cm}) \\
λ &≡ \text{wavelength}~(\pu{cm}) = 10^{-7} × \text{wavelength}~(\pu{nm}) \\
n &≡ (n_+ + n_-)/2 ≡ \text{mean index of refraction} \\
A_l &≡ \ln 10 (A_+ + A_-)/2 ≡ \text{mean absorbance} \\
η &≡ 2πnl/λ ≡ \text{phase (radians)} \\
[θ] &≡ \text{molar ellipticity}~(\pu{degree/cm M}) \\
[φ] &≡ \text{molar optical activity}~(\pu{degree/cm M}) \\
\mathrm{CD} &≡ \ln 10 (ε_- - ε_+)cl/2 ≡ 2[θ]cl(π/180) = 2θ \\
\mathrm{CB} &≡ 2π(n_- - n_+)l/λ ≡ 2[φ]cl(π/180) = 2φ \\
a &≡ \mathrm{CD}^2 - \mathrm{CB}^2 + \mathrm{LD}^2 - \mathrm{LB}^2 + \mathrm{LD'}^2 - \mathrm{LB'}^2 \\
b &≡ 2\mathrm{CD} × \mathrm{CB} + 2\mathrm{LD} × \mathrm{LB} + 2\mathrm{LD'} × \mathrm{LB'} \\B &≡ \left[(a^2 + b^2)^{0.5} + a\right]^{0.5}/2\sqrt{2} \\
C &≡ \left[(a^2 + b^2)^{0.5} - a\right]^{0.5}/2\sqrt{2} \\
Q &≡ B + iC
\end{align}
$$
The expressions for a, b, B, C, and Q are used in the general anisotropic medium Jones optical calculus and Mueller optical calculus matrices. These two matrices are denoted by J(GAM) and M(GAM), respectively, and are given in the paper by Jensen et al.