Part I
The CIP rules in fact are used to determine the priority. However, a different notation system has been developed for inorganic nomenclature of coordination compounds.
For the square planar geometry, SP-4-x descriptor (an instance of ‘polyhedral symbol’) is used, where x is a CIP-based priority number of the ligand in trans (opposite) position relative to the ligand with the highest priority number (1).
In $\ce{[Mabcd]}$ complexes (ligand priority a > b > c > d, priority numbers 1, 2, 3, 4, respectively), for the 3 possible isomers, the descriptors are (forgive me the use of left/right superscript/subscript typography to depict square planar geometry, for the sake of simplicity):
- $\ce{^\style{font-weight:bold}{a}_cM^b_\style{font-weight:bold}{d}}$ SP-4-4
- $\ce{^\style{font-weight:bold}{a}_dM^b_\style{font-weight:bold}{c}}$ SP-4-3
- $\ce{^\style{font-weight:bold}{a}_dM^c_\style{font-weight:bold}{b}}$ SP-4-2
For $\ce{[Ma2b2]}$ (priority numbers 1, 1, 2, 2) and $\ce{[Ma2bc]}$ (priority numbers 1, 1, 2, 3) complexes, where only 2 isomers are possible, the cis/trans notation can be used as well. The E/Z notation from organic chemistry is not used at all for coordination compounds central atom configuration.
Examples:
$\ce{^{\style{color:grey}{(1)}{ }I}_{\style{color:grey}{(3)}{ }Cl}Pt^{Br{ }\style{color:grey}{(2)}}_{F{ }\style{color:grey}{(4)}}}$
(SP-4-4)-bromidochloridofluoridoiodidoplatinate(2−)
(hypothetical)
$\ce{^{\style{color:grey}{(1)}{ }Cl}_{\style{color:grey}{(2)}{ }NH_3}Pt^{NH_3{ }\style{color:grey}{(2)}}_{Cl{ }\style{color:grey}{(1)}}}$
(SP-4-1)-diamminedichloroplatinum(II)
trans-diamminedichloroplatinum(II)
$\ce{^{\style{color:grey}{(1)}{ }Cl}_{\style{color:grey}{(1)}{ }Cl}Pt^{NH_3{ }\style{color:grey}{(2)}}_{NH_3{ }\style{color:grey}{(2)}}}$
(SP-4-2)-diamminedichloroplatinum(II)
cis-diamminedichloroplatinum(II)
cisplatin
$\ce{^{\style{color:grey}{(1)}{ }Cl}_{\style{color:grey}{(2)}{ }Ph_{3}P}Ir^{PPh_{3}{ }\style{color:grey}{(2)}}_{CO{ }\style{color:grey}{(3)}}}$
(SP-4-3)-carbonylchloridobis(triphenylphosphane)iridium(I)
trans-carbonylchloridobis(triphenylphosphane)iridium(I)
Vaska's complex (a different IUPAC name descriptor claimed there is disputed)
(References: see Loong's comment below the question.)
Part II
Now, your question is self-contradictory. You would like to use E/Z (two) descriptors for three different isomers.
Maybe you wanted to ask something like
Why are there 3 geometrical isomers of square planar [Mabcd] complexes, but only 2 isomers of tetrasubstituted ethene C2abcd?
(Note the latter statement is not true)
It's because for $\style{font-weight:bold}{\ce{[Mabcd]}}$, the four different ligands are at the vertices (corners) of a square. Which can be abstracted as a ‘closed loop’, i.e. a cycle graph. The combinatorics can be abstracted to free circular permutations, $P'_{n}={1 \over 2} (n-1)!$,
for $n=4$: $P'_{4}=3$.
For $\style{font-weight:bold}{\ce{C2abcd}}$, the situation is somewhat different. This is a combination of positional isomers $\ce{xyC=Czw}$, where the set $\{x,y\}$ is $\{\text{a}, \text{combination}_2(\{\text{b},\text{c},\text{d}\})\}$, and $\{z,w\}$ is a complement. There are (concidentally?) 3 of them (${n \choose k} = {{n!} \over {k! (n-k)!}}, {3 \choose 2} = 3$):
- $\ce{abC=Ccd}$
- $\ce{acC=Cbd}$
- $\ce{adC=Cbc}$
(Here the different sortings of substituents at each carbon does not matter.)
However, the isomerism one is talking about when using the E/Z descriptors applies to geometrical isomers of a single positional isomer (take e.g. $\ce{abC=Ccd}$). Now, the sorting of substituents at each carbon does matter. What makes the isomers different is matching vs non-matching sort order at each carbon:
- $\ce{abC=Ccd}$ ≡ $\ce{baC=Cdc}$ (ascending-ascending, descending-descending – matching)
- $\ce{abC=Cdc}$ ≡ $\ce{baC=Ccd}$ (ascending-descending, descending-ascending – non‑matching)
which are 2 possibilities, and correspond to E and Z configurations, respectively (provided that the lexicographical order corresponds with the CIP priority order).
So, the total number of C2abcd isomers is 6 (3 positional isomers, each has 2 geometrical isomers). One think of the alkene as an object similar to the square planar metal complex, where the C=C can be arranged in the center in two orientations, horizontal and vertical, hence 2× as many possibilities than ${\ce{[Mabcd]}}$.
(I hope exactly zero mathematicians and chemists were harmed during the reading.)