Sometimes a molecular formula is written using groups, for instance $\ce{Al2(SO4)3}$.
Are there examples of formulas with nested groups, such as $\ce{ABC[MNO2(XYZ)4]3}$ or is this avoided?
Too long for a comment: Chemical formulas for polymers can utilize nested enclosing glyphs but, from what I've seen, the accepted format is to use parentheses for the inner glyphs, and square brackets for the outer glyphs, rather than nested parentheses.
An example would be polystyrene, whose monomers look like this (source: Wikipedia: Polystyrene):
Given this, the formula for polystyrene could be given as:
$$\ce{CH3–(C8H8)_{n}\! -CH3}$$
But some might prefer to instead give it, more informatively, as the following, in which case the nested enclosing glphys make an appearance:
$$\ce{CH3–[CH2-CH(C6H5)]_n –CH3}$$
[See William Reusch's polymers website (MSU), Section 2. Writing Formulas for Polymeric Macromolecules , which provides many such examples.]
Using parentheses inside square brackets is also the convention given here for complex ions: Wikipedia: Chemical formula
Since you're trying to write a program, you might find something here to be of use:
Stack Overflow: Can nested parentheticals be parsed in chemical formulae?
The closest I could find is mixed cation-anion salt:
Also make sure to check double salts and this question.
References
HCo(CO)3(P(OPh)3)
that could qualify? Or are nested groups written with square brackets for the outer group and parentheses for the inner one? $\endgroup$