Deciding which group twists out of the plane when ortho effect takes place

Question

I'm going to take a few stills from 3D conformers of the following compounds available on PubChem in which ortho effect is applicable to show what I'm referring to.

2-Nitrobenzoic acid:

N,N-Dimethyl-2-nitroaniline:

2'-Aminoacetophenone:

In the compounds above due to steric repulsions between the groups present, one group needs to be twisted out of the plane of the benzene ring. My question is, how do we theoretically explain which group shifts out of the plane, and which doesn't?

To my understanding, two things would effect this; one would be the extra stabilisation/destabilisation of a group due to not being in resonance any longer once it moves out of the plane of the benzene ring, and the second would be the "bulkiness" of the group itself, which would energetically be measured as the group's A-Value.

For the above three compounds in the same order, from the perspective of A-Values (the text in brackets mentions what actually happens):

1. ($\ce{COOH}$ twists out of the plane) The A-Values are $\ce{COOH} = 1.2, \ce{NO2} = 1.0$, so it makes sense for $\ce{COOH}$ to stay in the plane of the ring.

2. ($\ce{NMe2}$ twists out of the plane) A-Values $\ce{NMe2} = 2.1, \ce{NO2} = 1.0$ so it makes sense for $\ce{NMe2}$ to stay in the plane.

3. ($\ce{COMe}$ slightly twists out of the plane) A-Values $\ce{NH2} = 1.6, \ce{COMe} = 1.17$ so it makes sense for $\ce{NH2}$ to stay in the plane.

(data from here and here)

I understand that A-Values are not the only thing which we need to consider, since resonance plays a part (maybe even other stuff?), but I haven't been able to figure out convincing reasoning as to why in the above examples the group which shifts out of the plane does so. Does the volume occupied by the groups play a crucial part as well, since so far I am yet to come across a compound in which the smaller group (by volume) shifts out of the plane? Are A-Values even relevant to such a scenario?

(I'm looking for a list of factors which need to be considered, whether this can even be explained/predicted theoretically, and/or explanations for the any of the above compounds. I understand that this could be slightly lengthy, so I'd be more than happy for answers which address any one of these, but I feel talking about the three compounds was necessary to explain the question properly, and hence the somewhat broad question.)

Answer 1

I am not sure the 3D conformation stills as reported on PubChem are all that accurate. Although crystal structures are sometimes referenced for specific molecules, some of the displays are in fact models, some of which do not make sense to me.

Here are some observations regarding the three molecules you noted.

2-Nitrobenzoic acid
The PubChem model shows both oxygen atoms of the nitro- group to be precisely coplanar with the phenyl ring. In the crystal structure, these oxygen atoms have a dihedral angle of $125.5^\circ$ and $129.1^\circ$ with respect to the aryl, i.e. away from coplanarity.

The PubChem model shows both oxygen atoms of the carboxylic acid to be precisely normal to the phenyl ring. In the crystal structure, these oxygen atoms have a dihedral angle of $152.9^\circ$ and $162.7^\circ$ with respect to the aryl, i.e. closer to coplanarity.

Pubchem	Crystal structure entry CCSJWHN

N,N-Dimethyl-2-nitroaniline
I could not find the crystal structure of this molecule. However, I’m not sure it makes sense that the N,N-dimethyl amine adopt an $\ce{sp^2}$ character at the nitrogen center, at least as far as the Pubchem model representation is concerned. In such a pose, there can’t be extended resonance from the phenyl to amine as obtained with unsubstituted aniline. Either the methyls are near coplanar with the aryl to allow the pi orbitals to enter resonance or the amine is in $\ce{sp^3}$.

2’-Aminoacetophenone
I could not find the crystal structure of this molecule. What I found was the 4,5-dimethyl variant as 1-(2-amino-4,5-dimethylphenyl)ethan-1-one.

The 2’-Aminoacetophenone PubChem model shows the acetone oxygen to be precisely $30.0^\circ$ out of the aniline plane. In contrast, the crystal structure of 1-(2-amino-4,5-dimethylphenyl)ethan-1-one indicates that oxygen to be precisely in plane with the aniline. This coplanarity may not be surprising in this case as there is an intramolecular hydrogen bond between the two substituents.

Pubchem	Crystal structure entry 1825318 (BEXTUW)
2'-Aminoacetophenone	1-(2-amino-4,5-dimethylphenyl)ethan-1-one

Original Post:

To my understanding, two things would effect this; one would be the extra stabilisation/destabilisation of a group due to not being in resonance any longer once it moves out of the plane of the benzene ring, and the second would be the "bulkiness" of the group itself, which would energetically be measured as the group's A-Value.

I think steric hindrance, you noted as “bulkiness” and estimated with an A-Values is the primary factor. There is little room for compromise on this one as two atoms can’t occupy the same space. The second factor is resonance as you noted. Another influencing factor is the presence of intramolecular interactions, such as a hydrogen bond. Solvation and pH are other influential factors to the ortho effect.