Of course there is a systematic approach to solving structure. The key is that it is an iterative process, and it is not uncommon to have to repeat the cycle several times. For complicated molecular structures (for example, alkaloids with C35+ frameworks) you might get 20 or more possible candidate structures that you have to work through very closely.
Of course, the method you use will depend on the class of compound, but for unknown organic molecules, I would start with a molecular mass and NMR data from 1H, COSY, HSQC, HMBC. Subsequent iterations or refinements might require NOESY, TOCSY and if there is enough material maybe even an ADEQUATE.
The general approach that I teach is as follows. The first step is to identify the molecular formula from mass spec data and calculate the double bond equivalence. Subsequently:
1. Environments
- From 1H and 13C data, determine number of carbon environments, and account for all carbon atoms, as these will represent the individual jigsaw pieces
- Determine proton environments attached to 13C - tally up CH3, CH2, CH groups.
- Tally up integrals, identify symmetry and equivalence.
- This should account for almost all of your C and H atoms
2. Chemical shifts
- Establish functional group types based on chemical shifts (aromatic, alkene etc).
- Look for possibility of heteroatom inclusion that ties in with the molecular formula
3. Couplings and spin systems
- From obvious 1D coupling and COSY data, start assembling spin system fragments
- From HMBC start assembling C–C connectivities
- Consider drawing out a correlation map to show connectivities, especially for complicated systems
- You should be reasonably confident about the spin system fragments leading into the next stage
4. Structure hypothesis
- Assemble spin system fragments into allowable possible structures, and test the structure against the data - that is rationalise the chemical shifts and couplings for all signals to see that they fit the model. If not, refine and try again.
A number of structure elucidation software packages exist, and they essentially base their method on building up a molecular correlation map, and then doing a chemical shift prediction for the proposed structure to give an overall confidence score. The sort of molecular map you might get looks something like the picture below. On the left are all the explained correlations, and on the right the incorrect correlations. What is important is look closely and the incorrect correlations, and rationalise them. In this case, all of the incorrect correlations would be allowable, and therefore we would have good confidence in this structure (which is understandable, as we know exactly what it is!). If we couldn't rationalise some of the correlations, we need to either reconsider our molecular structure, or re-analyse our data.