As in geometry optimization, you are searching for stationary point on the potential energy surface (PES). Not for local minimum, but for saddle point, therefore in GAMESS, you specify RUNTYP=SADPOINT
. You would also need the correct (non-guess) Hessian matrix, which you can calculate separately with RUNTYP=HESSIAN
. In transition state (TS), you should have zero gradients (stationary point) and one imaginary eigenvalue of Hessian (the reaction coordinate in TS).
For working example, see e.g. SN2 transition state. In principle you need very good guess at TS geometry, which you obtain by scanning along some chosen coordinate. The point highest in energy should be reasonably similar to the TS, and you optimize to saddle point starting from there. Already at the guess geometry you must have one imaginary Hessian eigenvalue (resembling the reaction you are looking for, not methyl rotation somewhere else).
But beware, finding the transition states is one of the most difficult topics in computational chemistry and you are not guaranteed to find one, no matter how hard you try. One of the biggest problem is solvation, as you often wish to find reaction barrier in solvent (where majority of chemistry happens), but you can calculate only in vacuum. Just think of how carefully the workbench chemist finds the correct solvent for given reaction, in computer you have only very crude approximations to it (and most often just vacuum).
That said, it is great fun and rewarding to find the transition states. For best results, use the simplest theory available, as it is better to have smooth and well behaving PES, even though it is slightly wrong, so it is no harm to start with HF or BLYP (or B3LYP). To get some ideas, start with simple reactions, i.e. where no charged species are involved, so electrocyclic reactions are probably best choice, e.g. Diels-Alder.