Given a set of reactants, what aspects should I consider when predicting possible products?
You need to build a working knowledge of known reaction mechanisms that include the functional groups of your reactants. Put another way, (1) identify the functional groups present in your reactants and (2) check known reaction mechanisms that proceed by reaction of these functional groups.
Really, the only way to be good at this is to study and memorize organic reactions. There's a number of resources to help with this (e.g. online flashcards, and I really like Chemistry by Design, an app that you can use on a smartphone).
Sharpen up your general chem skills. One skill is recognizing redox reactions. Many organic reactions are redox. In problems where you have the reactants and products and are prompted to give the reagents you can recognize reductions and oxidations. For the problems where you have the reactant and conditions you can more easily recognize oxidizers and reducers. You can work on relative acidity and identify the most acidic proton in a molecule. You can recognize the different types of carbon in a molecule and remember which are more reactive to which conditions.
Also, keep a notebook where you put in 5-10 reactions a day and memorize them. You probably don't want to start today but you wish you started a month ago and a month from now you'll wish you started today. Just learning name reactions would be a good start.
Short answer: You can't.
Long answer: for simple substrates and simple reagents there is a big stock of reaction protocols with more-or-less known general mechanisms, that can give a good guess of resulting product. However, in case of complex substrates or tricky reagents (yet unknown) strange transformations may occur. They can get ad-hoc explanations, but.. that's not satisfactory enough. A really good book, considering key intermediates and most important reactions with mechanisms is March's Organic Chemistry (that's probably one of the best books on the subject).