There are a number of factors you have to take care of while deciding nucleophilicity. I will try to cover all the important ones.
First we need to clear something, polarizability is the most important factor only if the nucleophile is electrically neutral (uncharged)$^{[1]}$. Of course, this is not hard and fast, but in most cases we generally deal with, this is true. For example, triphenylphosphine is a better nucleophile than triphenylamine, because the former's central atom has a more polarizable lone pair.
In our examples, the ions are charged, so we will put aside polarizability and look at the other factors. For charged species, size of the anion should be small in order for it to be a better nucleophile. Why? Two reasons:
- One, the same charge packed into a small volume is attracted more strongly by the electron-deficient area, and it can reach quite close to the positive region. This means that potential energy of the system will fall more, thus increasing stability of the product.
- Two, the central atom being small, its overlapping orbital can overlap more effectively with the empty orbital of the electrophile, since the orbital will be smaller in size, and hence less diffuse.
The last factor that remains is the solvent. In fact, the order is same for both the nucleophiles in both the kinds of solvents (protic and aprotic), so I will not go into the analysis of solvent effects. Just a comment, here the solvent effect seems unimportant, seeing that bromide ion is quite behind $\ce{SH^-}$ in the order of nucleophilicity in protic solvents (my source is Solomons and Fryhle and a website mentioned later).
Note: Electronegativity is not important here either, even though it would produce the same order. We generally use electronegativity only to compare nucleophiles with central atoms from the same period.
So ultimately we see that $\ce{SH^-}$ is more nucleophilic than $\ce{Br^-}$.
Most of what I said can be found in the textbook Solomons and Fryhle's Organic Chemistry, and the following link:
Here are some more useful links:
- http://chem.libretexts.org/Core/Organic_Chemistry/Alkyl_Halides/Properties_of_Alkyl_Halides/Introduction_to_Alkyl_Halides/Leaving_groups
- http://chem.libretexts.org/Core/Organic_Chemistry/Alkyl_Halides/Reactivity_of_Alkyl_Halides/Alkyl_Halide_Reactions/Substitution_and_Elimination_Reactions_of_Alkyl_Halides/SN1_Substitution_Reactions/Nucleophilicity_and_Solvent_Effects
- http://chem.libretexts.org/Core/Organic_Chemistry/Reactions/Substitution_Reactions/SN2/Substrate