Chloride ion does not react with the water to any significant extent. To understand this, we must first accept that all chemical reactions are equilibriums. This means that all chemical reactions are reversible to some extent. Some reactions aren't very reversible at all (at least not in the short or medium term) - such as the combustion reaction in your car's gasoline engine. But if you give it a few million years, the carbon dioxide released by your car's combustion process will be fixed (biological term) by plants and these plants might eventually turn into fossil fuels again.
So we've established that all chemical reactions are equilibriums. Now we can consider the case of the chloride ion and why this reaction does not happen, as you say, or does not happen to a significant extent, as I say:
$\ce{Cl^- + H_3^+O -> HCl + HO^-}$
The reason is that if the above reaction happened to any significant extent, then this would make HCl a weak acid. Strong acids are defined to dissociate nearly completely in water. HCl is a strong acid by any account. Think of it as a waterfall. This reaction represents going down a waterfall (this reaction is large extent; heck, it's practically 100%):
$\ce{HCl + H_2O -> Cl^- + H_3^+O}$
So if the above reaction is the going down part of the water fall, its reverse reaction must represent the going up part of the waterfall. Going up the waterfall is hard no matter how you slice it; the only difference is whether you're trying to go up a small waterfall or Niagara falls. The reaction of hydrochloric acid with water is akin to going down Niagara falls - this is a favorable reaction. This is basically a one-way street.
Therefore, chloride ion can't affect solution pH; if chloride ion could somehow react again to form more hydrochloric acid then we'd violate the definition of a strong acid - nearly complete dissociation. This is what makes hydrochloric acid a strong acid!
And this is empirically true. Chloride ion can't even be protonated effectively by the strongest acid in a solution of hydrochloric acid, which is hydronium ion, so there is no way that you can expect chloride ion to be protonated by water (which is only about a trillion times weaker than hydronium ion as an acid). Thus, your solution of ammonium chloride has a pH controlled not by the chloride ion but by the ammonium ion.