Chart of Electron Affinities
The Electron Affinity can be thought of as the "electrical advantage" given by adding an electron to an atom. So, if you have a halogen that gains an electron, it becomes more stable because now it has its octet, orbitals filled, etc. This is why the 1st EA is large.
But if you want to add an electron, it needs to go to a whole new energy level, which will not be very desirable in terms of the electronic structure of the atom. So the difference in electric stability or potential formed is nearly 0. In other words, the halide ion has "0" want for another electron, since that would require adding another energy shell.
Another way to think about it is like this: the 2nd EA of a halide is basically the same as the 1st Ionization energy of the alkali metal to which it is isoelectronic (since adding an electron releases as much energy as removing that electron will take). Since the 1st Ionization energy of a Group 1 metal is normally very small because the electron is so far away and shielded from the nucleus, so is the 1st electron affinity of a noble gas, and thus, the 2nd electron affinity of a halide.
Furthermore, now that the halogen has a negative charge, it experiences repulsion with the electron soon to be added. (Why does oxygen not like to be doubly anionized?)