To quote chemguide:
The first ionisation energy is the energy required to remove the most loosely held electron from one mole of gaseous atoms to produce 1 mole of gaseous ions each with a charge of 1+.
That means, the ionization energy of fluorine is the energy of the following reaction:
$$\ce{F -> F+ + e-}$$
The ionization energy of chlorine is the energy of the following reaction:
$$\ce{Cl -> Cl+ + e-}$$
Since the most loosely held electron in fluorine is closer to the nucleus than chlorine, it takes more energy to remove the electron from fluorine than in chlorine.
The ionization energy of chloride ion ($\ce{Cl-}$) is the energy of the following reaction:
$$\ce{Cl- -> Cl + e-}$$
The electron affinity of $\ce{Cl}$ is the energy of the following reaction:
$$\ce{Cl + e- -> Cl-}$$
As a result, their energies are equal in magnitude but opposite in sign.
Why does chlorine have a higher electron affinity than fluorine?
To quote this answer:
Fluorine, though higher than chlorine in the periodic table, has a very small atomic size. This makes the fluoride anion so formed unstable (highly reactive) due to a very high charge/mass ratio. Also, fluorine has no d-orbitals, which limits its atomic size. As a result, fluorine has an electron affinity less than that of chlorine.
As can be seen in the second photo, this effect seems to be true for Period 2 elements.
(source: lardbucket.org)
(source: lardbucket.org)