As we know, we have to apply large amount of energy to cause an inert atom lose electron, so in that case, shouldn't its electron affinity be negative?
Clarification is likely required for ionization energy versus electronic affinity. Although the trends of both across the periodic table have some similarity, they are not directly related to one another.
Ionization energy is the energy required to remove an electron (or more) from the element (gaining overall positive charge), while electron affinity is the energy difference (required or released) if an additional electron we're to bind to the element (resulting in an overall negative charge). Inherently then, ionization energy quantifies the difficulty of removing an electron already present, whereas electron affinity is the desire of the element to pick up more electrons; a noble element, for instance, is both "tight-fisted" (holding vehemently onto its electrons already present) and uninterested in further pairing/adding of electrons (is fully satisfied electronically), resulting in large ionization energies and small (e.g. zero or even positive) electron affinities.