I am certain that the reason involves intermolecular forces, but since both $Xe$ and $Ne$ are noble and non-polar gases, shouldn't these forces have a much smaller effect or negligible on boiling point?
The main interaction between noble gases is Dispersion. (inert gas that's where the name comes from) Dispersion is larger in heavier atoms. (Considerable larger, 18+18+8 more electrons difference is massive)
an example: When Interaction energy passes zero point and thermal energy you can have the solid phase. This is why you will not have solid helium.