Based on a quite old reference 1 (which I'm using because it's available free by open access), peptide bond formation at 25 C is unfavorable only because of a large enthalpy change, on the order of 1.5 kcal/mol (6.3 kJ/mol). The entropy change is actually favorable, with $T\Delta S$ being about 1 kcal/mol (4kJ/mol), so the net free energy change is ~ + 500 kcal/mol (approx. +2 kJ/mol).
The reason for both values being what they are is because the free acid and amine groups are ionized at neutral pH. Thus, the reaction can be represented as
$\ce{R-CO2- + ^+H3N-R -> R-CONH-R + H2O}$
This can be broken into two steps conceptually:
$\ce{R-CO2- + ^+H3N-R -> R-CO2H + H2N-R }$
$\ce{R-CO2H + H2N-R -> R-CONH-R + H2O}$
As we know, proton transfer from an acid to a base typically involves a large negative enthalpy change, so reversing that to neutralize both has a positive enthalpy change. Since the products remain neutral, we never get that energy back. And the bond energies of a neutral carboxylic acid and amine are not much different from an amide and water, so the neutralization dominates.
For the entropy change, we start with two molecules and end with two, and the degrees of freedom are similar. The authors of the cited paper propose that the positive entropy change is also related to the loss of ionization. They believe that the more structured hydration shell of the ionic species vs neutral is sufficient to make the free amino acids lower in entropy than the neutral products, making bond formation entropically favorable, but not enough to overcome the unfavorable enthalpy change.
Later paper have further shown that the amide formation reaction becomes favorable around 60 C because the favorable entropy change finally overcomes the enthalpy change at that temp.
1 Dobry, A., Fruton, J.S. and Sturtevant, J.M. (1952) J. Biol. Chem. 195, 149–154.
(Available by open access here)