The simple version:
Petroleum-based rubbers can be dissolved by petroleum jelly eating away at the mechanical integrity.
The slightly longer version:
Petroleum jelly incorporates its way into the rubber, changing the plasticity (ie how rubbery or stiff) the rubber is. Changes in plasticity change the mechanical strength and toughness such that the rubber is then outside the material property specifications of the product it is with/on/in, and hence fails.
Even longer version:
Most rubbers have small molecules inside them that help the molecular chain is the rubber move. These molecules are called plasticizers (and occasionally tackifiers). They are small molecules usually 200-500 MW which is less than the several hundred thousand of the rubber itself. Petrleum jelly is basically hydrocarbon of roughly the size of the tackifiers. The petroleum jelly molecules can 'wiggle' their way into the rubber, moving and adjusting how the existing plasticizers interact with the rubber. In some cases the plasticizers may be extracted out, also changing the interaction, fundamentally changing the energies involves in intermolecular motion which is then related to toughness and strength (many steps in that derivation omitted). In extreme cases in poor quality rubbers, some of the rubber molecules themselves can be dissolved by the petroleum jelly. Changes in toughness, plasticity and mechanical integrity are common when petroleum jelly interacts with rubbers, and hence it should be avoided as a lubricant.
As you have pointed out silicone oil is a costly replacement, but not unbearable more expensive for most jobs.