Essentially you need to understand that the silica gel packet is just $\ce{SiO2}$. These packets are used to absorb moisture (water vapor) to protect goods from spoiling.
To be honest with you here, nothing ever happens when you eat $\ce{SiO2}$. You consume it all the time from food and water. But, the reason these packets are so dangerous is because they might contain impurities such as $\ce{CoCl2}$ (blue color) and $methyl~violet$. These colored products are used to indicate moisture. These products are carcinogenic and are mutatic poisons. In most cases, it will not be fatal immediately, but given time, yes.
Generally, if the silica gel is colored, it is highly likely that they might be very dangerous. So don't eat them. :)
On top of this the silica beads tend to fracture so you have essentially consumed broken glass which will be quite painful as it tears up your digestive tract causing you internal bleeding.
Just to add a couple references: 1, 2, 3.
I don't know how far the the second and third ones are true but the first one is the most accurate.