Polymer properties in water often relate to the relative strength of polymer-polymer interactions versus polymer-water interactions. For example, the solubility of amylopectin > amylose > cellulose is attributed to how strongly molecules of each of these polymers interact with water molecules compared to other molecules of themselves.
In the paper making process, the cellulose fibers of paper dry out and interactions with water tend to get replaced by interactions with nearby polymer molecules. The resulting structure is strong because of polymer-polymer interactions as well as purely physical effects (e.g., "matting" of fibers, and surprisingly strong surface-surface interactions), all of which is enhanced by the compression and tension forces placed on the paper during its manufacture (google it to see photos).
When paper is later made wet, the entrance of water eventually disrupts these interactions, leading to local regions of what you might think of as micro solubility. Then, when the paper dries out again as you describe, it probably does so without any compression/tension forces on the paper, thereby reducing the extent to which the matting of the fibers and chemical interactions between polymers takes place, and so it crumbles because of those micro regions of "solubility".
(Disclaimer: The above is just rationalizing off the top of my head. But since it is based on expertise as a PhD Chemist for nearly three decades, I'd give it a 70+% chance of being accurate ;)
The real science begins when you test the above ideas: For example, one test would be drying out paper under different degrees of compression and/or for different time periods and then testing the crumbling behavior. Another might be to wet the paper while it's under different degrees of compression. But the best test of the above ideas would be if there were some way to assay the extent of polymer-polymer versus polymer-water interactions -- no doubt such interactions result in very different vibrational modes. So there, I've outlined a Ph.D. thesis for you.