Your intuition is not wrong... but the atoms near the surface have to be free to move or exactly aligned, and must not covered by an adhered layer of air.
For example, by dissolving a salt in water and letting the water evaporate, you can watch crystals grow, as atoms (well, ions) find their place in the lattice. Read Crystals and Crystal Growing by Holden and Morrison, for example, download a free classic, such as Tutton's Crystals, or look online, such as Science Notes, How to Grow a Crystal.
Of course, there are crystals of metals and non-ionic compounds, too. Consider that only mildly-ionic water vapor condenses to make snowflake crystals, investigated by Bentley.
Simply holding pieces in your hand and hoping the crystal structure to align ignores theses issues:
The surface of the crystals would have to be atomically smooth, not jagged. Examine the surface of even a carefully cleaved crystal under an atomic force microscope and you'll find few smooth areas. The jagged peaks and valleys prevent the surfaces from meeting except at a few points.
As soon as a crystal is cleaved in air, the surface is covered with an adhered film. Germer states in The Study of Crystal Surfaces, "One of the most significant facts about real surfaces is that they quickly become dirty; a freshly cleaved crystal
soon adsorbs foreign atoms on its surface." The effect is like using waxed paper to keep dough from sticking to surfaces.
However, given smooth surfaces, they do stick together in the process of cold welding.