In my fab experience, the answer is no, you are not going to inhibit deposition of silicon with some other material. You might well change the final grain size, but it will deposit. Furthermore, LPCVD generally has good sidewall coverage, so you can't use a lift-off mask either.
The OP asked for references, so lets dig in to those. First, it is somewhat difficult to find specific references to deposition rates on various substrates that can be broadly compared - reactor specifics, gas flow, temperatures, etc., all make it hard to compare experiment-to-experiment. But, we can pin down a few things.
First, going back to Andy Grove's (yes, Intel's Andy Grove) book "Physics and Technology of Semiconductor Devices" (John Wiley and Sons, New York NY, 1967) one finds no mention of polysilicon deposition. At that early stage in technology it was not used - metal gates were the way to go.
Polysilicon became a thing with poly-Si gates and the LOCOS (local oxidation of silicon) process. Here one might end up depositing poly over a wafer that included single crystal silicon (to make source-drain contacts), oxide (gate and field isolation), and silicon nitride (for the LOCOS process). Looking at cross sections from LOCOS devices (in various more modern text books - one I have is Stephen A. Campbell, "The Science and Engineering of Microelectronic Fabrication" (Oxford University Press, New York NY, 1996)), you see no variation in poly thickness across those different materials.
There is much more literature available around questions like polysilicon grain size vs initial surface roughness, annealing temperature, etc., such as Voutsas and Hatalis, "Surface Treatment Effect on the Grain Size and Surface Roughness of as-Deposited LPCVD Polysilicon Films", J. Electrochem. Soc. 140(1) 282-288 (1993).
Finally, a number of people have done work on the kinetics of LPCVD poly-Si growth, with one early reference being M.L. Hitchman et al., "Polysilicon growth kinetics in a low pressure chemical vapour deposition reactor", Thin Solid Films 59 231-247 (1979).
There are two takeaways from papers like this. First, under LPCVD reactor conditions, the mean free path of a silane molecule (or any other reactant) is quite long. This is, of course, highly desired to have uniform deposition across many wafers in, e.g., a tube furnace. The flip side is that the probability of a silane molecule reacting upon hitting a surface is quite low - they bounce around a lot. And, once a surface has any poly-Si on it, it all looks the same.
Second, the only way to not have poly-Si deposition is to never allow nucleation in the first place. Yet, point one above implies that lots of silane is hitting the surface, and if any one molecule manages to stick you now have a place to grow more. Further, all that seems to be needed to get some silane reactivity is the presence of hydrogen on the surface, and there will be lots of hydrogen around as silane decomposes elsewhere (plus hydrogen on surfaces is pretty much a given - it is by far the easiest surface termination on just about anything, particularly after a chemical process in an aqueous solution, like an acid clean).