The potential of an electrode in the absence or paucity of redox reactions at the interface is not predictable using the Nernst equation. This situation is known as an unpoised electrode and the electrode's measured potential is related to the formation of an electrical double layer at the interface, which behaves very much like a capacitor.
If you try to measure the potential of an electrode that won't easily oxidize or reduce in deionized water, due to the very low concentration of ions, establishing the double layer is slow and easily disrupted. You can think of it as having a large solution resistance in series with the double layer capacitance. When there are few ions, the resistance is very high. On that same note, how good the voltage measurement instrument is becomes important. Any current that a voltmeter draws will induce a large voltage at the electrode (as there is a large resistance to bringing ions to the interface, excess charge at the electrode is not easily balanced)—a voltmeter with a 10 nA input bias current may induce 100–200 mV of error in a very resistive solution, so it takes very good instruments to deal with these types of measurements.
This error is proportional to the solution resistance and even adding a tiny amount of ions to deionized water will dramatically reduce the resistance. I'm guessing that repeating the experiment with a supporting electrolyte will prevent this effect.
The other trouble here is that neither of your electrodes can be assumed to maintain a fixed potential in your solution. The potential of the silver electrode should vary according to the Nernst equation once you have an appreciable quantity of silver ions in solution, but the graphite electrode is not in equilibrium with any components of your solution and its potential is free to fluctuate as the solution composition changes. Without a supporting electrolyte, adding the silver nitrate may have a large effect on the potential of the graphite electrode.