What type of voltmeter are you using? to verify the answer to my question, you need to determine 1) the precision and 2) the detection limit of the instrument .
The precision is a measure of the error in a typical measurement. The smaller the standard deviation in repeat measurements, the higher the precision. To get a guess at the precision, prepare a system that should generate a known potential difference in the same range as your current measurements (like 100 mV). If the variation in that value is similar to what you are seeing, then you are recording the noise.
The detection limit is the lowest value your instrument can reliably measure. Generally, you cannot reliably measure values smaller than the twice the precision of your instrument. If you determine a standard deviation of 50 mV when determining precision, then 100 mV is your the detection limit. If this is true, you are recording the noise.
The precision and detection limit of an instrument depend on quality (price), but also on purpose. A voltmeter designed to be used by an electrician will have high precision in the range 1-200 V, but low precision below 1 V (because that range is not as important). A sensitive voltmeter used for electrochemical measurements may have high precision between 1 uV and 1 V, and behave miserably outside that range.