SWAGging here...but:

1) I'm assuming the LO is some form of DDS which is controlled by a PIC or similar embedded controller that outputs a serial data stream to the device. How does the supply voltage to the DDS look in terms of ripple and jitter?

2) Look at the output of the DDS with a scope. Is it within acceptable limits?

3) If the output appears slightly low in amplitude, what happens to the output waveform if you disconnect the DDS from the bandpass filter network and terminate it with a resistance equal to the input impedance of the filter?

Link a schematic to this post and let me examine it further, if you try the above and it sheds no light on the problem.