How critical is the selection of the window function in STFTs?

I have a sum of periodic signals that I am trying to untangle using time-frequency analysis. I seem to get wildly different results depending on the window length and shape. This is a problem because I want to develop an automated, and hopefully sequential algorithm to do the job.

Answer

Window functions have an inherent tradeoff between two of their frequency-domain properties:

• 
  

  Main lobe width: Any tapered window function will cause some "smearing" in the frequency domain. This is visualized by the width of the center lobe in the window function's frequency response. The wider the main lobe, the more difficult it is to resolve two tones that are close in frequency (if they are closer to one another than the main lobe width, they will tend to smear together). So ideally, you would like to have a window function that has a very narrow main lobe.

  
  


• 
  

  Maximum sidelobe height: Many window functions have frequency responses that consist of a single main lobe surrounded by repeated sidelobes that decay at some window-specific rate. The height of these sidelobes can make it difficult to resolve two tones that are separated in frequency, but differ greatly in amplitude. So ideally, you would like to have a window function that has very low sidelobes.

  
  

The problem: if you decrease the main lobe width of a window function, the sidelobes will grow, and vice versa. So, you need to strike an application-specific balance when choosing a window, based upon the distances in frequency and amplitude that you expect between your signals of interest. Given specific parameters of your system, it's possible to choose a window that (hopefully) meets your requirements.

As far as choosing the length of your window (which is equivalent to choosing the length of the DFT), you're best served with making your observation as long as possible within the constraints that your application might impose (e.g. latency requirements, how long the signals of interest can be considered stationary, computational resources, etc.). Your ability to resolve in frequency is directly proportional to the observation length (measured in time, not necessarily based on the FFT length, which can be zero-padded with no improvement in frequency resolution).