In my foray into detections, I am trying to understand how best to generate receiver operating characteristic (ROC) curves, and some questions have come up.
In my studies, I have found out that ROC curve generation is simply the plot of true-positive rate (TPR) versus false-positive rate (FPR). Naturally one wants TPR to be 1 and FPR to be 0. I also understand that each point on the ROC curve, (a point with one TPR and FPR coordinate) corresponds to one set of detector 'settings' (threshold(s)). With one setting fixed, we will compare all positive 'detects' to all actual number of true positives, and true negatives, and thus yield TPR and FPR. My questions are as follows:
It seems to me that we must have a labelled data set(s) in order to do this to begin with. That is, we have to manually or otherwise, go through all the data we have, and label them as 'yes' or 'no'. (I realize this might seem obvious but I want to confirm). Is this true?
Somewhat related to the above - if in order to generate a true ROC curve that properly captures the detector performance we have to have 'no' data, then it seems to me that we have to have 'no' data that captures almost every possibility that exists in the universe. Clearly this is not feasible, but then how can one claim with any good faith that a detector truly has a certain false alarm rate? This is what confuses me the most.
Is it unheard of for detectors to have ROC curves that almost never false alarm, but that have normal looking variance on the y-axis? Put another way: Is it unheard of for detectors to have very little variance on the false-positive axis (x-axis) but a lot more variance on the true-positive axis (y-axis), for a fixed SNR?
I think that just about summarizes my questions, thanks.
EDIT Number 1:
I do not think I understand the feedback.
Theoretical aspect:
How can I statistically 'model' every single possibility in order for me to theoretically characterize it to begin with? For example, lets say that I have a detector that detects the presence of a signal of 10 Hz. The input can be anything under the sun. Maybe its 10Hz in noise. Maybe its 11Hz and no noise. Maybe its 10.5Hz on top of a linear chirp, or a bird song. Maybe its brown noise overlayed with human voice, overlayed with a 60Hz power supply. How does one model that, or any of the myriad combinations therein?
I do not know if I can get the theoretical performance of my detector, because quite simply I do not see how thats possible for realistic scenarios where anything can be your input! This goes back to Point#2, whereby, how can one possibly know of every possible input into the detector? I could be wrong on this so please show me how.
Practical aspect:
Therefore in this sense, I suppose my question is: I already have a detector in place. It takes an input vector, and spits out a decision, 'yes' or 'no'. I understand that I have to label my data sets in order to come up with true positive and false positive rates. In this case it seems to me that the only possibility is to come up with a data set that represents just about anything possible under the sun, and test that against your detector.
What am I not getting here?
Answer
Question 1) Yes, you must know what the right answer is before-hand in order to judge the receiver's performance. I am doing this right now on a project. I generate data, scramble it, modulate it, add noise and carrier offset, etc., then I give it to the receiver. The bits I started out with are my "yes/no" labels.
Question 2) All such measurements are statistical in nature. Noise is statistical. Jitter is statistical. A priori channel conditions are statistical. Of course the ROC curve is also statistical. How could it be otherwise?
Question 3) I don't quite understand this question. A picture would help.
EDIT: Receivers are not tested in every set of conditions under the sun. First, the designers rely on everyone obeying the FCC requirements for a given band, so in a cellular band they assume that there won't be any TV stations, or radar chirps, and that all other cellular signals will be at specified frequencies and power levels, with specified maximum carrier offsets.
The designers then model "typical" environments (i.e. channels). That implies certain transmission losses due to free-space path loss, antenna gains, multi-path, etc. They then put together various typical and worst-case scenarios and test it under those conditions. For example, a worst case interference scenario would be having adjacent signals at the maximum allowed power.
If the performance is as expected for those selected conditions, the designers assume, through "interpolation", that it will perform acceptably in the other non-tested environments. Conditions can always be worse than the worst-case test scenario of course. That is when the phone drops the call. :-)
No comments:
Post a Comment