Doubt is to be uncertain about which of the two matters has occurred, and doubt is not taken notice of in matters of worship in three cases:
1- If it is just a self-delusion, having no reality, like devilish whisperings.
2- If it occurs very frequently to a person such that he does not perform any act of worship except that he is caused to doubt in it.
3- If it occurs after the completion of the acts of worship, then it is not taken account of, as long as he is not certain of it, in which case he will act upon what he is certain of. An example of this is that a person prays Zhuhr (Noon), and after finishing his prayer he doubts whether he prayed three or four rak`ahs (units of prayer). Then he takes no notice of this doubt unless he is certain that he only prayed three rak`ahs, in which case he should complete his prayer if it occurs a short while later, then he should give the tasleem (salutation to end the prayer), prostrate for forgetfulness and then perform the tasleem. If however, he does not remember until a long time has passed, then he must repeat the whole prayer afresh. As for doubt in other than these three cases, then it is taken account of.
Doubt in the prayer will be one of two kinds:
1-That one of the two matters is more weighty in his mind, so he will act upon what is more weighty to him, then he will complete his prayer based upon that, after which he should give the tasleem, prostrate for forgetfulness and finally give the tasleem. An example of this is if a person prays Zhuhr and doubts in a rak`ah (as to whether) it is the second or the third rak`ah. But the weightier case in his mind is that it is the third, so he makes it the third. So after he performs one further rak`ah, he gives the tasleem, prostrates for forgetfulness and then he gives the tasleem.
The proof for what is established in the hadith of ‘Abdullah ibn Mas`ud (may Allah be pleased with him) that the Prophet (peace be upon him) said, “If one of you doubts in his prayer, then let him strive to arrive at what is correct, and complete it upon that basis. Then let him give the tasleem and perform two prostrations.” (Al-Bukhari)
2- If neither of the two possibilities are more weightier in his mind, he should act upon what he is certain of, which will be the lesser of the two, and complete his prayer based upon that, and then prostrate for forgetfulness before giving the tasleem, and then give the tasleem.
An example of this is if a person is praying ‘Asr (Afternoon) and doubts in a rak`ah (as to) whether it is the second or the third, and neither its being the second nor its being the third is more weighty in his mind. So he makes it the second, performs the first tashahhud and two rak`ahs after it, and then prostrates for forgetfulness and gives tasleem.
The proof for this is what Muslim reports from Abu Sa`eed Al-Khudri (may Allah be pleased with him) that the Prophet (peace be upon him) said, “If one of you doubts in his prayer and does not know how many he has prayed- whether it is three or four – then let him cast away the doubt and build upon what he is certain of. Then let him perform two prostrations before giving the tasleem. So if he has prayed five, then they will make his prayer even for him, and if he has prayed the full four, then they will be a humiliation for Satan.”
And from the examples of doubt is a person who arrives whilst the Imam is in the bowing (ruku`). So he gives the initial takbeer whilst he is standing up straight, and then he performs the ruku`, and this will result in one of three cases:
1- That he is certain that he has reached the Imam whilst he was performing ruku`, before he raises up from it. So he has caught that rak`ah and recitation of Surat Al-Fatihah is not required from him in this case.
2- That he is certain that the Imam raised up from the ruku` before he reached him, then this rak`ah has escaped him.
3- That he doubts as to whether he caught up to the Imam whilst he was performing ruku` – such that he catches that ruku`, or whether the Imam raised up from the ruku` before he caught him – such that he will have missed that rak`ah. So if one of the two is more weighty in his mind, then he acts upon that and completes his prayer upon that basis and he gives the tasleem, performs prostration for forgetfulness, and then gives the salutation. This is unless it is the case that he did not miss anything from the prayer because no prostration will be due upon him in that case. But if neither of the two cases has more weight in his mind then he acts upon what is certain (which is that he has missed a rak`ah), so he completes his prayer upon that basis1 and he prostrates for forgetfulness before giving the tasleem, then he gives the tasleem.
Source: Quoted with slight modifications form Sheikh Muhammad ibn Salih Al-`Uthaimeen’s The Prostration of Forgetfulness.