All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad sallallaahu`alayhi wa sallam, is His Slave and Messenger.
Your pants don't become impure simply by suspicion that they have been affected with some drops of urine. You may pray with them, because the basic principle is that all things are pure unless proven impure with certainty. It is stated in Majmoo 'al-Fataawa by Sheykh Ibn 'Uthaymeen: “The basic principle is that things remain as they originally were, so if one doubts the impurity of something pure, then it is pure, or if one doubts the impurity of something impure, then it is impure, because the basic principle is that things remain as they were.” [End of Quote]
We warn you against whispers, because their harm is great. You should avoid them, and pay no attention to them, and that is the effective remedy for them.
Allah Knows best.