All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad, sallallaahu 'alayhi wa sallam, is His slave and Messenger.
If the place where that drop (of urine) fell has dried and no trace of the urine remains, then anything moist that comes into contact with it—such as bedding, clothing, or anything else—does not become impure, because the actual substance of the impurity has disappeared and only its ruling remains, and the ruling does not transfer. Additionally, the exact location of the drop is unknown, and therefore it is not possible to ascertain whether the sweat actually touched the precise impure spot.
In any case, what you mentioned about your trousers becoming impure does not justify leaving the prayer, nor even delaying it beyond its time. Changing the trousers is easy, and praying without trousers—if your garment sufficiently covers you—is also permissible. In fact, even if we assume something that is normally very unlikely in this era, that you cannot find any pure garment to pray in, then pray in the impure garment, because the condition of having pure clothing for prayer is tied to one’s ability to obtain a pure garment.
Allah knows best.