All perfect praise be to Allah, the Lord of the worlds. I testify that there is none worthy of worship except Allah and that Muhammad is His slave and Messenger.
Scholars differed in opinion about such wordings as "I swear", which is devoid of whom one is swearing by; the majority of the scholars hold that it is an oath even if the speaker did not intend it as an oath, but the Shaafi'i School is of the view that it is not an oath even if the sayer intended an oath by it. The Maaliki School elaborated on the issue and said that it would be considered as an oath if he had intended after it "by Allah"; and if he did not intend it [by Allah], then it is not an oath. But in order to be on the safe side, it is better to consider it as an oath, especially if the speaker had intended to swear by Allah.
If we consider that this is an effective oath, then you are obliged to expiate for each time you break the oath, because your words "every time" entails repetition. In this case, the expiation is repeated with the repetition of breaking the oath.
Expiating every time is in order to be on the safe side and free yourself of your liability of any obligation, as we have already mentioned. However, if it is hard for you to do so, then we hope that there is no harm on you, Allah willing, to act according to the view of the Shaafi’i School that it is not an oath.
Some scholars stated that it is permissible to act according to concessions in order to avoid hardship when in need.
The Shaafi'i scholar As-Subki said in Al-Ibhaaj, "It is permissible for a person without knowledge to imitate a scholar and to take the concession of the scholars some times when dire need arises without following the concessions in every respect. It is from this perspective that it is correct to say that the difference of opinion (among scholars) is a mercy, because concessions are a mercy."
Allah knows best.