All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
This swearing is strictly forbidden, but you do not go out of the fold of Islam because of this swearing. This is because it appears that your intention is not to disbelieve, but rather to refrain from doing that thing. You have to offer an expiation for breaking an oath, and you must also repent to Allah sincerely and not return to this forbidden act.
For more benefit on making an oath that if one does not do something, he becomes a Kaafir, please refer to Fatwa 374413.
Allah Knows best.