All perfect praise be to Allaah, The Lord of the Worlds. I testify that there is none worthy of worship except Allaah, and that Muhammad, sallallaahu ‘alayhi wa sallam, is His Slave and Messenger.
First, you must make sure that your work is in accordance with the rules set by Islamic Law and free from what might jeopardize or oppose it. As the manager does not permit you to pray anything other than the Fardh prayer, then it is not permissible for you to delay your work to pray the Nawaafil, because fulfilling the work contract is Waajib (obligatory) and those other prayers are Naafilah (not obligatory), and when the Waajib conflicts with the Naafilah, precedence is given to the Waajib.
However, it is important that you clarify your situation to him and request permission for a short delay that does not take up more than a few minutes so you can perform the prayer with its Nawaafil. Also, try your best not to prolong the prayer so that you do not take up a lot of time.
If he is not content with this, then you have the choice to either adhere with these terms, leave the job or delay the Nawaafil until you go home, if that is possible.Allaah Knows best.