Praise be to Allah, the Lord of the Worlds; and may His blessings and peace be upon our Prophet Muhammad and upon all his Family and Companions.
If you have not performed your obligatory Hajj (as an adult, i.e. when Hajj becomes obligatory on you), you have to make Hajj with this money that you have, because Hajj is an obligation that becomes obligatory as soon as the person gets enough means to cover its costs, according to the majority of scholars, and this is the preponderant opinion.
If you have already performed your obligatory Hajj, then we advise you to try and save the money that you need, and buy a new house in another area, free of all that you have mentioned. There is no doubt that a person is likely to be influenced by the environment in which he lives, and by the people he mixes with, like neighbours. If he remains safe from that; his wife and children may be affected and influenced.
Your wife does not have to perform Hajj unless she can afford it (by herself). If you are asking whether it is an obligation on you to afford the money for your wife to make Hajj, the answer is no.
However, if you do that, it is something good and you will get the reward, Insha Allah. Children have to perform Hajj when they reach the age of puberty. If a child performs Hajj before the age of puberty and his Hajj is correct, he will get the reward, Insha Allah, but he still has to perform the obligatory Hajj as an adult.
Allah knows best.