If you cannot afford to do Hajj for any reason, it is not FARDH on you.
Your mulla is dead wrong.
'... those who have the circumstantial means to undertake the visit ...'
3:97 Therein are clear Messages (to achieve that coveted goal and recall) how Abraham once took his stand (against all divisions of humanity and was granted the leadership of all mankind 2:124-125). Those who enter the System symbolized thereby shall find inner peace and external security. Pilgrimage to this House is a duty all mankind owe to God, those who have the circumstantial means to undertake the visit. One who denies this command should know that God is absolutely Independent of all His creation. [Pilgrimage = Joining the Hajj Convention. Mankind can realize the blessings of unity by rallying around this Divinely appointed Center of Ideology. 22:25-28. The idolaters are forbidden only to ‘ya’mur’ or administer the Masjid for obvious ideological reasons. 9:17-18]