Custodial parent?

i had custodial custody we were awarded child support in 2005 ive always let him see him when he wanted let him move in for 5 months because he lost another job had no where else to go he smoked pot in my house drank all the time no interaction with our son complete nightmare hes never filed for visitation never filed for custody nothing we told him of our relocating to tenn said nothing now hes claiming he wasnt informed sent our response to the judge and his lawyer they are asking the return of our son to florida and asking for full custody i have many witnesses to his drug use and witnesses to me telling him of us moving i informed the department of revenue of us leaving also so can he actually take my son from me and can a judge tell me to move back