They have nothing better to do and are bored with their lives...

... they probably wish they'd been at home for their children too... but weren't and are jealous of you

Only you will know if you're doing the right thing when you've given yourself time to settle in and enjoy the job! It's like any job - you get teething problems... but at least you don't have to deal with the office letch and the nasty boss LOL