There is no short answer to this. Maybe the best way to put it was that Harry and Jimmy weren't only tough on Tony but on the team as a whole. They had reason to, this was a talented club that simply wasn't getting it done.
Remember in 1982 the Sox started out 20-10, but by mid July had dropped back to the .500 mark after a disasterous series in Milwaukee. Ron Scheuler, Tony's pitching coach was fired over it.
1983 started off so poorly that JR and EE actually brought in Bobby Winkles, a former Sox base coach to evaluate the entire situation. If Bobby had said a change was needed Tony was out the door according to EE.
Bobby however said in so many words, 'boys...don't worry about it. It's gonna turn around...' And history shows it did.
For a little more insight here is my interview with Jimmy Piersall:
In part II of my conversation with Tony, he gets into why the team started slowly, how he tried not to let the talk of firing impact him and something that I had long suspected (from comments told to me by others) that Tony was close to returning as manager after Gene Lamont got the boot.