The babysitter allegedly responded that she was “too tired to carry him”