I’ll say one thing for GLaDOS: she’s an entity of comparatively modest ambitions. She’s not disassembling the solar system on the molecular level so that she can build megascale test chambers, or devouring civilizations to increase Aperture’s market share. She’s not building herself a matrioshka brain. All she wants are a few cubic kilometers of subterranean hellhole and a thousand or so people she can torture to death in the name of product testing.
Is that really such a terrible thing?
Throughout the game, before, during, and after Wheatley’s betrayal, she refers to him as Chell’s “friend” un-ironically, genuinely believing them to be friends, despite everything that happens. Of particular note is the moment when she asks Chell “is it all right if I kill him” in a manner that comes across as if she doesn’t expect Chell to actually say “yes”. Even though he’s just tried to kill her, GLaDOS honestly thinks Chell still cares for him. When she asks Chell if she likes revenge it again comes across as if she doesn’t expect Chell to hate him as much as she should. Then there’s the moment during the co-op campaign when GLaDOS tells the bots that their friendship is second only Chell and Wheatley’s. She admits this only after giving up on trying to turn them against each other. Not only does GLaDOS give the bots SECOND PLACE after FAILING to turn them on each other, but her whole motivation to try and drive them apart seems to stem from a genuine fear, rather than simple sadism. When she notices them getting closer she says “I’m not sure I like that” and after she concedes defeat she remarks that the bots are too stupid to understand the concept of betrayal. All along she was afraid that their bond would trump their fear of her, just like Chell and Wheatley’s did.
GLaDOS is a bit like Azula from Avatar; she rules through fear because she knows that self-preservation is usually the highest priority for any sentient being. So when someone comes along and challenges her despite being well aware of the standing threat of death to all who would defy her, it scares her because suddenly she doesn’t have complete control over the situation. And until Wheatley, nothing like that ever happened. The bots all knew their place and valued themselves more than any smelly humans. Then Wheatley came along and found SOMETHING, maybe not love, but nonetheless something more important to him than avoiding her wrath and GLaDOS had no way to deal with it other than death traps that he and the mute lunatic kept escaping. And that scared the crap out of her. Getting killed by Chell once was one thing. But with a bot on her side she not only escaped several traps that would’ve been impossible without Wheatley’s help, but navigated through the facility to her lair in record time, disabling her defenses along the way, and put her in a completely helpless position, something she never wants to experience again.
That’s why she ejected Wheatley into space instead of torturing him, or handing him over to Chell to punish. She firmly believed that whatever bond Wheatley had with Chell, if it was stronger than his fear of her, than it was strong enough to overcome his betrayal, and she couldn’t risk letting them reconcile and become an unstoppable, murderous, facility-destroying duo she can’t defend against.