Is it? You can make GPT-4 agentic, just run it in a loop with a goal and some actions to take (email someone, move robot, create note, lookup note, etc)
OpenAI's alignment carries over to this. Quick experiment to demonstrate:
The way OpenAI uses alignment to refer to GPT-4's behavior is misleading. Getting a model to mostly produce content you want and not produce content you don't want is very different than aligning a strongly agentic system. The latter has goals and can take autonomous actions