You are right in most cases. Some pilots tend to stay with their aircraft until its too late, so making the decision early to eject is important. The man/seat performance envelope dictates where you can eject, unfortunately the aircraft's performance envelope may exceed the seat/man envelope. For example, the North American X-15 could fly Mach 6.7 (accomplished at 102,000 ft) and 67 miles altitude (flown at Mach 4.98). The X-15 seat was rated for Mach 4 at 120,000 feet, so the pilot would have to make the decision to elect to stay with the aircraft and ride the cockpit down until the seat was within its envelope or eject and get what you get.
This occurred when Bill Park flew the M-12 (SR-71 derivative) with D-21 drone on it back in 1966. The drone contacted the aircraft after launch causing the aircraft to break up. The pilots rode the tumbling aircraft down to a speed and altitude in which it was safer to eject.