Come to think of it you're right about the detection zone, I can get behind a ship pretty close. However, I think my proposed solution remains valid. A spherical collision detection zone makes ships jerk away from objects they would pass just fine when it is unnecessary. The 'spatial awareness' should be an elongated paraboloid facing forward, so the AI can begin evasive maneuvers sooner the more directly an object is in its path. If not directly in the AI's path, a ship should be able to get much closer than if it is. If an elipsoid is hard to manage because its difficult to define, I would suggest a somewhat elongated cylinder, perhaps narrowing at the far end. That is definitely not harder to figure than a sphere, and will give the AI the forward bias it needs. A sphere is defined by 3 polar coordinates; a cylinder by 2 polar and 1 cartesian coordinate, so I don't see problem with implementing the math.
My other proposal is to keep track of the distance between objects so the AI knows how fast a ship is closing with an object, and have it turn away if the closing speed is greater than some proportion of that ship's current speed. Then things the ship is headed directly for, it will avoid, but things it is skimming past will have low closing speed and won't alert the AI to jerk away. The AI already determines distances between objects, so this leverages information that is already available.
I think this would even make navigating around asteroids smoother, because once a ship is on a course that would take it around the asteroid, it won't tend to keep jerking away as it skims near it. Of course, if Egosoft has simplified asteroids to point objects for collision detection, this won't work very well. The ship will think it is going to skim past, then plow right into the edge of it. In this case I'd suggest making asteroid collision avoidance a special case with different rules, like maybe a much larger collision detection zone, or perhaps they could use its clipping boundary instead of simplifying it to a point.
I don't think the AI has to figure much at all for OOS, just keep updating the position of each ship according to its course and speed. If you do jump in, as others can tell you from experience, sudden collisions between things that were fine OOS aren't a rare event, so I don't think the AI is doing any collision avoidance on OOS sectors.
Further I'd make it a 2 step system, first determining if any collision avoidance awareness is even needed, by checking if the ships are within 5 km or so of each other. Then if they are, the actual spatial awareness field can be activated. So what I'm suggesting is a 5 km spherical detection zone to determine whether to apply the cylindrical detection zone. This would also be an ideal time to determine whether to use regular or asteroid collision detection zone, when two objects in proximity have been identified.
I'm not sure I agree with you on the unfeasibility of meeting the computational requirements. The AI only has to figure collision avoidance in sector, and only for ships within say 5 km of each other. I've seen pathfinding work in other games on thousands of units simultaneously with much slower hardware than what I have now. This is with avoiding terrain, each other, and still finding a reasonably efficient path to where I sent it, as opposed to mere collision avoidance in the X games. Collision avoidance is simply turning a ship away if another ship is detected in its spatial awareness field, it is much less computation than pathfinding. So I think the problem is they haven't found an efficient way to do it.
On the other hand, if it is more computational work than simply using a hemisphere, where the AI only need determine distance and whether an object is in front of or behind the plane perpendicular to the direction of travel of the ship, perhaps it isn't worth it to use a fancier collision detection method. However, considering they've nerfed ship speeds to account for this problem, which in turn greatly affects the game, I'd say there's a good case to be made for a smarter collision detection method.
Sins of a Solar Empire is a game where there are again thousands of ships to keep track of, and they do collision avoidance on ALL of them, even the ones in another sector in the game. They do not actually collide, so clipping isn't checked, but they do avoid each other pretty well. On top of THAT they are often in combat as well. Thousands. The game will eventually slow down some at this point, but it still manages at a playable speed. So I'm unimpressed with how hard mere collision avoidance on ships near enough to consider it in a single sector should be, computationally.