[Solved] GCC – modify where execution continues after function return [closed]

Question

After giving it some extra thought, it is not as simple as it initially appeared. There is one thing that prevents it from working – the code of functions is not context aware – there is no way of knowing the frame it was invoked in, and that has two implications:

1 – modifying the instruction pointer if not portable is easy enough, as every implementation defines a consistent place for it, it is usually the first thing on the stack, however modifying its value to skip the escape trap will also skip the code which restores the previous frame state, since that code is there, not in the current frame – it can’t perform state restoration since it has no information of it, the remedy for that, if an extra check and jump are to be omitted is to duplicate the state restoring code in both locations, unfortunately, this can only be done in assembly

2 – the amount of instructions which need to be skipped is unknown too, and depends on which is the previous stack frame, depending on the number of locals that need destruction it will vary, it will not be a uniform value, the remedy for that would be to push both the error and success instruction pointers on the stack when the function is invoked, so it can restore one or the other depending on whether an error occurs or not. Unfortunately, that too can only be done in assembly.

Seems that such a scheme can only be implemented on level compiler, demanding its own calling convention, which pushes two return locations and inserts state restoration code at both. And the potential savings from this approach hardly merit the effort to write a compiler.

Accepted Answer

After giving it some extra thought, it is not as simple as it initially appeared. There is one thing that prevents it from working – the code of functions is not context aware – there is no way of knowing the frame it was invoked in, and that has two implications:

1 – modifying the instruction pointer if not portable is easy enough, as every implementation defines a consistent place for it, it is usually the first thing on the stack, however modifying its value to skip the escape trap will also skip the code which restores the previous frame state, since that code is there, not in the current frame – it can’t perform state restoration since it has no information of it, the remedy for that, if an extra check and jump are to be omitted is to duplicate the state restoring code in both locations, unfortunately, this can only be done in assembly

2 – the amount of instructions which need to be skipped is unknown too, and depends on which is the previous stack frame, depending on the number of locals that need destruction it will vary, it will not be a uniform value, the remedy for that would be to push both the error and success instruction pointers on the stack when the function is invoked, so it can restore one or the other depending on whether an error occurs or not. Unfortunately, that too can only be done in assembly.

Seems that such a scheme can only be implemented on level compiler, demanding its own calling convention, which pushes two return locations and inserts state restoration code at both. And the potential savings from this approach hardly merit the effort to write a compiler.