The complex numbers are compared first by magnitude, then by phase angle (if there is a tie for the maximum magnitude.)
From help max
:
When X is complex, the maximum is computed using the magnitude
MAX(ABS(X)). In the case of equal magnitude elements, then the phase
angle MAX(ANGLE(X)) is used.
NaN's are ignored when computing the maximum. When all elements in X
are NaN's, then the first one is returned as the maximum.
solved How does matlab compare two complex numbers?