[Solved] How to rotate a vector in opengl?

Question

Yes OpenGL uses 4×4 uniform transform matrices internally. But the glRotate API uses 4 parameters instead of 3:

glMatrixMode(GL_MODELVIEW);
glRotatef(angle,x,y,z);

it will rotate selected matrix around point (0,0,0) and axis [(0,0,0),(x,y,z)] by angle angle [deg]. If you need to rotate around specific point (x0,y0,z0) then you should also translate:

glMatrixMode(GL_MODELVIEW);
glTranslatef(+x0,+y0,+z0);
glRotatef(angle,x,y,z);
glTranslatef(-x0,-y0,-z0);

This is old API however and while using modern GL you need to do the matrix stuff on your own (for example by using GLM) as there is no matrix stack anymore. GLM should have the same functionality as glRotate just find the function which mimics it (looks like glm::rotate is more or less the same). If not you can still do it on your own using Rodrigues rotation formula.

Now your examples make no sense to me:

(5,0,0) -> glm::rotate (0,1,0) -> (-5,0,0)

implies rotation around y axis by 180 degrees? well I can see the axis but I see no angle anywhere. The second (your desired API) is even more questionable:

 (4,0,0) -> wanted API -> (3,0,0)

vectors should have the same magnitude after rotation which is clearly not the case (unless you want to rotate around some point other than (0,0,0) which is also nowhere mentioned. Also after rotation usually you leak some of the magnitude to other axises your y,z are all zero that is true only in special cases (while rotation by multiples of 90 deg).

So clearly you forgot to mention vital info or do not know how rotation works.

Now what you mean by you want to rotate on X,Y,Z arrows? Want incremental rotations on key hits ? or have a GUI like arrows rendered in your scene and want to select them and rotate if they are drag?

[Edit1] new example

I want a API so I can rotate vec3(0,4,0) by 180 deg and result
will be vec3(3,0,0)

This is doable only if you are talking about points not vectors. So you need center of rotation and axis of rotation and angle.

// knowns
vec3 p0 = vec3(0,4,0);  // original point
vec3 p1 = vec3(3,0,0);  // wanted point
float angle = 180.0*(M_PI/180.0); // deg->rad
// needed for rotation
vec3 center = 0.5*(p0+p1); // vec3(1.5,2.0,0.0) mid point due to angle = 180 deg
vec3 axis = cross((p1-p0),vec3(0,0,1)); // any perpendicular vector to `p1-p0` if `p1-p0` is parallel to (0,0,1) then use `(0,1,0)` instead
// construct transform matrix
mat4 m =GLM::identity(); // unit matrix
m = GLM::translate(m,+center);
m = GLM::rotate(m,angle,axis);
m = GLM::translate(m,-center); // here m should be your rotation matrix
// use transform matrix
p1 = m*p0; // and finaly how to rotate any point p0 into p1 ... in OpenGL notation

I do not code in GLM so there might be some little differencies.

Accepted Answer