No you can’t test it in simulator.
You get reach of prompt enter apple ID or Password for buying product, but its remaining process not allow in to simulatore that’s why you got Prompt of Transaction failed error.
You must test it In Device with relevant Provisioning code sign with your Test Apple id or password.
solved Can test In App purchase in simultor? [closed]