Found the issue.
The profiler is not recording included libraries. In the magento/lib folder nothing is recorded even if you explicitly set it to profile.
It seems that we need to use an all-encompassing php profiler
solved What can delay “Time to First byte” by almost 2s?