The compilation would probably happen either at install time or whenever the system switched MPI implementations, not every time the application ran. The most flexible way to write a morph layer (don't know if MorphMPI or PnMPI are written like this) is to write it in two layers. The top layer would provide a consistent MPI interface to the application, with its own representation for MPI objects and would be linked with the application before the application is distributed. The bottom layer would convert from the morph layer's internal handles to the native MPI's objects. This layer would need to be compiled for the local MPI implementation, probably as part of the application's installation process or by the sys-admins. It could be statically linked to the top layer during install or dynamically linked when the application runs. The former would be more efficient but would require a re-link of the application whenever the MPI implementation changed. The latter would be fully flexible, including the ability to use the same binary with multiple MPI implementations.... on diskless cluster nodes with no compiler installed? This is incompatible with the way lots of clusters are administered. And when it broke, it would probably be not very user-friendly to fix.
If we're talking about ABI-related issues then it would be the morph layer's responsibility to test their interactions with any particular MPI. The software vendor would need to test functionality-related stuff but that problem remains regardless of what we do with the ABI.It means the combination was never tested. Commercial vendors of MPI-using programs generally don't support something they haven't tested.