Due to the hacks that Apple uses in CNContactViewController, if we want to use it in a SwiftUI view the only way to have it display properly, is in a sheet. Putting it in a sheet introduces other issues, for example interactively dismissing the sheet is not possible by default because navigationBar.isTranslucent = true, which means there's nothing for the swipe down to grab onto. If you do that and then try to set setNavigationBarHidden(true, animated: false), it will mess up the contact photo and poster.
Another issue is if you try adding a Done button to the CNContactViewController it won't match the system buttons.
After many hours of trying different options I think the best solution is to leave the CNContactViewController the way it is, and place an overlay in the SwiftUI view with a Done button and a fake navigation bar. The fake navigation bar, gives the sheet a place to grab for interactively dismissing, as well as the styling for the button is much easier and matches almost exactly the style of the CNContactViewController built in Edit button.