We can sometimes call ProxyDestroyed() on a proxy that never had a wrapper set up so we should just bail out of ProxyDestroyed() in that case because there is nothing to do.