Bug 1438401 - Quietly fail shmget() in sandboxed content processes. r=gcp

The X11 symbol interposition isn't enough, possibly because Cairo can
also use XCB.  Interposing XCB is more difficult because the API exposes
more protocol details.  Instead, just allow shmget to be called and
fail; this will tell Cairo that it can't use SysV IPC with the X server,
which is what we want.

MozReview-Commit-ID: 5y9tE7UXMTE

--HG--
extra : rebase_source : bb1e81116742a299bc4e412062327e69032ab3b3

security/sandbox/linux/SandboxFilter.cpp

@@ -690,11 +690,15 @@ public:
 #ifdef DESKTOP
   Maybe<ResultExpr> EvaluateIpcCall(int aCall) const override {
     switch(aCall) {
-      // These are a problem: SysV shared memory follows the Unix
-      // "same uid policy" and can't be restricted/brokered like file
-      // access.  But the graphics layer might not be using them
-      // anymore; this needs to be studied.
+      // These are a problem: SysV IPC follows the Unix "same uid
+      // policy" and can't be restricted/brokered like file access.
+      // We're not using it directly, but there are some library
+      // dependencies that do; see ContentNeedsSysVIPC() in
+      // SandboxLaunch.cpp.  Also, Cairo as used by GTK will sometimes
+      // try to use MIT-SHM, so shmget() is a non-fatal error.  See
+      // also bug 1376910 and bug 1438401.
     case SHMGET:
+      return Some(mAllowSysV ? Allow() : Error(EPERM));
     case SHMCTL:
     case SHMAT:
     case SHMDT:

security/sandbox/linux/SandboxHooks.cpp

@@ -124,17 +124,3 @@ inotify_init1(int flags)
   errno = ENOSYS;
   return -1;
 }
-
-#ifdef MOZ_X11
-// We're already preventing the use of X11 MIT-SHM like this in
-// widget/gtk/mozgtk/mozgtk.c because of bug 1271100, but that's not
-// quite enough: sometimes libXext can be preloaded, so we have to
-// defeat that with our own preload library.  (With just the mozgtk
-// interposition, we saw crashes when we blocked the SysV IPC
-// syscalls; see bug 1376910 comment #14.)
-extern "C" MOZ_EXPORT Bool
-XShmQueryExtension(Display* aDisplay)
-{
-  return False;
-}
-#endif