linux/mm
Kirill A. Shutemov 1d798ca3f1 mm: make compound_head() robust
Hugh has pointed that compound_head() call can be unsafe in some
context. There's one example:

	CPU0					CPU1

isolate_migratepages_block()
  page_count()
    compound_head()
      !!PageTail() == true
					put_page()
					  tail->first_page = NULL
      head = tail->first_page
					alloc_pages(__GFP_COMP)
					   prep_compound_page()
					     tail->first_page = head
					     __SetPageTail(p);
      !!PageTail() == true
    <head == NULL dereferencing>

The race is pure theoretical. I don't it's possible to trigger it in
practice. But who knows.

We can fix the race by changing how encode PageTail() and compound_head()
within struct page to be able to update them in one shot.

The patch introduces page->compound_head into third double word block in
front of compound_dtor and compound_order. Bit 0 encodes PageTail() and
the rest bits are pointer to head page if bit zero is set.

The patch moves page->pmd_huge_pte out of word, just in case if an
architecture defines pgtable_t into something what can have the bit 0
set.

hugetlb_cgroup uses page->lru.next in the second tail page to store
pointer struct hugetlb_cgroup. The patch switch it to use page->private
in the second tail page instead. The space is free since ->first_page is
removed from the union.

The patch also opens possibility to remove HUGETLB_CGROUP_MIN_ORDER
limitation, since there's now space in first tail page to store struct
hugetlb_cgroup pointer. But that's out of scope of the patch.

That means page->compound_head shares storage space with:

 - page->lru.next;
 - page->next;
 - page->rcu_head.next;

That's too long list to be absolutely sure, but looks like nobody uses
bit 0 of the word.

page->rcu_head.next guaranteed[1] to have bit 0 clean as long as we use
call_rcu(), call_rcu_bh(), call_rcu_sched(), or call_srcu(). But future
call_rcu_lazy() is not allowed as it makes use of the bit and we can
get false positive PageTail().

[1] http://lkml.kernel.org/g/20150827163634.GD4029@linux.vnet.ibm.com

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Andrea Arcangeli <aarcange@redhat.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-11-06 17:50:42 -08:00
..
kasan kasan: always taint kernel on report 2015-11-05 19:34:48 -08:00
backing-dev.c mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd 2015-11-06 17:50:42 -08:00
balloon_compaction.c mm: page migration trylock newpage at same level as oldpage 2015-11-05 19:34:48 -08:00
bootmem.c
cleancache.c
cma_debug.c
cma.c mm/cma.c: suppress warning 2015-11-05 19:34:48 -08:00
cma.h
compaction.c mm, compaction: distinguish contended status in tracepoints 2015-11-05 19:34:48 -08:00
debug-pagealloc.c
debug.c mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
dmapool.c mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd 2015-11-06 17:50:42 -08:00
early_ioremap.c mm/early_ioremap: use offset_in_page macro 2015-11-05 19:34:48 -08:00
fadvise.c
failslab.c mm, page_alloc: rename __GFP_WAIT to __GFP_RECLAIM 2015-11-06 17:50:42 -08:00
filemap.c mm, fs: introduce mapping_gfp_constraint() 2015-11-06 17:50:42 -08:00
frame_vector.c mm: fix docbook comment for get_vaddr_frames() 2015-11-05 19:34:48 -08:00
frontswap.c
gup.c mm: introduce VM_LOCKONFAULT 2015-11-05 19:34:48 -08:00
highmem.c
huge_memory.c mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
hugetlb_cgroup.c mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
hugetlb.c mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
hwpoison-inject.c
init-mm.c
internal.h mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
interval_tree.c
Kconfig mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
Kconfig.debug
kmemcheck.c
kmemleak-test.c
kmemleak.c mm/kmemleak.c: remove unneeded initialization of object to NULL 2015-11-05 19:34:48 -08:00
ksm.c ksm: unstable_tree_search_insert error checking cleanup 2015-11-05 19:34:48 -08:00
list_lru.c memcg: simplify and inline __mem_cgroup_from_kmem 2015-11-05 19:34:48 -08:00
maccess.c mm/maccess.c: actually return -EFAULT from strncpy_from_unsafe 2015-11-05 19:34:48 -08:00
madvise.c
Makefile
memblock.c mm/memblock: make memblock_remove_range() static 2015-11-05 19:34:48 -08:00
memcontrol.c mm/memcontrol.c: uninline mem_cgroup_usage 2015-11-06 17:50:42 -08:00
memory_hotplug.c mm/page_alloc: remove unused parameter in init_currently_empty_zone() 2015-11-05 19:34:48 -08:00
memory-failure.c mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
memory.c mm, dax: fix DAX deadlocks 2015-10-16 11:42:28 -07:00
mempolicy.c
mempool.c mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd 2015-11-06 17:50:42 -08:00
memtest.c
migrate.c mm, page_alloc: rename __GFP_WAIT to __GFP_RECLAIM 2015-11-06 17:50:42 -08:00
mincore.c mm/mincore: use offset_in_page macro 2015-11-05 19:34:48 -08:00
mlock.c mm: mlock: add mlock flags to enable VM_LOCKONFAULT usage 2015-11-05 19:34:48 -08:00
mm_init.c
mmap.c mm: introduce VM_LOCKONFAULT 2015-11-05 19:34:48 -08:00
mmu_context.c
mmu_notifier.c
mmzone.c
mprotect.c
mremap.c mm/mremap: use offset_in_page macro 2015-11-05 19:34:48 -08:00
msync.c mm/msync: use offset_in_page macro 2015-11-05 19:34:48 -08:00
nobootmem.c
nommu.c mm/nommu.c: drop unlikely inside BUG_ON() 2015-11-05 19:34:48 -08:00
oom_kill.c mm/oom_kill.c: introduce is_sysrq_oom helper 2015-11-06 17:50:42 -08:00
page_alloc.c mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
page_counter.c mm: page_counter: let page_counter_try_charge() return bool 2015-11-05 19:34:48 -08:00
page_ext.c
page_idle.c
page_io.c
page_isolation.c
page_owner.c
page-writeback.c writeback: fix incorrect calculation of available memory for memcg domains 2015-10-12 10:31:13 -06:00
pagewalk.c
percpu-km.c
percpu-vm.c
percpu.c mm/percpu: use offset_in_page macro 2015-11-05 19:34:48 -08:00
pgtable-generic.c mm,thp: introduce flush_pmd_tlb_range 2015-10-17 17:48:20 +05:30
process_vm_access.c
quicklist.c
readahead.c mm, fs: introduce mapping_gfp_constraint() 2015-11-06 17:50:42 -08:00
rmap.c mm: page migration use migration entry for swapcache too 2015-11-05 19:34:48 -08:00
shmem.c mm: page_alloc: hide some GFP internals and document the bits and flag combinations 2015-11-06 17:50:42 -08:00
slab_common.c mm/slab_common.c: initialize kmem_cache pointer to NULL 2015-11-05 19:34:48 -08:00
slab.c slab, slub: use page->rcu_head instead of page->lru plus cast 2015-11-06 17:50:42 -08:00
slab.h memcg: unify slab and other kmem pages charging 2015-11-05 19:34:48 -08:00
slob.c
slub.c slab, slub: use page->rcu_head instead of page->lru plus cast 2015-11-06 17:50:42 -08:00
sparse-vmemmap.c
sparse.c
swap_cgroup.c
swap_state.c
swap.c mm: make compound_head() robust 2015-11-06 17:50:42 -08:00
swapfile.c
truncate.c
userfaultfd.c
util.c mm/util: use offset_in_page macro 2015-11-05 19:34:48 -08:00
vmacache.c mm/vmacache: inline vmacache_valid_mm() 2015-11-05 19:34:48 -08:00
vmalloc.c mm: page_alloc: hide some GFP internals and document the bits and flag combinations 2015-11-06 17:50:42 -08:00
vmpressure.c
vmscan.c mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd 2015-11-06 17:50:42 -08:00
vmstat.c mm, page_alloc: reserve pageblocks for high-order atomic allocations on demand 2015-11-06 17:50:42 -08:00
workingset.c
zbud.c mm: zsmalloc: constify struct zs_pool name 2015-11-06 17:50:42 -08:00
zpool.c mm: zsmalloc: constify struct zs_pool name 2015-11-06 17:50:42 -08:00
zsmalloc.c zsmalloc: use page->private instead of page->first_page 2015-11-06 17:50:42 -08:00
zswap.c zswap: use charp for zswap param strings 2015-11-06 17:50:42 -08:00