linux/kernel
Nadav Amit 16af97dc5a mm: migrate: prevent racy access to tlb_flush_pending
Patch series "fixes of TLB batching races", v6.

It turns out that Linux TLB batching mechanism suffers from various
races.  Races that are caused due to batching during reclamation were
recently handled by Mel and this patch-set deals with others.  The more
fundamental issue is that concurrent updates of the page-tables allow
for TLB flushes to be batched on one core, while another core changes
the page-tables.  This other core may assume a PTE change does not
require a flush based on the updated PTE value, while it is unaware that
TLB flushes are still pending.

This behavior affects KSM (which may result in memory corruption) and
MADV_FREE and MADV_DONTNEED (which may result in incorrect behavior).  A
proof-of-concept can easily produce the wrong behavior of MADV_DONTNEED.
Memory corruption in KSM is harder to produce in practice, but was
observed by hacking the kernel and adding a delay before flushing and
replacing the KSM page.

Finally, there is also one memory barrier missing, which may affect
architectures with weak memory model.

This patch (of 7):

Setting and clearing mm->tlb_flush_pending can be performed by multiple
threads, since mmap_sem may only be acquired for read in
task_numa_work().  If this happens, tlb_flush_pending might be cleared
while one of the threads still changes PTEs and batches TLB flushes.

This can lead to the same race between migration and
change_protection_range() that led to the introduction of
tlb_flush_pending.  The result of this race was data corruption, which
means that this patch also addresses a theoretically possible data
corruption.

An actual data corruption was not observed, yet the race was was
confirmed by adding assertion to check tlb_flush_pending is not set by
two threads, adding artificial latency in change_protection_range() and
using sysctl to reduce kernel.numa_balancing_scan_delay_ms.

Link: http://lkml.kernel.org/r/20170802000818.4760-2-namit@vmware.com
Fixes: 2084140594 ("mm: fix TLB flush race between migration, and
change_protection_range")
Signed-off-by: Nadav Amit <namit@vmware.com>
Acked-by: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
Acked-by: Minchan Kim <minchan@kernel.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-08-10 15:54:07 -07:00
..
bpf bpf: fix bpf_prog_get_info_by_fd to dump correct xlated_prog_len 2017-07-29 23:29:41 -07:00
cgroup cpuset: fix a deadlock due to incomplete patching of cpusets_enabled() 2017-08-02 17:16:12 -07:00
configs config: android-base: disable CONFIG_NFSD and CONFIG_NFS_FS 2017-06-09 11:47:38 +02:00
debug
events Merge branch 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2017-07-21 11:12:48 -07:00
gcov
irq genirq/cpuhotplug: Revert "Set force affinity flag on hotplug migration" 2017-07-27 15:40:02 +02:00
livepatch livepatch: Fix stacking of patches with respect to RCU 2017-06-20 10:42:19 +02:00
locking Merge branch 'locking-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2017-07-21 11:11:23 -07:00
power mm: fix global NR_SLAB_.*CLAIMABLE counter reads 2017-08-10 15:54:06 -07:00
printk Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/pmladek/printk 2017-07-05 11:11:26 -07:00
rcu rcu: Remove RCU CPU stall warnings from Tiny RCU 2017-06-08 18:52:45 -07:00
sched sched/core: Fix some documentation build warnings 2017-07-25 11:17:02 +02:00
time timers: Fix overflow in get_next_timer_interrupt 2017-08-01 14:20:53 +02:00
trace trace: fix the errors caused by incompatible type of RCU variables 2017-07-20 09:27:29 -04:00
.gitignore
acct.c
async.c
audit_fsnotify.c
audit_tree.c
audit_watch.c
audit.c Merge branch 'stable-4.13' of git://git.infradead.org/users/pcmoore/audit 2017-07-20 10:22:26 -07:00
audit.h audit: style fix 2017-06-12 18:07:43 -04:00
auditfilter.c
auditsc.c Merge branch 'stable-4.13' of git://git.infradead.org/users/pcmoore/audit 2017-07-05 11:24:05 -07:00
backtracetest.c
bounds.c
capability.c
compat.c Merge branch 'misc.compat' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2017-07-06 20:57:13 -07:00
configs.c
context_tracking.c
cpu_pm.c
cpu.c smp/hotplug: Replace BUG_ON and react useful 2017-07-11 22:25:44 +02:00
crash_core.c kdump: protect vmcoreinfo data under the crash memory 2017-07-12 16:26:00 -07:00
crash_dump.c
cred.c
delayacct.c
dma.c
elfcore.c
exec_domain.c
exit.c kernel/exit.c: avoid undefined behaviour when calling wait4() 2017-07-10 16:32:36 -07:00
extable.c lib/extable.c: use bsearch() library function in search_extable() 2017-07-10 16:32:35 -07:00
fork.c mm: migrate: prevent racy access to tlb_flush_pending 2017-08-10 15:54:07 -07:00
freezer.c
futex_compat.c
futex.c futex: Remove unnecessary warning from get_futex_key 2017-08-09 14:00:54 -07:00
groups.c kernel/groups.c: use sort library function 2017-07-10 16:32:34 -07:00
hung_task.c
irq_work.c
jump_label.c
kallsyms.c kernel/kallsyms.c: replace all_var with IS_ENABLED(CONFIG_KALLSYMS_ALL) 2017-07-10 16:32:34 -07:00
kcmp.c kcmp: add KCMP_EPOLL_TFD mode to compare epoll target files 2017-07-12 16:26:01 -07:00
Kconfig.freezer
Kconfig.hz
Kconfig.locks
Kconfig.preempt
kcov.c
kexec_core.c kdump: protect vmcoreinfo data under the crash memory 2017-07-12 16:26:00 -07:00
kexec_file.c kexec_file: adjust declaration of kexec_purgatory 2017-07-12 16:26:02 -07:00
kexec_internal.h kexec_file: adjust declaration of kexec_purgatory 2017-07-12 16:26:02 -07:00
kexec.c kdump: protect vmcoreinfo data under the crash memory 2017-07-12 16:26:00 -07:00
kmod.c kmod: throttle kmod thread limit 2017-07-14 15:05:13 -07:00
kprobes.c kprobes: Ensure that jprobe probepoints are at function entry 2017-07-08 11:05:35 +02:00
ksysfs.c kexec: move vmcoreinfo out of the kernel's .bss section 2017-07-12 16:25:59 -07:00
kthread.c
latencytop.c
Makefile kernel/watchdog: split up config options 2017-07-12 16:26:02 -07:00
membarrier.c
memremap.c mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory 2017-07-06 16:24:32 -07:00
module_signing.c
module-internal.h
module.c Modules updates for v4.13 2017-07-12 17:22:01 -07:00
notifier.c
nsproxy.c
padata.c
panic.c
params.c
pid_namespace.c
pid.c pid: kill pidhash_size in pidhash_init() 2017-08-02 16:34:46 -07:00
profile.c
ptrace.c
range.c
reboot.c
relay.c
resource.c
seccomp.c seccomp: Switch from atomic_t to recount_t 2017-06-26 09:24:00 -07:00
signal.c Fix compat_sys_sigpending breakage 2017-08-06 11:48:27 -07:00
smp.c
smpboot.c
smpboot.h
softirq.c
stacktrace.c
stop_machine.c
sys_ni.c
sys.c fix a braino in compat_sys_getrlimit() 2017-07-12 09:15:00 -07:00
sysctl_binary.c kernel/sysctl_binary.c: check name array length in deprecated_sysctl_warning() 2017-07-12 16:26:00 -07:00
sysctl.c kernel/watchdog: split up config options 2017-07-12 16:26:02 -07:00
task_work.c
taskstats.c
test_kprobes.c
torture.c
tracepoint.c
tsacct.c
ucount.c
uid16.c
up.c
user_namespace.c
user-return-notifier.c
user.c
utsname_sysctl.c
utsname.c
watchdog_hld.c kernel/watchdog: split up config options 2017-07-12 16:26:02 -07:00
watchdog.c kernel/watchdog.c: use better pr_fmt prefix 2017-07-14 15:05:13 -07:00
workqueue_internal.h
workqueue.c workqueue: Work around edge cases for calc of pool's cpumask 2017-07-28 11:05:52 -04:00