linux/lib
Tejun Heo 82607adcf9 workqueue: implement lockup detector
Workqueue stalls can happen from a variety of usage bugs such as
missing WQ_MEM_RECLAIM flag or concurrency managed work item
indefinitely staying RUNNING.  These stalls can be extremely difficult
to hunt down because the usual warning mechanisms can't detect
workqueue stalls and the internal state is pretty opaque.

To alleviate the situation, this patch implements workqueue lockup
detector.  It periodically monitors all worker_pools periodically and,
if any pool failed to make forward progress longer than the threshold
duration, triggers warning and dumps workqueue state as follows.

 BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 31s!
 Showing busy workqueues and worker pools:
 workqueue events: flags=0x0
   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=17/256
     pending: monkey_wrench_fn, e1000_watchdog, cache_reap, vmstat_shepherd, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, release_one_tty, cgroup_release_agent
 workqueue events_power_efficient: flags=0x80
   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=2/256
     pending: check_lifetime, neigh_periodic_work
 workqueue cgroup_pidlist_destroy: flags=0x0
   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1
     pending: cgroup_pidlist_destroy_work_fn
 ...

The detection mechanism is controller through kernel parameter
workqueue.watchdog_thresh and can be updated at runtime through the
sysfs module parameter file.

v2: Decoupled from softlockup control knobs.

Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Don Zickus <dzickus@redhat.com>
Cc: Ulrich Obergfell <uobergfe@redhat.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Chris Mason <clm@fb.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
2015-12-08 11:29:47 -05:00
..
842 crypto: 842 - Add CRC and validation support 2015-10-14 22:23:17 +08:00
fonts fonts: Add 6x10 font 2014-10-09 11:35:48 +03:00
lz4 lz4: fix system halt at boot kernel on x86_64 2015-05-24 11:56:29 -07:00
lzo lzo: check for length overrun in variable length encoding. 2014-09-28 11:08:01 +02:00
mpi asm-generic cleanups 2015-11-06 14:22:15 -08:00
raid6 md/raid6: delta syndrome for ARM NEON 2015-08-31 19:29:05 +02:00
reed_solomon
xz lib/xz: enable all filters by default in Kconfig 2014-06-04 16:54:18 -07:00
zlib_deflate zlib_deflate/deftree: remove bi_reverse() 2015-09-10 13:29:01 -07:00
zlib_inflate zlib: clean up some dead code 2014-08-06 18:01:24 -07:00
.gitignore
argv_split.c
asn1_decoder.c ASN.1: Handle 'ANY OPTIONAL' in grammar 2015-08-05 13:38:07 +01:00
assoc_array.c assoc_array: Include rcupdate.h for call_rcu() definition 2015-01-07 16:08:41 +00:00
atomic64_test.c atomic: Add simple atomic_t tests 2015-07-27 14:06:24 +02:00
atomic64.c atomic: Provide atomic_{or,xor,and} 2015-07-27 14:06:24 +02:00
audit.c syscalls: implement execveat() system call 2014-12-13 12:42:51 -08:00
bcd.c
bch.c
bitmap.c lib/bitmap.c: bitmap_parselist can accept string with whitespaces on head or tail 2015-09-10 13:29:01 -07:00
bitrev.c ARM: 8187/1: add CONFIG_HAVE_ARCH_BITREVERSE to support rbit instruction 2014-12-22 16:43:06 +00:00
bsearch.c
btree.c lib/btree.c: fix leak of whole btree nodes 2014-06-04 16:54:18 -07:00
bug.c module: Sanitize RCU usage and locking 2015-05-28 11:31:52 +09:30
build_OID_registry
bust_spinlocks.c
check_signature.c
checksum.c lib/checksum.c: fix build for generic csum_tcpudp_nofold 2015-01-29 11:57:38 -08:00
clz_ctz.c
clz_tab.c
cmdline.c lib: Add a generic cmdline parse function parse_option_str 2014-10-03 18:40:58 +01:00
compat_audit.c
cordic.c
cpu_rmap.c sched/topology: Rename topology_thread_cpumask() to topology_sibling_cpumask() 2015-05-27 15:22:15 +02:00
cpu-notifier-error-inject.c
cpumask.c revert "cpumask: don't perform while loop in cpumask_next_and()" 2015-06-18 17:00:23 -10:00
crc7.c
crc8.c
crc16.c
crc32.c lib: crc32: Add some additional __pure annotations 2014-06-25 16:04:00 -07:00
crc32defs.h
crc-ccitt.c
crc-itu-t.c lib: crc-itu-t.[ch] fix 0x0x prefix in integer constants 2015-05-26 15:26:43 +02:00
crc-t10dif.c lib: introduce crc_t10dif_update() 2015-05-30 22:42:24 -07:00
ctype.c
debug_info.c kbuild: include core debug info when DEBUG_INFO_REDUCED 2015-06-11 15:08:32 +02:00
debug_locks.c
debugobjects.c lib/debugobjects.c: convert printk(KERN_DEBUG to pr_debug 2014-06-04 16:53:53 -07:00
dec_and_lock.c
decompress_bunzip2.c lib/decompressors: use real out buf size for gunzip with kernel 2015-09-10 13:29:01 -07:00
decompress_inflate.c lib/decompressors: use real out buf size for gunzip with kernel 2015-09-10 13:29:01 -07:00
decompress_unlz4.c lib/decompressors: use real out buf size for gunzip with kernel 2015-09-10 13:29:01 -07:00
decompress_unlzma.c lib/decompress_unlzma: Do a NULL check for pointer 2015-09-10 13:29:01 -07:00
decompress_unlzo.c lib/decompressors: use real out buf size for gunzip with kernel 2015-09-10 13:29:01 -07:00
decompress_unxz.c lib/decompressors: use real out buf size for gunzip with kernel 2015-09-10 13:29:01 -07:00
decompress.c lib/decompress: set the compressor name to NULL on error 2015-07-17 16:39:54 -07:00
devres.c devres: fix a for loop bounds check 2015-10-05 04:49:54 +01:00
digsig.c KEYS: Merge the type-specific data with the payload data 2015-10-21 15:18:36 +01:00
div64.c remove abs64() 2015-11-09 15:11:24 -08:00
dma-debug.c dma-debug: check nents in dma_sync_sg* 2015-11-06 17:50:42 -08:00
dump_stack.c
dynamic_debug.c lib/dynamic_debug.c: use kstrdup_const 2015-11-06 17:50:42 -08:00
dynamic_queue_limits.c lib/dynamic_queue_limits.c: simplify includes 2015-02-12 18:54:15 -08:00
earlycpio.c
extable.c
fault-inject.c fault-inject: fix inverted interval/probability values in printk 2015-10-23 17:55:10 +09:00
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
fdt.c
find_bit.c lib: rename lib/find_next_bit.c to lib/find_bit.c 2015-04-17 09:03:54 -04:00
flex_array.c
flex_proportions.c proportions: add @gfp to init functions 2014-09-08 09:51:30 +09:00
gcd.c
gen_crc32table.c lib: crc32: constify crc32 lookup table 2015-02-13 21:21:35 -08:00
genalloc.c genalloc: add support of multiple gen_pools per device 2015-09-04 16:54:41 -07:00
glob.c lib/glob.c: add CONFIG_GLOB_SELFTEST 2014-08-06 18:01:25 -07:00
halfmd4.c lib/halfmd4.c: use rol32 inline function in the ROUND macro 2015-11-06 17:50:42 -08:00
hexdump.c lib/hexdump.c: truncate output in case of overflow 2015-11-06 17:50:42 -08:00
hweight.c Make ARCH_HAS_FAST_MULTIPLIER a real config variable 2014-09-13 11:14:53 -07:00
idr.c mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd 2015-11-06 17:50:42 -08:00
inflate.c
int_sqrt.c
interval_tree_test.c
interval_tree.c lib/interval_tree.c: simplify includes 2015-02-12 18:54:15 -08:00
iomap_copy.c
iomap.c
iommu-common.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc 2015-11-05 16:34:48 -08:00
iommu-helper.c
ioremap.c x86, mm: support huge KVA mappings on x86 2015-04-14 16:49:04 -07:00
iov_iter.c Merge branch 'iov_iter' into for-next 2015-04-11 22:26:51 -04:00
irq_regs.c
is_single_threaded.c lib/is_single_threaded.c: change current_is_single_threaded() to use for_each_thread() 2015-11-06 17:50:42 -08:00
jedec_ddr_data.c
kasprintf.c lib/kasprintf.c: introduce kvasprintf_const 2015-11-06 17:50:42 -08:00
Kconfig lib/Kconfig: ZLIB_DEFLATE must select BITREVERSE 2015-10-16 11:42:28 -07:00
Kconfig.debug workqueue: implement lockup detector 2015-12-08 11:29:47 -05:00
Kconfig.kasan mm, slub, kasan: enable user tracking by default with KASAN=y 2015-11-05 19:34:48 -08:00
Kconfig.kgdb kdb: Allow access to sensitive commands to be restricted by default 2014-11-11 09:31:52 -06:00
Kconfig.kmemcheck
kfifo.c kfifo: use BUG_ON 2014-08-08 15:57:25 -07:00
klist.c klist: implement klist_prev() 2015-07-28 08:50:42 +01:00
kobject_uevent.c lib/kobject_uevent.c: remove redundant include 2015-02-12 18:54:15 -08:00
kobject.c lib/kobject.c: use kvasprintf_const for formatting ->name 2015-11-06 17:50:42 -08:00
kstrtox.c kstrto*: accept "-0" for signed conversion 2015-09-10 13:29:01 -07:00
kstrtox.h
lcm.c block: fix blk_stack_limits() regression due to lcm() change 2015-03-31 09:45:50 -06:00
libcrc32c.c crypto: LLVMLinux: Remove VLAIS usage from libcrc32c.c 2014-10-14 10:51:23 +02:00
list_debug.c
list_sort.c lib/list_sort: use late_initcall to hook in self tests 2015-06-16 14:12:35 -04:00
llist.c lib/llist.c: fix data race in llist_del_first 2015-11-06 17:50:42 -08:00
locking-selftest-hardirq.h
locking-selftest-mutex.h
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h
locking-selftest-rsem.h
locking-selftest-softirq.h
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h
locking-selftest-wsem.h
locking-selftest.c locking/lockdep: Revert qrwlock recusive stuff 2014-10-03 06:09:30 +02:00
lockref.c locking/lockref: Remove homebrew cmpxchg64_relaxed() macro definition 2015-08-12 11:59:04 +02:00
lru_cache.c lru_cache: remove use of seq_printf return value 2015-04-15 16:35:25 -07:00
Makefile test_printf: test printf family at runtime 2015-11-06 17:50:42 -08:00
md5.c lib/md5.c: simplify include 2015-02-12 18:54:15 -08:00
memory-notifier-error-inject.c
memweight.c
net_utils.c mac_pton: Use bool not int return 2014-06-25 17:45:43 -07:00
nlattr.c netlink: pad nla_memcpy dest buffer with zeroes 2015-03-31 14:07:24 -04:00
nmi_backtrace.c ARM: 8439/1: Fix backtrace generation when IPI is masked 2015-10-03 16:40:51 +01:00
notifier-error-inject.c
notifier-error-inject.h
of-reconfig-notifier-error-inject.c
oid_registry.c
once.c once: make helper generic for calling functions once 2015-10-08 05:26:36 -07:00
parser.c
pci_iomap.c libnvdimm for 4.3: 2015-09-08 14:35:59 -07:00
percpu_counter.c percpu_counter: batch size aware __percpu_counter_compare() 2015-05-29 07:39:34 +10:00
percpu_ida.c mm, page_alloc: rename __GFP_WAIT to __GFP_RECLAIM 2015-11-06 17:50:42 -08:00
percpu_test.c
percpu-refcount.c percpu_ref: make INIT_ATOMIC and switch_to_atomic() sticky 2014-09-24 13:31:50 -04:00
plist.c lib/plist.c: remove redundant include 2015-02-12 18:54:16 -08:00
pm-notifier-error-inject.c
proportions.c proportions: add @gfp to init functions 2014-09-08 09:51:30 +09:00
radix-tree.c mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd 2015-11-06 17:50:42 -08:00
random32.c random32: add prandom_init_once helper for own rngs 2015-10-08 05:26:38 -07:00
ratelimit.c
rational.c
rbtree_test.c
rbtree.c rbtree: Make lockless searches non-fatal 2015-05-28 11:32:04 +09:30
reciprocal_div.c
rhashtable.c lib: fix data race in rhashtable_rehash_one 2015-09-22 17:36:07 -07:00
scatterlist.c scatterlist: allow limited chaining without ARCH_HAS_SG_CHAIN 2015-08-17 08:12:51 -06:00
seq_buf.c seq_buf: Fix seq_buf_bprintf() truncation 2015-03-04 23:40:19 -05:00
sg_split.c lib: scatterlist: add sg splitting function 2015-08-24 14:28:01 -06:00
sha1.c lib: EXPORT_SYMBOL sha_init 2015-03-23 22:12:08 -04:00
show_mem.c lib/show_mem.c: correct reserved memory calculation 2015-09-08 15:35:28 -07:00
smp_processor_id.c
sort.c lib/sort: Add 64 bit swap function 2015-06-25 17:00:40 -07:00
stmp_device.c lib/stmp_device.c: replace module.h include 2015-02-12 18:54:16 -08:00
string_helpers.c lib/string_helpers.c: fix infinite loop in string_get_size() 2015-09-17 21:16:07 -07:00
string.c lib/string.c: add ULL suffix to the constant definition 2015-11-10 16:32:11 -08:00
strncpy_from_user.c lib: move strncpy_from_unsafe() into mm/maccess.c 2015-08-31 12:36:10 -07:00
strnlen_user.c Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2015-06-22 15:52:04 -07:00
swiotlb.c Merge branch 'for-4.2/sg' of git://git.kernel.dk/linux-block 2015-06-25 15:22:36 -07:00
syscall.c
test_bpf.c bpf: add mod default A and X test cases 2015-11-05 00:05:50 -05:00
test_firmware.c test: add firmware_class loader test 2014-07-17 18:44:19 -07:00
test_kasan.c lib: test_kasan: add some testcases 2015-11-05 19:34:48 -08:00
test_module.c
test_printf.c test_printf: test printf family at runtime 2015-11-06 17:50:42 -08:00
test_rhashtable.c rhashtable-test: extend to test concurrency 2015-08-17 14:33:47 -07:00
test_static_key_base.c locking/static_keys: Provide a selftest 2015-08-03 11:51:12 +02:00
test_static_keys.c locking/static_keys: Make verify_keys() static 2015-08-05 09:53:40 +02:00
test_user_copy.c
test-hexdump.c hexdump: Make test data really const 2015-06-25 17:00:40 -07:00
test-kstrtox.c kstrto*: accept "-0" for signed conversion 2015-09-10 13:29:01 -07:00
test-string_helpers.c lib/test-string_helpers.c: add string_get_size() tests 2015-11-06 17:50:42 -08:00
textsearch.c lib/textsearch.c: remove textsearch_put reference from comments 2014-10-14 02:18:14 +02:00
timerqueue.c timerqueue: Let timerqueue_add/del return information 2015-04-22 17:06:49 +02:00
ts_bm.c
ts_fsm.c
ts_kmp.c
ucs2_string.c
usercopy.c
uuid.c
vsprintf.c lib/vsprintf.c: update documentation 2015-11-06 17:50:42 -08:00