FEX-Emu/linux
1
0
Fork 0
mirror of https://github.com/FEX-Emu/linux.git synced 2025-01-12 12:22:42 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
linux/mm
History
Roman Gushchin 475d0487a2 mm: memcontrol: use per-cpu stocks for socket memory uncharging 
We've noticed a quite noticeable performance overhead on some hosts with
significant network traffic when socket memory accounting is enabled.

Perf top shows that socket memory uncharging path is hot:
  2.13%  [kernel]                [k] page_counter_cancel
  1.14%  [kernel]                [k] __sk_mem_reduce_allocated
  1.14%  [kernel]                [k] _raw_spin_lock
  0.87%  [kernel]                [k] _raw_spin_lock_irqsave
  0.84%  [kernel]                [k] tcp_ack
  0.84%  [kernel]                [k] ixgbe_poll
  0.83%  < workload >
  0.82%  [kernel]                [k] enqueue_entity
  0.68%  [kernel]                [k] __fget
  0.68%  [kernel]                [k] tcp_delack_timer_handler
  0.67%  [kernel]                [k] __schedule
  0.60%  < workload >
  0.59%  [kernel]                [k] __inet6_lookup_established
  0.55%  [kernel]                [k] __switch_to
  0.55%  [kernel]                [k] menu_select
  0.54%  libc-2.20.so            [.] __memcpy_avx_unaligned

To address this issue, the existing per-cpu stock infrastructure can be
used.

refill_stock() can be called from mem_cgroup_uncharge_skmem() to move
charge to a per-cpu stock instead of calling atomic
page_counter_uncharge().

To prevent the uncontrolled growth of per-cpu stocks, refill_stock()
will explicitly drain the cached charge, if the cached value exceeds
CHARGE_BATCH.

This allows significantly optimize the load:
  1.21%  [kernel]                [k] _raw_spin_lock
  1.01%  [kernel]                [k] ixgbe_poll
  0.92%  [kernel]                [k] _raw_spin_lock_irqsave
  0.90%  [kernel]                [k] enqueue_entity
  0.86%  [kernel]                [k] tcp_ack
  0.85%  < workload >
  0.74%  perf-11120.map          [.] 0x000000000061bf24
  0.73%  [kernel]                [k] __schedule
  0.67%  [kernel]                [k] __fget
  0.63%  [kernel]                [k] __inet6_lookup_established
  0.62%  [kernel]                [k] menu_select
  0.59%  < workload >
  0.59%  [kernel]                [k] __switch_to
  0.57%  libc-2.20.so            [.] __memcpy_avx_unaligned

Link: http://lkml.kernel.org/r/20170829100150.4580-1-guro@fb.com
Signed-off-by: Roman Gushchin <guro@fb.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Vladimir Davydov <vdavydov.dev@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-09-08 18:26:47 -07:00
..
kasan
Merge branch 'linus' into locking/core, to pick up fixes
2017-08-10 12:20:53 +02:00
backing-dev.c
bdi: Drop 'parent' argument from bdi_register[_va]()
2017-04-20 12:09:55 -06:00
balloon_compaction.c
mm/migrate: new migrate mode MIGRATE_SYNC_NO_COPY
2017-09-08 18:26:46 -07:00
bootmem.c
mm/bootmem.c: cosmetic improvement of code readability
2017-02-22 16:41:29 -08:00
cleancache.c
fs: switch ->s_uuid to uuid_t
2017-06-05 16:59:12 +02:00
cma_debug.c
mm/cma_debug.c: fix stack corruption due to sprintf usage
2017-08-18 15:32:02 -07:00
cma.c
cma: fix calculation of aligned offset
2017-07-10 16:32:32 -07:00
cma.h
cma: Store a name in the cma structure
2017-04-18 20:41:12 +02:00
compaction.c
mm, compaction: skip over holes in __reset_isolation_suitable
2017-07-06 16:24:32 -07:00
debug_page_ref.c
mm/page_ref: add tracepoint to track down page reference manipulation
2016-03-17 15:09:34 -07:00
debug.c
mm: make tlb_flush_pending global
2017-08-10 15:54:07 -07:00
dmapool.c
lib/vsprintf.c: remove %Z support
2017-02-27 18:43:47 -08:00
early_ioremap.c
x86/mm: Add support to access boot related data in the clear
2017-07-18 11:38:02 +02:00
fadvise.c
mm: fadvise: avoid fadvise for fs without backing device
2017-09-08 18:26:47 -07:00
failslab.c
mm: fault-inject take over bootstrap kmem_cache check
2016-03-15 16:55:16 -07:00
filemap.c
Merge branch 'akpm' (patches from Andrew)
2017-09-06 20:49:49 -07:00
frame_vector.c
treewide: use kv[mz]alloc* rather than opencoded variants
2017-05-08 17:15:13 -07:00
frontswap.c
mm, frontswap: convert frontswap_enabled to static key
2016-07-26 16:19:19 -07:00
gup.c
mm/device-public-memory: device memory cache coherent with CPU
2017-09-08 18:26:46 -07:00
highmem.c
mm/highmem: make nr_free_highpages() handles all highmem zones by itself
2016-05-19 19:12:14 -07:00
hmm.c
mm/hmm: avoid bloating arch that do not make use of HMM
2017-09-08 18:26:46 -07:00
huge_memory.c
mm: soft-dirty: keep soft-dirty bits over thp migration
2017-09-08 18:26:45 -07:00
hugetlb_cgroup.c
mm, hugetlb_cgroup: round limit_in_bytes down to hugepage size
2016-05-20 17:58:30 -07:00
hugetlb.c
powerpc updates for 4.14
2017-09-07 10:15:40 -07:00
hwpoison-inject.c
mm: hwpoison: call shake_page() unconditionally
2017-05-03 15:52:12 -07:00
init-mm.c
mm: Add a user_ns owner to mm_struct and fix ptrace permission checks
2016-11-22 11:49:48 -06:00
internal.h
mm, oom: do not rely on TIF_MEMDIE for memory reserves access
2017-09-06 17:27:30 -07:00
interval_tree.c
…
Kconfig
mm/hmm: avoid bloating arch that do not make use of HMM
2017-09-08 18:26:46 -07:00
Kconfig.debug
mm: enable page poisoning early at boot
2017-05-03 15:52:10 -07:00
khugepaged.c
mm: make PR_SET_THP_DISABLE immediately active
2017-07-10 16:32:31 -07:00
kmemcheck.c
mm: Rename SLAB_DESTROY_BY_RCU to SLAB_TYPESAFE_BY_RCU
2017-04-18 11:42:36 -07:00
kmemleak-test.c
mm: convert printk(KERN_<LEVEL> to pr_<level>
2016-03-17 15:09:34 -07:00
kmemleak.c
mm: kmemleak: treat vm_struct as alternative reference to vmalloc'ed objects
2017-07-06 16:24:34 -07:00
ksm.c
mm/ksm.c: constify attribute_group structures
2017-09-06 17:27:27 -07:00
list_lru.c
mm/list_lru.c: fix list_lru_count_node() to be race free
2017-07-10 16:32:33 -07:00
maccess.c
x86: remove more uaccess_32.h complexity
2016-05-22 17:21:27 -07:00
madvise.c
mm/device-public-memory: device memory cache coherent with CPU
2017-09-08 18:26:46 -07:00
Makefile
mm/hmm: avoid bloating arch that do not make use of HMM
2017-09-08 18:26:46 -07:00
memblock.c
mm/memblock.c: reversed logic in memblock_discard()
2017-08-25 16:12:46 -07:00
memcontrol.c
mm: memcontrol: use per-cpu stocks for socket memory uncharging
2017-09-08 18:26:47 -07:00
memory_hotplug.c
mm/ZONE_DEVICE: new type of ZONE_DEVICE for unaddressable memory
2017-09-08 18:26:46 -07:00
memory-failure.c
x86/mm, mm/hwpoison: Clear PRESENT bit for kernel 1:1 mappings of poison pages
2017-08-17 10:30:49 +02:00
memory.c
mm/memory.c: remove reduntant check for write access
2017-09-08 18:26:47 -07:00
mempolicy.c
mm: remove useless vma parameter to offset_il_node
2017-09-08 18:26:46 -07:00
mempool.c
sched/wait: Rename wait_queue_t => wait_queue_entry_t
2017-06-20 12:18:27 +02:00
memtest.c
…
migrate.c
mm/hmm: avoid bloating arch that do not make use of HMM
2017-09-08 18:26:46 -07:00
mincore.c
mm: remove shmem_mapping() shmem_zero_setup() duplicates
2017-02-24 17:46:56 -08:00
mlock.c
mm/mlock.c: use page_zone() instead of page_zone_id()
2017-09-08 18:26:47 -07:00
mm_init.c
mm: convert printk(KERN_<LEVEL> to pr_<level>
2016-03-17 15:09:34 -07:00
mmap.c
mm: oom: let oom_reap_task and exit_mmap run concurrently
2017-09-06 17:27:30 -07:00
mmu_context.c
sched/headers: Prepare to move the task_lock()/unlock() APIs to <linux/sched/task.h>
2017-03-02 08:42:38 +01:00
mmu_notifier.c
mm/mmu_notifier: kill invalidate_page
2017-08-31 16:13:00 -07:00
mmzone.c
mm/mmzone.c: swap likely to unlikely as code logic is different for next_zones_zonelist()
2017-02-22 16:41:29 -08:00
mprotect.c
mm/ZONE_DEVICE: new type of ZONE_DEVICE for unaddressable memory
2017-09-08 18:26:46 -07:00
mremap.c
mm: thp: check pmd migration entry in common path
2017-09-08 18:26:45 -07:00
msync.c
…
nobootmem.c
mm: discard memblock data later
2017-08-18 15:32:01 -07:00
nommu.c
mm: rename global_page_state to global_zone_page_state
2017-09-06 17:27:29 -07:00
oom_kill.c
mm: oom: let oom_reap_task and exit_mmap run concurrently
2017-09-06 17:27:30 -07:00
page_alloc.c
mm: change the call sites of numa statistics items
2017-09-08 18:26:47 -07:00
page_counter.c
…
page_ext.c
mm, page_ext: periodically reschedule during page_ext_init()
2017-09-06 17:27:26 -07:00
page_idle.c
mm/page_idle.c: constify attribute_group structures
2017-09-06 17:27:27 -07:00
page_io.c
Merge branch 'for-4.14/block' of git://git.kernel.dk/linux-block
2017-09-07 11:59:42 -07:00
page_isolation.c
mm: unify new_node_page and alloc_migrate_target
2017-07-10 16:32:31 -07:00
page_owner.c
mm, page_owner: don't grab zone->lock for init_pages_in_zone()
2017-09-06 17:27:26 -07:00
page_poison.c
mm: enable page poisoning early at boot
2017-05-03 15:52:10 -07:00
page_vma_mapped.c
mm/migrate: support un-addressable ZONE_DEVICE page in migration
2017-09-08 18:26:46 -07:00
page-writeback.c
mm: rename global_page_state to global_zone_page_state
2017-09-06 17:27:29 -07:00
pagewalk.c
mm/hugetlb: add size parameter to huge_pte_offset()
2017-07-06 16:24:34 -07:00
percpu-internal.h
percpu: skip chunks if the alloc does not fit in the contig hint
2017-07-26 17:41:05 -04:00
percpu-km.c
percpu: replace area map allocator with bitmap
2017-07-26 17:41:05 -04:00
percpu-stats.c
percpu: add first_bit to keep track of the first free in the bitmap
2017-07-26 17:41:05 -04:00
percpu-vm.c
percpu: fix static checker warnings in pcpu_destroy_chunk
2017-06-29 11:23:38 -04:00
percpu.c
percpu: update header to contain bitmap allocator explanation.
2017-07-26 17:41:06 -04:00
pgtable-generic.c
mm: thp: enable thp migration in generic path
2017-09-08 18:26:45 -07:00
process_vm_access.c
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/mm.h>
2017-03-02 08:42:28 +01:00
quicklist.c
fix Christoph's email addresses
2016-03-17 15:09:34 -07:00
readahead.c
mm: don't cap request size based on read-ahead setting
2016-12-12 18:55:08 -08:00
rmap.c
mm/migrate: support un-addressable ZONE_DEVICE page in migration
2017-09-08 18:26:46 -07:00
rodata_test.c
mm: remove rodata_test_data export, add pr_fmt
2017-05-03 15:52:09 -07:00
shmem.c
mm, swap: VMA based swap readahead
2017-09-06 17:27:29 -07:00
slab_common.c
mm: allow slab_nomerge to be set at build time
2017-07-06 16:24:31 -07:00
slab.c
mm: memcontrol: account slab stats per lruvec
2017-07-06 16:24:35 -07:00
slab.h
locking/lockdep: Rework FS_RECLAIM annotation
2017-08-10 12:29:03 +02:00
slob.c
locking/lockdep: Rework FS_RECLAIM annotation
2017-08-10 12:29:03 +02:00
slub.c
mm/slub.c: constify attribute_group structures
2017-09-06 17:27:27 -07:00
sparse-vmemmap.c
mm, sparse, page_ext: drop ugly N_HIGH_MEMORY branches for allocations
2017-09-06 17:27:26 -07:00
sparse.c
mm, sparse, page_ext: drop ugly N_HIGH_MEMORY branches for allocations
2017-09-06 17:27:26 -07:00
swap_cgroup.c
mm, THP, swap: delay splitting THP during swap out
2017-07-06 16:24:31 -07:00
swap_slots.c
mm/swap_slots.c: don't disable preemption while taking the per-CPU cache
2017-07-10 16:32:32 -07:00
swap_state.c
mm, swap: add sysfs interface for VMA based swap readahead
2017-09-06 17:27:29 -07:00
swap.c
mm/device-public-memory: device memory cache coherent with CPU
2017-09-08 18:26:46 -07:00
swapfile.c
swap: choose swap device according to numa node
2017-09-06 17:27:30 -07:00
truncate.c
mm/truncate.c: fix THP handling in invalidate_mapping_pages()
2017-07-10 16:32:32 -07:00
usercopy.c
mm/usercopy: Drop extra is_vmalloc_or_module() check
2017-04-05 12:30:18 -07:00
userfaultfd.c
userfaultfd: shmem: wire up shmem_mfill_zeropage_pte
2017-09-06 17:27:28 -07:00
util.c
mm: rename global_page_state to global_zone_page_state
2017-09-06 17:27:29 -07:00
vmacache.c
sched/headers: Prepare to move 'init_task' and 'init_thread_union' from <linux/sched.h> to <linux/sched/task.h>
2017-03-02 08:42:38 +01:00
vmalloc.c
mm/vmalloc.c: don't reinvent the wheel but use existing llist API
2017-09-06 17:27:29 -07:00
vmpressure.c
mm, vmpressure: pass-through notification support
2017-07-10 16:32:31 -07:00
vmscan.c
mm, THP, swap: add THP swapping out fallback counting
2017-09-06 17:27:28 -07:00
vmstat.c
mm: consider the number in local CPUs when reading NUMA stats
2017-09-08 18:26:47 -07:00
workingset.c
mm: memcontrol: per-lruvec stats infrastructure
2017-07-06 16:24:35 -07:00
z3fold.c
z3fold: use per-cpu unbuddied lists
2017-09-06 17:27:30 -07:00
zbud.c
…
zpool.c
…
zsmalloc.c
mm/zsmalloc.c: change stat type parameter to int
2017-09-08 18:26:47 -07:00
zswap.c
mm/zswap.c: delete an error message for a failed memory allocation in zswap_dstmem_prepare()
2017-07-06 16:24:35 -07:00