munmap() should not fail when mm_struct.map_count temporarily reaches max_map_count

Solution Verified - Updated -

Issue

  • The munmap() syscall fails even  if the number of VMAs of a process (mm_struct.map_count) does not yet  reach the upper limit (sysctl_max_map_count).  The munmap() fails with ENOMEM if the number of VMAs equals the "limit - 1"  and it does not unmap a whole VMA but a part of a VMA.

  • In that case, the number of VMAs temporarily reaches the limit during munmap() and it backs to below the limit at the end of munmap(). So it
      should not fail.

Environment

  • Red Hat Enterprise Linux 5 Update 4
  • Red Hat Enterprise Linux 4 Update 8
  • ia64

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content