Race condition in glibc between _dl_lookup_symbol_x() and dlopen/dlclose/etc

Solution Verified - Updated -

Issue

  • We got a rare but serious race condition in glibc, that resulted in a failure of resolving dependencies of a library symbol.

  • The _dl_lookup_symbol_x() called from _dl_runtime_resolve updates the l_used member, a bit field in the link_map structure, without getting dl_load_lock.

         It causes a race condition between _dl_lookup_symbol_x() and one of other procedures, dlopen, dlclose, etc., during their updates of the bit field members in this structure.

_dl_lookup_symbol_x() 
[./elf/dl-lookup.c]
 380 
 381   /* The object is used.  */
 382   current_value.m->l_used = 1; 
  • When a bit field is updated, other neighbor fields are also loaded from the memory and stored into the memory at the same time.

  • So when the l_used is updated, the following members in the link_map structure are also loaded/stored simultaneously:

           l_type
           l_relocated
           l_init_called
           l_global
           l_reserved
           l_phdr_allocate
           l_soname_added
           l_faked
           l_l_need_tls_int
           l_auditing
           l_audit_any_plt
           l_removed
           l_contiguous
           l_fini_called
    
  • The l_reserved is updated to 0 in _dl_map_object_deps(*) and it supposed to be.

    _dl_map_object_deps()
    [./elf/dl-deps.c]
     510 
     511   for (nlist = 0, runp = known; runp; runp = runp->next)
     512     {
     513       if (__builtin_expect (trace_mode, 0) && runp->map->l_faked)
     514         /* This can happen when we trace the loading.  */
     515         --map->l_searchlist.r_nlist;
     516       else
     517         map->l_searchlist.r_list[nlist++] = runp->map;
     518 
     519       /* Now clear all the mark bits we set in the objects on the search list
     520          to avoid duplicates, so the next call starts fresh.  */
     521       runp->map->l_reserved = 0;                                     <--------------(*)
     522     }
    
  • But it did not become 0 because of a race condition between _dl_lookup_symbol_x and _dl_map_object_deps like below:

    [Thread A]
    <_dl_lookup_symbol_x+6369>:            movl r16=0x80000000000;;
    <_dl_lookup_symbol_x+6384>: [MMI]       adds r15=784,r15;;
    <_dl_lookup_symbol_x+6385>:            ld8 r14=[r15]    ------ (A-1) load the
    bit fields
    <_dl_lookup_symbol_x+6386>:            nop.i 0x0;;
    <_dl_lookup_symbol_x+6400>: [MMI]       or r14=r16,r14;; ------ (A-2) set
    l_used bit
                                                                       (A-3)
    interrupted or stalled
    <_dl_lookup_symbol_x+6401>:            st8 [r15]=r14 --------- (A-4) store the
    bit fields
    <_dl_lookup_symbol_x+6402>:            mov r14=1028
    <_dl_lookup_symbol_x+6416>: [MMI]       ld4 r15=[r41];;
    <_dl_lookup_symbol_x+6417>:            and r14=r15,r14
    
    [Thread B]
    <_dl_map_object_deps+3088>: [MMI]       ld8 r14=[r15];; ------- (B-1) load the
    bit fields
    <_dl_map_object_deps+3089>:            nop.m 0x0
    <_dl_map_object_deps+3090>:            dep r14=0,r14,37,2;; -- (B-2) clear
    l_reserved bit
    <_dl_map_object_deps+3104>: [MMI]       st8 [r15]=r14 --------- (B-3) store the
    bit fields
    <_dl_map_object_deps+3105>:            ld8 r41=[r16]
    <_dl_map_object_deps+3106>:            nop.i 0x0;;
    
    steps | [Thread A]| [Thread B] | l_reserved value in the memory
    ------+-----------+------------+----------------------------
       1 |(A-1)      |            | 1
       2 |           |(B-1)       | 1
       3 |(A-2)      |            | 1
       4 |           |(B-2)       | 1
       5 |(A-3)      |            | 1
       6 |           |(B-3)       | 0
       7 |(A-4)      |            | 1
    
  • As a result, the l_reserved remained 1 even after _dl_map_object_deps().

  • A library with its link_map having nonzero l_reserved field will not be added to the l_searchlist of other libraries at all hereafter, resulting unresolved dependencies for the library.

Environment

  • Red Hat Enterprise Linux 5.3
  • Red Hat Enterprise Linux 4.4

  • glibc-2.5-34

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.