Race condition in glibc between _dl_lookup_symbol_x() and dlopen/dlclose/etc
Issue
-
We got a rare but serious race condition in glibc, that resulted in a failure of resolving dependencies of a library symbol.
-
The _dl_lookup_symbol_x() called from _dl_runtime_resolve updates the l_used member, a bit field in the link_map structure, without getting dl_load_lock.
It causes a race condition between _dl_lookup_symbol_x() and one of other procedures, dlopen, dlclose, etc., during their updates of the bit field members in this structure.
_dl_lookup_symbol_x()
[./elf/dl-lookup.c]
380
381 /* The object is used. */
382 current_value.m->l_used = 1;
-
When a bit field is updated, other neighbor fields are also loaded from the memory and stored into the memory at the same time.
-
So when the l_used is updated, the following members in the link_map structure are also loaded/stored simultaneously:
l_type l_relocated l_init_called l_global l_reserved l_phdr_allocate l_soname_added l_faked l_l_need_tls_int l_auditing l_audit_any_plt l_removed l_contiguous l_fini_called -
The l_reserved is updated to 0 in _dl_map_object_deps(*) and it supposed to be.
_dl_map_object_deps() [./elf/dl-deps.c] 510 511 for (nlist = 0, runp = known; runp; runp = runp->next) 512 { 513 if (__builtin_expect (trace_mode, 0) && runp->map->l_faked) 514 /* This can happen when we trace the loading. */ 515 --map->l_searchlist.r_nlist; 516 else 517 map->l_searchlist.r_list[nlist++] = runp->map; 518 519 /* Now clear all the mark bits we set in the objects on the search list 520 to avoid duplicates, so the next call starts fresh. */ 521 runp->map->l_reserved = 0; <--------------(*) 522 } -
But it did not become 0 because of a race condition between _dl_lookup_symbol_x and _dl_map_object_deps like below:
[Thread A] <_dl_lookup_symbol_x+6369>: movl r16=0x80000000000;; <_dl_lookup_symbol_x+6384>: [MMI] adds r15=784,r15;; <_dl_lookup_symbol_x+6385>: ld8 r14=[r15] ------ (A-1) load the bit fields <_dl_lookup_symbol_x+6386>: nop.i 0x0;; <_dl_lookup_symbol_x+6400>: [MMI] or r14=r16,r14;; ------ (A-2) set l_used bit (A-3) interrupted or stalled <_dl_lookup_symbol_x+6401>: st8 [r15]=r14 --------- (A-4) store the bit fields <_dl_lookup_symbol_x+6402>: mov r14=1028 <_dl_lookup_symbol_x+6416>: [MMI] ld4 r15=[r41];; <_dl_lookup_symbol_x+6417>: and r14=r15,r14 [Thread B] <_dl_map_object_deps+3088>: [MMI] ld8 r14=[r15];; ------- (B-1) load the bit fields <_dl_map_object_deps+3089>: nop.m 0x0 <_dl_map_object_deps+3090>: dep r14=0,r14,37,2;; -- (B-2) clear l_reserved bit <_dl_map_object_deps+3104>: [MMI] st8 [r15]=r14 --------- (B-3) store the bit fields <_dl_map_object_deps+3105>: ld8 r41=[r16] <_dl_map_object_deps+3106>: nop.i 0x0;; steps | [Thread A]| [Thread B] | l_reserved value in the memory ------+-----------+------------+---------------------------- 1 |(A-1) | | 1 2 | |(B-1) | 1 3 |(A-2) | | 1 4 | |(B-2) | 1 5 |(A-3) | | 1 6 | |(B-3) | 0 7 |(A-4) | | 1 -
As a result, the l_reserved remained 1 even after _dl_map_object_deps().
- A library with its link_map having nonzero l_reserved field will not be added to the l_searchlist of other libraries at all hereafter, resulting unresolved dependencies for the library.
Environment
- Red Hat Enterprise Linux 5.3
-
Red Hat Enterprise Linux 4.4
-
glibc-2.5-34
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
