'virt-manager' was killed by signal 6 (SIGABRT)

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6.6;

    • virt-manager-0.9.0-28.el6.x86_64
    • libvirt-client-0.10.2-46.el6_6.1.x86_64
    • libvirt-python-0.10.2-46.el6_6.1.x86_64
    • spice-gtk-0.22-7.el6.x86_64
    • glibc-2.12-1.149.el6.x86_64
    • glib2-2.28.8-4.el6.x86_64
    • python-2.6.6-52.el6.x86_64

Issue

  • 'virt-manager' was killed by signal 6 (SIGABRT), generating a process core file.

  • Four virtual machines were changed to "boot automatically when host is started" and then started. After the third was started, 'virt-manager' crashed.

Resolution

A private bug has been opened to track this.

Root Cause

It seems that this is similar to an upstream bug, in which a file is being closed while potentially being used by another component, resulting in a race condition.

Diagnostic Steps

The following is from a debug environment that was set up in order to analyse the process core file;

  • running gdb against the core file;
Core was generated by `/usr/bin/python /usr/share/virt-manager/virt-manager.py'.
Program terminated with signal 6, Aborted.
#0  0x00000039e0c32625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
  • backtrace (only a single thread);
(gdb) bt
#0  0x00000039e0c32625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00000039e0c33e05 in abort () at abort.c:92
#2  0x00000039e244b89a in g_logv (log_domain=0x39e48c9292 "GLib-GIO", log_level=<value optimized out>, format=0x39e48d58c0 "creating GSocket from fd %d: %s\n", args1=0x7f564afff5f0)
    at gmessages.c:557
#3  0x00000039e244b933 in g_log (log_domain=<value optimized out>, log_level=<value optimized out>, format=<value optimized out>) at gmessages.c:577
#4  0x00000039e4866625 in g_socket_details_from_fd (object=0x268c140) at gsocket.c:339
#5  g_socket_constructed (object=0x268c140) at gsocket.c:523
#6  0x00000039e3413c07 in g_object_newv (object_type=1, n_parameters=<value optimized out>, parameters=<value optimized out>) at gobject.c:1507
#7  0x00000039e341445d in g_object_new_valist (object_type=34855968, first_property_name=<value optimized out>, var_args=0x7f564afff970) at gobject.c:1596
#8  0x00000039e48514ce in g_initable_new_valist (object_type=34855968, first_property_name=0x39e48dca9a "fd", var_args=0x7f564afff970, cancellable=0x0, error=0x0) at ginitable.c:215
#9  0x00000039e48515cb in g_initable_new (object_type=<value optimized out>, cancellable=<value optimized out>, error=<value optimized out>, first_property_name=<value optimized out>)
    at ginitable.c:139
#10 0x0000003370a20520 in spice_channel_coroutine (data=0x213ce40) at spice-channel.c:2232
#11 0x0000003370a47e5b in coroutine_trampoline (cc=0x213cf00) at coroutine_ucontext.c:63
#12 0x0000003370a47e43 in continuation_trampoline (i0=<value optimized out>, i1=<value optimized out>) at continuation.c:55
#13 0x00000039e0c438f0 in ?? () from /lib64/libc-2.12.so
#14 0x000000000213d2c8 in ?? ()
#15 0x0000000000000000 in ?? ()
  • extract from 'bt full' for frame 4;
#4  0x00000039e4866625 in g_socket_details_from_fd (object=0x268c140) at gsocket.c:339
        fd = 28                           <<<<
        optlen = 4
        value = <value optimized out>
        bool_val = <value optimized out>
        address = {ss_family = 37792, __ss_align = 39837408, 
          __ss_padding = "p\367\377JV\177\000\000\260\247@\341\071\000\000\000\240\223@\341\071", '\000' <repeats 11 times>, "\020\325g\002\000\000\000\000\353XA\343\071", '\000' <repeats 27 times>"\370, \250B\343\003\000\000\000@\301h\002\000\000\000\000@OC\002", '\000' <repeats 19 times>}
        addrlen = <value optimized out>
        errsv = 9
  • abort() called here, in glogv();
539       if ((test_level & G_LOG_FLAG_FATAL) && !masquerade_fatal)
540             {
541 #ifdef G_OS_WIN32
542           gchar *locale_msg = g_locale_from_utf8 (fatal_msg_buf, -1, NULL, NULL, NULL);
543           
544           MessageBox (NULL, locale_msg, NULL,
(gdb) 
545               MB_ICONERROR|MB_SETFOREGROUND);
546           if (IsDebuggerPresent () && !(test_level & G_LOG_FLAG_RECURSION))
547         G_BREAKPOINT ();
548           else
549         abort ();
550 #else
551 #if defined (G_ENABLE_DEBUG) && defined (SIGTRAP)
552           if (!(test_level & G_LOG_FLAG_RECURSION))
553         G_BREAKPOINT ();
554           else
(gdb) 
555         abort ();
556 #else /* !G_ENABLE_DEBUG || !SIGTRAP */
557           abort ();                                   <<<<
558 #endif /* !G_ENABLE_DEBUG || !SIGTRAP */
559 #endif /* !G_OS_WIN32 */
560         }
  • g_socket_details_from_fd ();
308 g_socket_details_from_fd (GSocket *socket)
309 {

        ................

323   fd = socket->priv->fd;
324   optlen = sizeof value;
325   if (getsockopt (fd, SOL_SOCKET, SO_TYPE, (void *)&value, &optlen) != 0)
326     {
327       errsv = get_socket_errno ();
328 
329       switch (errsv)
330     {
331 #ifdef ENOTSOCK
332      case ENOTSOCK:
333 #endif
334 #ifdef WSAENOTSOCK
335      case WSAENOTSOCK:
336 #endif
337      case EBADF:                                        <<<<
338       /* programmer error */
339       g_error ("creating GSocket from fd %d: %s\n",
340            fd, socket_strerror (errsv));
  • virt-manager.log;
[Fr, 28 Nov 2014 13:21:21 virt-manager 12688] DEBUG (engine:1021) Starting vm 'web1'.
[Fr, 28 Nov 2014 13:21:21 virt-manager 12688] DEBUG (console:1150) Starting connect process for proto=spice trans=ssh connhost=hosting-test connuser=root connport=None gaddr=127.0.0.1 gport=5904 gsocket=None
[Fr, 28 Nov 2014 13:21:21 virt-manager 12688] DEBUG (console:109) Creating SSH tunnel: ssh -l root hosting-test sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then   CMD="nc -q 0 127.0.0.1 5904";else   CMD="nc 127.0
.0.1 5904";fi;eval "$CMD";'
[Fr, 28 Nov 2014 13:21:21 virt-manager 12688] DEBUG (console:132) Tunnel PID=14787 OUTFD=30 ERRFD=32
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:109) Creating SSH tunnel: ssh -l root hosting-test sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then   CMD="nc -q 0 127.0.0.1 5904";else   CMD="nc 127.0
.0.1 5904";fi;eval "$CMD";'
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:132) Tunnel PID=14792 OUTFD=29 ERRFD=37
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:109) Creating SSH tunnel: ssh -l root hosting-test sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then   CMD="nc -q 0 127.0.0.1 5904";else   CMD="nc 127.0
.0.1 5904";fi;eval "$CMD";'
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:132) Tunnel PID=14793 OUTFD=36 ERRFD=39
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:109) Creating SSH tunnel: ssh -l root hosting-test sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then   CMD="nc -q 0 127.0.0.1 5904";else   CMD="nc 127.0
.0.1 5904";fi;eval "$CMD";'
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:132) Tunnel PID=14794 OUTFD=38 ERRFD=41
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:109) Creating SSH tunnel: ssh -l root hosting-test sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then   CMD="nc -q 0 127.0.0.1 5904";else   CMD="nc 127.0
.0.1 5904";fi;eval "$CMD";'
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:132) Tunnel PID=14795 OUTFD=40 ERRFD=43
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:1061) Viewer connected
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:109) Creating SSH tunnel: ssh -l root hosting-test sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then   CMD="nc -q 0 127.0.0.1 5904";else   CMD="nc 127.0
.0.1 5904";fi;eval "$CMD";'
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:132) Tunnel PID=14801 OUTFD=42 ERRFD=45
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (engine:475) window counter decremented to 1
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:109) Creating SSH tunnel: ssh -l root hosting-test sh -c 'nc -q 2>&1 | grep -q "requires an argument";if [ $? -eq 0 ] ; then   CMD="nc -q 0 127.0.0.1 5904";else   CMD="nc 127.0
.0.1 5904";fi;eval "$CMD";'
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:132) Tunnel PID=14803 OUTFD=28 ERRFD=44                                             <<<<  trying to open FD 28
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:150) Shutting down tunnel PID=14787 OUTFD=30 ERRFD=32
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:150) Shutting down tunnel PID=14792 OUTFD=29 ERRFD=37
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:150) Shutting down tunnel PID=14793 OUTFD=36 ERRFD=39
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:150) Shutting down tunnel PID=14794 OUTFD=38 ERRFD=41
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:150) Shutting down tunnel PID=14795 OUTFD=40 ERRFD=43
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:150) Shutting down tunnel PID=14801 OUTFD=42 ERRFD=45
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:150) Shutting down tunnel PID=14803 OUTFD=28 ERRFD=44
[Fr, 28 Nov 2014 13:21:22 virt-manager 12688] DEBUG (console:1033) Viewer disconnected

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.