A system slows down and tcsh consumes a lot of memory due to a corrupted .history file

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 2.1
  • Red Hat Enterprise Linux 3
  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • RHEL 5: tcsh-6.14-14.el5
  • RHEL 6: tcsh-6.17-8.el6
  • RHEL 7: tcsh-6.18.01-15.el7

Issue

  • The data in the .history file becomes malformed and its size gets larger and larger.
  • The system slows down because csh uses a lot of memory to read such a large .history file.

Resolution

  • This issue has not been addressed on Red Hat Enterprise Linux 4 and earlier.
  • The file locking issue has been solved in RHEL 5, 6 and 7:
    • tcsh617-6.17-7.el5 has been released in RHEL 5.9 with Advisory RHBA-2013:0052
      Note: we released errata only for the tcsh617 package for RHEL 5. If you're using the package based on tcsh 6.14, you need to switch to tcsh617.
    • tcsh-6.17-19.el6_2has been released in RHEL 6.2 with Advisory RHBA-2012:0687 *
    • tcsh-6.17-38.el6has been released in RHEL 6.10 to fix a regression, with Advisory RHBA-2017:0731
    • tcsh-6.18.01-7.el7 has been released in RHEL7 GA
  • The high memory consumption due to a corrupted .history has been fixed in RHEL 7:
    • tcsh-6.18.01-16.el7has been released in RHEL 7.8 with Advisory RHBA-2020:1177
    • tcsh-6.18.01-17.el7_9.1 has been released in RHEL 7.9 to fix a regression, with Advisory RHBA-2020:5001

Workaround:

Add a line containing this command to ~/.cshrc or type it on the command line:

set savehist=

This will prevent any writing to ~/.history file (even the empty file still gets truncated every time) thus there won't be any delay in running scripts and no bug will show up.

Root Cause

  • The first cause of this issue was that tcsh did not handle ~/.history file exclusively. The "merge" option causes possibility a little more unexpected behavior with warning mentioned in man page regarding the "-S" built-in command:

    history [-hTr] [n]
history -S|-L|-M [filename] (+)
history -c (+)
    The first form prints the history event list.  If n is given
    only the n most recent events are printed or saved.  With -h,
    the history list is printed without leading numbers.  If -T is
    specified, timestamps are printed also in comment form.  (This
    can be used to produce files suitable for loading with 'history
    -L' or 'source -h'.)  With -r, the order of printing is most
    recent first rather than oldest first.

    With -S, the second form saves the history list to filename.
    If the first word of the savehist shell variable is set to a
    number, at most that many lines are saved.  If the second word
    of savehist is set to `merge', the history list is merged with
    the existing history file instead of replacing it (if there is
    one) and sorted by time stamp.  (+) Merging is intended for an
    environment like the X Window System with several shells in
    simultaneous use.  If the second word of savehist is `merge'
    and the third word is set to `lock', the history file update
    will be serialized with other shell sessions that would
    possibly like to merge history at exactly the same time.

  • Additionally, both a value greater than 0 and merge option are set to savehist on csh and this is the default setting in Red Hat Enterprise Linux 5.4 or later:

    $ echo $savehist  
savehist (1024 merge)

  • The second cause of the issue was that tcsh did not impose a limit on the length of a line in the history file, which could lead to high memory consumption.

Diagnostic Steps

  • Each command history should be recorded in the file with a timestamp line a command entry line which should end with EOL(End Of Line).

  • Example of normal .history file:

    $ cat -n .history  
      1 #+1289787344  
      2 set
[...]

  • A timestamp and a run command entry are not recorded by turns. Not a run command entry but a timestamp is recorded unexpectedly (see the line 4). Additionally, some entries are merged unexpectedly (see the line 6 and 7 below).

  • Example of malformed .history file :

    $ cat -n .history  
      1 #+1289787344  
      2 test  
      3 #+1289787367  
      4 1289787366  
      5 #+1289787367  
      6 128978\#+12897\#+1289\#+12897test  
      7 #+1289\#+12897testls\#\#+1289787401l12\#+1289787402  
      8 st
[...]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments