swift container reports "unable to open database file" in Red Hat OpenStack Platform

Solution In Progress - Updated -

Issue

swift container reports "unable to open database file" in Red Hat OpenStack Platform

It seems that the database files cannot be accessed:

Sep  5 19:51:06 controller01 container-server: ERROR __call__ error
with PUT /sdc/121/AUTH_<UUID>/measure/4<UUID2>/<UUID3>_20180823_20%3A20%3A52
:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/swift/container/server.py",
line 604, in __call__
    res = getattr(self, req.method)(req)
  File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line
1591, in _timing_stats
    resp = func(ctrl, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/swift/container/server.py",
line 376, in PUT
    req.headers.get('x-meta-timestamp'))
  File "/usr/lib/python2.7/site-packages/swift/container/backend.py",
line 449, in put_object
    self.put_record(record)
  File "/usr/lib/python2.7/site-packages/swift/common/db.py", line
581, in put_record
    self._commit_puts([record])
  File "/usr/lib/python2.7/site-packages/swift/common/db.py", line
619, in _commit_puts
    self.merge_items(item_list)
  File "/usr/lib/python2.7/site-packages/swift/container/backend.py",
line 891, in merge_items
    return _really_merge_items(conn)
  File "/usr/lib/python2.7/site-packages/swift/container/backend.py",
line 873, in _really_merge_items
    for rec in to_add.values()))
OperationalError: unable to open database file (txn:
txe<UUID TX>)

Sep  5 19:51:06 controller01 container-server: 192.168.2.72 - -
[05/Sep/2018:19:51:06 +0000] "PUT
/sdc/121/AUTH_<UUID>/measure/4<UUID2>/<UUID3>_20180823_20%3A20%3A52"
500 1101 "PUT http://controller20.domain:13808/sdc/611/AUTH_<UUID>/measure/4<UUID2>/<UUID3>_20180823_20%3A20%3A52"
"txe<UUID TX>" "object-updater
983704,object-server 4141" 0.0342 "-" 804753 0
Sep  5 19:51:06 controller01 container-server: ERROR reading db
/srv/node/sdc/containers/470/88e/<UUID4>/<UUID4>.db:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/swift/common/db_replicator.py",
line 478, in _replicate_object
    now - (self.reclaim_age * 2))
  File "/usr/lib/python2.7/site-packages/swift/common/db.py", line
833, in reclaim
    ''', (sync_timestamp,))
  File "/usr/lib/python2.7/site-packages/swift/common/db.py", line
134, in execute
    self.timeout, self.db_file, lambda: sqlite3.Cursor.execute(
  File "/usr/lib/python2.7/site-packages/swift/common/db.py", line 72,
in _db_timeout
    return call()
  File "/usr/lib/python2.7/site-packages/swift/common/db.py", line
135, in <lambda>
    self, *args, **kwargs))
OperationalError: unable to open database file

The code is in /usr/lib/python2.7/site-packages/swift/common/db.py

124 class GreenDBCursor(sqlite3.Cursor):
125     """SQLite Cursor handler that plays well with eventlet."""
126
127     def __init__(self, *args, **kwargs):
128         self.timeout = args[0].timeout
129         self.db_file = args[0].db_file
130         super(GreenDBCursor, self).__init__(*args, **kwargs)
131
132     def execute(self, *args, **kwargs):
133         return _db_timeout(
134             self.timeout, self.db_file, lambda: sqlite3.Cursor.execute(
135                 self, *args, **kwargs))

Logs are full of:

      1 Sep  5 19:45:04 controller2 container-server: ERROR reading
db /srv/node/sdc/containers/113/6ef/<UUID6>/<UUID6>.db
      1 Sep  5 19:45:04 controller2 container-server: ERROR reading
db /srv/node/sdc/containers/121/eb4/<UUID5>/<UUID5>.db

Environment

Red Hat OpenStack Platform 10

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content