While upgrading OpenShift Enterprise from 1.1 to 1.2 fails with error - "ERROR: Migration failed for User User-name" , "E11000 duplicate key error index: openshift_broker_dev.domains.$canonical_namespace_1"

Solution Verified - Updated -

Issue

  • In OpenShift Enterprise 1.1, it has been seen numerous times where two users can both create the same namespace or have multiple namespaces. This has created inconsistencies in the database on numerous occasions.

  • This has been remedied in the past by clearing the database manually

  • Recently, during an upgrade from OSE 1.1 to 1.2, the issue occured again, with the following in the logs:

ERROR: Migration failed for User User-name: The operation: #<Moped::Protocol::Command
  @length=84
  @request_id=53
  @response_to=0
  @op_code=2004
  @flags=[]
  @full_collection_name="openshift_broker_dev.$cmd"
  @skip=0
  @limit=-1
  @selector={:getlasterror=>1, :safe=>true}
  @fields=nil>
failed with error 11000: "E11000 duplicate key error index: openshift_broker_dev.domains.$canonical_namespace_1  dup key: { : \"paastest\" }"

See https://github.com/mongodb/mongo/blob/master/docs/errors.md
for details about this error.
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:78:in `block in command'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:564:in `[]'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:564:in `block (3 levels) in flush'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:563:in `map'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:563:in `block (2 levels) in flush'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:129:in `ensure_connected'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:559:in `block in flush'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:574:in `logging'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:558:in `flush'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:307:in `pipeline'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/session/context.rb:58:in `block in insert'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/session/context.rb:109:in `block in with_node'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cluster.rb:205:in `block in with_primary'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/node.rb:189:in `ensure_primary'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cluster.rb:204:in `with_primary'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/session/context.rb:108:in `with_node'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/session/context.rb:56:in `insert'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/collection.rb:91:in `block in insert'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/session.rb:312:in `with'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/collection.rb:90:in `insert'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/persistence/operations/insert.rb:27:in `block in persist'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/persistence/insertion.rb:25:in `block (2 levels) in prepare'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:425:in `_run__2343854526983121115__create__292201033992947336__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_create_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/callbacks.rb:114:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/persistence/insertion.rb:24:in `block in prepare'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:403:in `_run__2343854526983121115__save__292201033992947336__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/callbacks.rb:114:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/persistence/insertion.rb:23:in `prepare'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/persistence/operations/insert.rb:26:in `persist'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/persistence.rb:50:in `insert'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/persistence.rb:79:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.16.1/app/models/domain.rb:63:in `save'
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/persistence.rb:95:in `save!'
/usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/1/data/migrate-mongo-2.0.23:147:in `create_domains'
/usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/1/data/migrate-mongo-2.0.23:331:in `migrate_user'
/usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/1/data/migrate-mongo-2.0.23:369:in `block in user_phase1'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:78:in `block in each'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `block in each'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/cursor.rb:26:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:77:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/moped-1.3.2/lib/moped/query.rb:77:in `each'
/usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/1/data/migrate-mongo-2.0.23:367:in `user_phase1'
/usr/lib/ruby/site_ruby/1.8/ose-upgrade/broker/upgrades/1/data/migrate-mongo-2.0.23:468:in `<main>'
jpala
Time to run phase 1 (5 users processed) - 0.063068127
***************************
Migration Done! - 1 failures
Failed user logins:
User-name

*****************************************************
  • During the upgrade, the user was simply deleted as it was deemed unnecessary.

  • How can we prevent this issue during an upgrade?

Environment

  • Openshift Enterprise (OSE)
  • Upgrade from 1.1 to 1.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.