Greetings,

In deploying a large RHN 6.0/6.1 Beta environment and troubleshooting we've stumbled upon the foreman database and found it useful for gaining some insight into an otherwise foggy window on the status/health of satellite. That said, is there any documentation or further insight into the foreman database tables, and their use inside satellite.

One such example is when navigating to Content -> Content View -> Versions, none of the content views we have update, sit at the spinning wait wheel. We've waited some 90 minutes to ensure it isn't just thinking, and found in the foreman production.log errors realted to:

2015-07-27 07:03:49 [I] Processing by Katello::Api::V2::ContentViewVersionsController#index as JSON
2015-07-27 07:03:49 [I]   Parameters: {"content_view_id"=>"9", "page"=>"1", "search"=>"", "api_version"=>"v2"}
2015-07-27 07:03:49 [I]   Rendered /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/views/katello/api/v2/content_view_versions/index.json.rabl within katello/api/v2/layouts/collection (224.3ms)
2015-07-27 07:03:49 [E] NoMethodError: undefined method `pending' for nil:NilClass
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/content_view_version.rb:80:in `block in active_history'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/associations/collection_association.rb:82:in `block in select'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/associations/collection_association.rb:82:in `select'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/associations/collection_association.rb:82:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/associations/collection_association.rb:82:in `select'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/associations/collection_proxy.rb:46:in `select'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/content_view_version.rb:80:in `active_history'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/helpers.rb:13:in `data_object'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:103:in `child'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:46:in `block in compile_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:45:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:45:in `compile_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:21:in `block in build'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:166:in `cache_results'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:20:in `build'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:51:in `to_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:39:in `block in render'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:285:in `cache_results'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:39:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/partials.rb:27:in `object_to_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/partials.rb:14:in `partial'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:124:in `extends'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:34:in `block in compile_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:33:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:33:in `compile_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:21:in `block in build'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:166:in `cache_results'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:20:in `build'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:53:in `block in to_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/delegation.rb:6:in `map'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/relation/delegation.rb:6:in `map'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:53:in `to_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:39:in `block in render'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:285:in `cache_results'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:39:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/partials.rb:27:in `object_to_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:108:in `child'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:46:in `block in compile_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:45:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:45:in `compile_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:21:in `block in build'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:166:in `cache_results'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/builder.rb:20:in `build'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:51:in `to_hash'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:63:in `to_json'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:39:in `block in render'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:285:in `cache_results'
/opt/rh/ruby193/root/usr/share/gems/gems/rabl-0.9.0/lib/rabl/engine.rb:39:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/views/katello/api/v2/content_view_versions/index.json.rabl:1:in `__opt_rh_ruby____root_usr_share_gems_gems_katello__________app_views_katello_api_v__content_view_versions_index_json_rabl___72845328717699318_131789540'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:145:in `block in render'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:125:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/template.rb:143:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:47:in `block (2 levels) in render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in `block in instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/abstract_renderer.rb:38:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:46:in `block in render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:54:in `render_with_layout'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:45:in `render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/template_renderer.rb:18:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:36:in `render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_view/renderer/renderer.rb:17:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:110:in `_render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/streaming.rb:225:in `_render_template'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:103:in `render_to_body'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/renderers.rb:28:in `render_to_body'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/compatibility.rb:50:in `render_to_body'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:88:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:16:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:40:in `block (2 levels) in render'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/core_ext/benchmark.rb:5:in `block in ms'
/opt/rh/ruby193/root/usr/share/ruby/benchmark.rb:295:in `realtime'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/core_ext/benchmark.rb:5:in `ms'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:40:in `block in render'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:83:in `cleanup_view_runtime'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/controller_runtime.rb:24:in `cleanup_view_runtime'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:39:in `render'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/katello/api/v2/rendering.rb:48:in `respond_with_template'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/katello/api/v2/rendering.rb:63:in `respond_with_template_collection'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/katello/api/v2/rendering.rb:76:in `try_specific_collection_template'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/katello/api/v2/rendering.rb:11:in `respond_for_index'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/controllers/katello/concerns/api/api_controller.rb:94:in `respond'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/controllers/katello/api/v2/content_view_versions_controller.rb:42:in `index'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:167:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rendering.rb:10:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:18:in `block in process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:528:in `block (4 levels) in _run__3770000441734492357__process_action__207940387653458120__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_12128'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:151:in `disable_json_root'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_12128'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:494:in `block (3 levels) in _run__3770000441734492357__process_action__207940387653458120__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_12127'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:in `around'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in `_callback_around_3675'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_12127'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:482:in `block (2 levels) in _run__3770000441734492357__process_action__207940387653458120__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_12126'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:33:in `clear_thread'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_12126'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:426:in `block in _run__3770000441734492357__process_action__207940387653458120__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:215:in `block in _conditional_callback_around_12125'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:326:in `around'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:310:in `_callback_around_13'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:214:in `_conditional_callback_around_12125'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:414:in `_run__3770000441734492357__process_action__207940387653458120__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/callbacks.rb:17:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rescue.rb:29:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `block in instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/notifications.rb:123:in `instrument'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/instrumentation.rb:29:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/controllers/katello/concerns/api/api_controller.rb:59:in `process_action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/base.rb:121:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/abstract_controller/rendering.rb:45:in `process'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:203:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_controller/metal.rb:246:in `block in action'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:36:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/mapper.rb:42:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/journey-1.0.4/lib/journey/router.rb:56:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/routing/route_set.rb:600:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/apipie-rails-0.2.5/lib/apipie/extractor/recorder.rb:97:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/apipie-rails-0.2.5/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/conditionalget.rb:25:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/head.rb:14:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
/usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/flash.rb:242:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/cookies.rb:339:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/query_cache.rb:64:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:473:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `_run__3249270323823888912__call__3038660194166812646__callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/lib/katello/middleware/silenced_logger.rb:29:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/request_id.rb:22:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/actionpack-3.2.8/lib/action_dispatch/middleware/static.rb:62:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/builder.rb:134:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:64:in `block in call'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:49:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:49:in `call'
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2015-07-27 07:03:49 [I] Completed 500 Internal Server Error in 510ms (Views: 0.4ms | ActiveRecord: 110.0ms)

Which lead us to the various tables katello_content_view_versions and katello_content_view_histories from the referencing the the backtrace similar to those like:

2015-07-27 07:03:49 [E] NoMethodError: undefined method `pending' for nil:NilClass
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/models/katello/content_view_version.rb:80:in `block in active_history'

Which seem to make this DBA at least wonder if there isn't some "pending" value of a column in content_view_version or active_history ... this lead to finding the katello_content_view_histories table and we saw:

foreman=# select * from katello_content_view_histories where status ilike '%progress' order by updated_at desc;
 id  | katello_content_view_version_id | katello_environment_id |               task_id                |  user   |   status    | notes |         created_at         |         updated_at         
-----+---------------------------------+------------------------+--------------------------------------+---------+-------------+-------+----------------------------+----------------------------
 101 |                              49 |                     15 | 8f1bfff2-207e-4918-bebb-f27d08c63d5c | bscalio | in progress |       | 2015-07-21 16:09:43.265817 | 2015-07-21 16:09:43.265817
 100 |                              46 |                     15 | 8f1bfff2-207e-4918-bebb-f27d08c63d5c | bscalio | in progress |       | 2015-07-21 16:09:43.188698 | 2015-07-21 16:09:43.188698
  99 |                              40 |                     15 | 8f1bfff2-207e-4918-bebb-f27d08c63d5c | bscalio | in progress |       | 2015-07-21 16:09:43.117741 | 2015-07-21 16:09:43.117741
  98 |                              47 |                     26 | 8f1bfff2-207e-4918-bebb-f27d08c63d5c | bscalio | in progress |       | 2015-07-21 16:09:42.728597 | 2015-07-21 16:09:42.728597
  97 |                              46 |                     26 | 8f1bfff2-207e-4918-bebb-f27d08c63d5c | bscalio | in progress |       | 2015-07-21 16:09:42.605265 | 2015-07-21 16:09:42.605265
  94 |                              49 |                     15 | ba24d947-f654-4448-82c6-e00b77cb6351 | bscalio | in progress |       | 2015-07-16 23:29:35.488391 | 2015-07-16 23:29:35.488391
  93 |                              46 |                     15 | ba24d947-f654-4448-82c6-e00b77cb6351 | bscalio | in progress |       | 2015-07-16 23:29:35.190117 | 2015-07-16 23:29:35.190117
  92 |                              40 |                     15 | ba24d947-f654-4448-82c6-e00b77cb6351 | bscalio | in progress |       | 2015-07-16 23:29:35.116406 | 2015-07-16 23:29:35.116406
  91 |                              47 |                     26 | ba24d947-f654-4448-82c6-e00b77cb6351 | bscalio | in progress |       | 2015-07-16 23:29:34.987611 | 2015-07-16 23:29:34.987611
  90 |                              46 |                     26 | ba24d947-f654-4448-82c6-e00b77cb6351 | bscalio | in progress |       | 2015-07-16 23:29:34.880032 | 2015-07-16 23:29:34.880032
  89 |                              49 |                     15 | bb712cd6-47fd-4341-bb08-beea7d9b0d1f | bscalio | in progress |       | 2015-07-15 13:12:13.67771  | 2015-07-15 13:12:13.67771
  88 |                              46 |                     15 | bb712cd6-47fd-4341-bb08-beea7d9b0d1f | bscalio | in progress |       | 2015-07-15 13:12:13.545379 | 2015-07-15 13:12:13.545379
  87 |                              40 |                     15 | bb712cd6-47fd-4341-bb08-beea7d9b0d1f | bscalio | in progress |       | 2015-07-15 13:12:13.406618 | 2015-07-15 13:12:13.406618
  86 |                              47 |                     26 | bb712cd6-47fd-4341-bb08-beea7d9b0d1f | bscalio | in progress |       | 2015-07-15 13:12:12.678219 | 2015-07-15 13:12:12.678219
  85 |                              46 |                     26 | bb712cd6-47fd-4341-bb08-beea7d9b0d1f | bscalio | in progress |       | 2015-07-15 13:12:12.539791 | 2015-07-15 13:12:12.539791
  84 |                              49 |                     15 | 6dffb2a8-1dd1-4126-9862-cd939fc751c9 | bscalio | in progress |       | 2015-07-15 13:09:08.547057 | 2015-07-15 13:09:08.547057
  83 |                              46 |                     15 | 6dffb2a8-1dd1-4126-9862-cd939fc751c9 | bscalio | in progress |       | 2015-07-15 13:09:08.147519 | 2015-07-15 13:09:08.147519
  82 |                              40 |                     15 | 6dffb2a8-1dd1-4126-9862-cd939fc751c9 | bscalio | in progress |       | 2015-07-15 13:09:08.063991 | 2015-07-15 13:09:08.063991
  81 |                              47 |                     26 | 6dffb2a8-1dd1-4126-9862-cd939fc751c9 | bscalio | in progress |       | 2015-07-15 13:09:07.883894 | 2015-07-15 13:09:07.883894
  80 |                              46 |                     26 | 6dffb2a8-1dd1-4126-9862-cd939fc751c9 | bscalio | in progress |       | 2015-07-15 13:09:07.694321 | 2015-07-15 13:09:07.694321
  72 |                              49 |                        | 83838ae9-a09d-4b3e-9ebe-42ec4104f713 | bscalio | in progress |       | 2015-06-16 20:43:35.369666 | 2015-06-16 20:43:35.369666
(21 rows)

So we opened a ticket with our TAM, this is 6.1 Beta via an upgrade path from 6.0.6 however I guess the general question is -- can we get any sort of detailed documentation on tables, use, foreman-rake katello:: options, etc to help gain insight and "unstick" the often sticky Web UI.

Note that using the hammer CLI content promotion and publish works, this is obviously just a Web UI issue.