Event logs reporting "ETL service sampling has encountered an error. Please consult the service log for more details."
Environment
- Red Hat Enterpise Virtualization (RHEV) 3.5
Issue
- Upgraded from RHEV-M version 3.4 to 3.5.3
- The RHEV-M event logs is reporting the following error message:
ETL service sampling has encountered an error. Please consult the service log for more details.
Resolution
- Have a current user access the console of VM in question from the RHEV-M GUI. This will update the 'console_user_id' of the VM with the new user_id.
- To determine the VM, please refer to the Diagnostic Steps below and/or contact Red Hat Technical Support for assistance.
- BZ 1270105 was added to correct this behaviour and is addressed in rhevm-dwh 3.5.5.
Root Cause
The RHEV-M engine database contains a VM that references a user_id that no longer exists. This user had connected to the console of this VM. This user was then removed prior to the upgrade and no other user has since accessed the console of the VM.
-[ RECORD 1 ]----------+--------------------------------------------------------------------------------------------------------------------
vm_guid | 5bb006af-57d7-419b-bfe0-adeefb5a5134
status | 1
vm_ip | XX.XX.XXX.XX
vm_host | VM_HOST
vm_pid |
last_start_time | 2013-07-18 17:54:50.793-04
guest_cur_user_name | None
guest_last_login_time |
guest_last_logout_time |
guest_os | 2.6.32-504.23.4.el6.x86_64
run_on_vds | 0c8f89b7-c4a7-434e-94e5-104637aa8ef9
migrating_to_vds |
app_list | kernel-2.6.32-431.29.2.el6,kernel-2.6.32-504.el6,rhevm-guest-agent-common-1.0.10-2.el6ev,kernel-2.6.32-504.23.4.el6
display | 5908
acpi_enable | t
session | 0
display_ip | XX.XX.XXX.XX
display_type | 0
kvm_enable | t
display_secure_port | -1
utc_diff | 0
last_vds_run_on |
client_ip |
guest_requested_memory |
hibernation_vol_handle |
boot_sequence | 0
exit_status | 0
pause_status | 0
exit_message |
hash | 7354872742116401709
console_user_id | df83832b-485b-4af0-b0bd-ed8d724bf46c <<========
guest_agent_nics_hash | -53872956
console_cur_user_name |
last_watchdog_event |
last_watchdog_action |
is_run_once | f
vm_fqdn | ABCDEFG.HIJK.com
cpu_name |
last_stop_time |
current_cd |
guest_cpu_count | 1
reason |
exit_reason | -1
Diagnostic Steps
- The following messages is logged in the ovirt-engine-dwh log file every 15 minutes:
2015-09-11 15:37:49|ETL Service Started
Exception in component tJDBCOutput_5
org.postgresql.util.PSQLException: ERROR: insert or update on table "vm_samples_history" violates foreign key constraint "vm_samples_history_current_user_id_fkey"
Detail: Key (current_user_id)=(df83832b-485b-4af0-b0bd-ed8d724bf46c) is not present in table "users_details_history".
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:321)
at ovirt_engine_dwh.statisticssync_3_5.StatisticsSync.tJDBCInput_10Process(StatisticsSync.java:10482)
at ovirt_engine_dwh.statisticssync_3_5.StatisticsSync$6.run(StatisticsSync.java:17437)
2015-09-11 15:38:03|ZdAtsf|REF9E9|of8mCx|OVIRT_ENGINE_DWH|StatisticsSync|Default|6|Java Exception|tJDBCOutput_5|org.postgresql.util.PSQLException:ERROR: insert or update on table "vm_samples_history" violates foreign key constraint "vm_samples_history_current_user_id_fkey"
Detail: Key (current_user_id)=(df83832b-485b-4af0-b0bd-ed8d724bf46c) is not present in table "users_details_history".|1
Exception in component tJDBCOutput_7
- The postgres logs reports the following messages:
STATEMENT: INSERT INTO host_samples_history (history_datetime,host_id,host_status,minutes_in_status,memory_usage_percent,ksm_shared_memory_mb,cpu_usage_percent,ksm_cpu_percent,active_vms,total_vms,total_vms_vcpus,cpu_load,system_cpu_usage_percent,user_cpu_usage_percent,swap_used_mb,host_configuration_version) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16)
ERROR: insert or update on table "vm_samples_history" violates foreign key constraint "vm_samples_history_current_user_id_fkey"
DETAIL: Key (current_user_id)=(df83832b-485b-4af0-b0bd-ed8d724bf46c) is not present in table "users_details_history".
STATEMENT: INSERT INTO vm_samples_history (history_datetime,vm_id,vm_status,minutes_in_status,cpu_usage_percent,memory_usage_percent,user_cpu_usage_percent,system_cpu_usage_percent,vm_ip,vm_client_ip,current_user_id,user_logged_in_to_guest,currently_running_on_host,vm_configuration_version,current_host_configuration_version) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15)
ERROR: current transaction is aborted, commands ignored until end of transaction block
STATEMENT: INSERT INTO vm_disk_samples_history (history_datetime,vm_disk_id,image_id,vm_disk_status,minutes_in_status,vm_disk_actual_size_mb,read_rate_bytes_per_second,read_latency_seconds,write_rate_bytes_per_second,write_latency_seconds,flush_latency_seconds,vm_disk_configuration_version) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12)
- The user with user id 'df83832b-485b-4af0-b0bd-ed8d724bf46c' does not exist in the RHEV-M engine or ovirt-engine-dwh database.. The user with this user id was removed prior to the upgrade of RHEV-M. Please contact Red Hat Technical Support for details about these commands.
- Further diagnostics requires to dump the all the tables of the engine database, then searching for user id 'df83832b-485b-4af0-b0bd-ed8d724bf46c' . Please contact Red Hat Technical Support for details about these commands.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments