Satellite 5.6: satellite-sync is very slow
Hi,
The initial sync for 'rhel-x86_64-server-6' from the channel content iso on a freshly installed Satellite 5.6, satellite-sync took 10 hours to finish, so I want to know is this normal or do I have something wrong?
# satellite-sync -m . -c rhel-x86_64-server-6
...
output truncated
...
01:31:56 Processing rpm packages complete
01:31:56
01:31:56 Importing package metadata
01:31:56 Importing relevant package metadata: rhel-x86_64-server-6 (11965)
________________________________________
Importing: ######################################## - complete
10:44:38
10:44:38 Linking packages to channels
10:46:12
10:46:12 Downloading errata data
10:46:12 Retrieving / parsing errata data: rhel-x86_64-server-6 (2224)
________________________________________
Downloading:######################################## - complete
10:46:39 Downloading errata data complete
10:46:39
10:46:39 Downloading kickstartable trees metadata
10:46:39 Retrieving / parsing kickstart data: rhel-x86_64-server-6 (7)
________________________________________
Downloading:######################################## - complete
10:46:39
10:46:39 Downloading kickstartable trees files
10:46:39 Retrieving / parsing kickstart tree files: rhel-x86_64-server-6 (853)
________________________________________
Downloading:######################################## - complete
10:49:22
10:49:22 Importing channel errata
10:49:56 Importing relevant errata: rhel-x86_64-server-6 (2224)
________________________________________
Downloading:######################################## - complete
11:06:38 Importing kickstartable trees (7)
11:06:53 Imported kickstartable trees (7)
Import complete:
Begin time: Sat Jun 14 00:47:28 2014
End time: Sat Jun 14 11:06:53 2014
Elapsed: 10 hours, 19 minutes, 24 seconds
Responses
Guru
6827 points
Hi Ahmed,
I too sync from isos, and depending on your system's horsepower, it may take a while. A while ago I built a satellite server for a customer, and initially I wasn't given a lot of resources, and when I added more ram and virtual cpus, it ran faster. There are other factors, but it can take a long time.
Have you synchronized the incremental channels yet? The latest incremental channels for rhel 6 and 5 are
- RHEL 6 (x86_64) + EUS + AUS + RHN Proxy/Tools + Supplementary (Incremental 2013-11-24:2014-05-25)
- RHEL 5 Client/Server (x86_64) + EUS + AMC + RHN Proxy/Tools + Supplementary (Incremental 2013-10-15:2014-05-18)
Kind Regards
Guru
6827 points
Ahmed, hopefully the remainder go faster (but it is the incremental channels, so less dvds). That is likely a good amount of ram and cpus. I remember running my base channel sync just before leaving for the day, but first I ran a "script -a /path/to/output.txt" and then the satellite sync.
Kind Regards,
Remmele
Guru
6222 points
Hi Ahmed and Remmele,
8GB of RAM sounds low to me, our company RHN Satellite has 92GB of RAM for the database (ok, RHN Satellite 5.5 with Oracle) demands a lot of RAM.
The test Satellite 5.6, a KVM guest with about 8GB is as slow as Ahmet describes.
So adding more RAM might help. I cannot test this, for my hypervisor has not enough memory to add more RAM to the guest.
Kind regards,
Jan Gerrit Kootstra
Guru
6827 points
Thanks Jan,
Jan, How many clients does your 92GB RAM system hold? I know when I added more RAM to that one customer's instance, the syncs did speed up (I mentioned it briefly above, but I didn't include the amounts).
I was looking for a chart a couple of days ago that I thought Red Hat provided regarding recommended RAM to support X amount of clients, the more clients, the more RAM. The only thing I could find was this, but I believe there is another resource somewhere that shows recommended RAM/resources based on clients.
Ahmed, are you able to add more RAM?
Kind Regards,
Remmele
I'm in the process of syncing rhel-i386-server-optional-6 which contains ~ 3000 packages and experience the same problems.
Our Satellite 5.6 runs on a virtual machine with 4 (2 GHz) vcpu and 8 GB RAM.
I had to abort and start over 3 times since it hang completely each time. Last attempt showing:
"14:34:04 Downloaded 6.02 GiB of 9.19 GiB. Estimated remaining time: 37 days, 9:03:51"
I don't know what processing is done to the downloaded .rpm:s , but the satellite-sync process uses 100% cpu (of 400), but only 2.3% memory.
At first it looks like a single-threaded application that maxes out the processor...but on the other hand I have no I/O wait at all so I suspect there is some other "internal race condition" causing this.
The virtual server has about 150MB MB free , 350 MB buffered and 4.8 GB cached memory, so if it is a memory issue then it has to be some limit set in the satellite-sync, unless we hit some magical internal OS limit.
It is possible that an increase of RAM would speed up things, but from the looks of monitoring the processes I'd say that a faster CPU would do the trick with the present satellite-sync.
On top of that, downloads from Red Hat CDN has always been very slow maxing out at around 2-5 Mbit/s, which of course adds to the total time for syncing a channel.
Was this issue ever resolved. I am having the same exact problem. Here are the specs on my host: (2) - Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 30 MB cache 2,700 MHz, 7.69 GB of RAM
Active Contributor
178 points
When the satellite-sync script is 'Diffing', (so more of an issue after initial sync) you may find that its spending most of its time in 'stat' -- ie. checking to see if a file exists (typically /var/satellite/redhat/....)
You can verify this with strace:
# pgrep satellite-sync
19794
# strace -p 19794 -c
Process 19794 attached
^CProcess 19794 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.64 0.016998 486 35 stat
0.26 0.000045 1 36 mmap
0.09 0.000016 0 72 lseek
0.00 0.000000 0 72 read
0.00 0.000000 0 36 open
0.00 0.000000 0 36 close
0.00 0.000000 0 108 fstat
0.00 0.000000 0 36 poll
0.00 0.000000 0 36 munmap
0.00 0.000000 0 72 rt_sigprocmask
0.00 0.000000 0 36 access
0.00 0.000000 0 36 sendto
0.00 0.000000 0 36 recvfrom
0.00 0.000000 0 72 fcntl
------ ----------- ----------- --------- --------- ----------------
100.00 0.017059 719 total
Verify what it's doing with stat:
# strace -p 19794 -e trace=stat -T
Process 19794 attached
stat("/var/satellite/redhat/NULL/1dd/mysql-bench/5.1.66-2.el6_3/x86_64/1ddc905fff118c7fb7aa8d8395222b7ec6d8acb88cd294d488fff74abfd11858/mysql-bench-5.1.66-2.el6_3.x86_64.rpm", {st_mode=S_IFREG|0644, st_size=437028, ...}) = 0 <0.094101>
stat("/var/satellite/redhat/NULL/9a8/mysql-server/5.1.66-2.el6_3/x86_64/9a89095c33504366080a825856190b193f18a08c9e1204e1b20ce91384fdfd42/mysql-server-5.1.66-2.el6_3.x86_64.rpm", {st_mode=S_IFREG|0644, st_size=9014336, ...}) = 0 <0.103244>
stat("/var/satellite/redhat/NULL/954/mysql/5.1.66-2.el6_3/x86_64/9549ba562f73f2681e55132f0d48403ede58e54b0398d6e5e861e6137e442ef8/mysql-5.1.66-2.el6_3.x86_64.rpm", {st_mode=S_IFREG|0644, st_size=906096, ...}) = 0 <0.077392>
stat("/var/satellite/redhat/NULL/e42/mysql-devel/5.1.66-2.el6_3/x86_64/e42afdffda9a6c45f34037da20aaa2094e07b4738f4c366bc547d52b54c8ac6c/mysql-devel-5.1.66-2.el6_3.x86_64.rpm", {st_mode=S_IFREG|0644, st_size=130624, ...}) = 0 <0.089786>
...^C
In my case, /var/satellite/redhat is pretty huge (> 300GB), and is backed by NFS from a NAS... which is not as fast for such operations as our previous one; nfsiostat reports an avg RTT of 32ms for reading. The filesystem is mounted as follows:
# nfsstat -m
/var/satellite from hcs-nfs-p01.otago.ac.nz:/ifs/444/system/critical/rhn-satellite/
Flags: rw,fscontext=system_u:object_r:spacewalk_data_t:s0,noatime,nodiratime,vers=4,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=REDACTED,minorversion=0,local_lock=none,addr=REDACTED
It's the only client for that directory, so I imagine there is something I could change with regard to client NFS cache settings (see nfs(5) section "DATA AND METADATA COHERENCE" -- Directory entry caching), but then again, it's cache would likely be cold for the type of crawling operation that it's doing.
You could speed this up by pre-warming the cache (assuming it fits).
find /var/satellite/redhat/ > /dev/null &
You could also look at cachefilesd
# yum info cachefilesd.x86_64
Loaded plugins: presto, rhnplugin, security, verify
This system is receiving updates from RHN Classic or RHN Satellite.
Available Packages
Name : cachefilesd
Arch : x86_64
Version : 0.10.2
Release : 3.el6
Size : 36 k
Repo : rhel-x86_64-server-6
Summary : CacheFiles userspace management daemon
URL : http://people.redhat.com/~dhowells/fscache/
License : GPL2+
Description : The cachefilesd daemon manages the caching files and directory that are
: that are used by network filesystems such a AFS and NFS to
: do persistent caching to the local disk.
Red Hat could probably rewrite satellite-sync to help improve this (doing IO asynchronously and in batches), but that's unlikely to happen for Satellite 5.x, I would think.
Cheers, Cameron
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.