Running on EC2 -- /var/lib/pulp on S3?
My client has requested that we run Satellite on EC2. The partition sizes are fairly straight forward. What I want to know is, is anyone using FUSE to mount an S3 bucket for /var/lib/pulp?
For that matter, is anyone using FUSE to mount S3 at all?
For the record, I'm thinking the answer is "no, don't do it," but I know my client will google this project: https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon and I'd like to refute this if it is truly a bad idea.
Responses
Why would you use such an indirect method? On EC2, you can have Elastic Block Store (EBS) volumes, which are certainly a more efficient way to access S3 storage from within EC2.
s3fs-fuse might be appropriate if you need to set up a way to move data from non-cloud systems to S3 and/or vice versa, but using it from within EC2 seems... circuitous.
Personally, I wouldn't even necessarily opt for EBS - elastic or legacy modes - when EFS is available. With EBS - even using the elastic extensions - you're still chasing volume-sizes. With EFS, it's wholly pay for what you consume. Plus, EFS makes it stupid-easy to create cross-AZ designs for scalability and/or resiliency.
Historically, the various S3-as-block projects haven't been reliably maintained. If your customer is concerned about paying for unused pre-allocated storage, EFS is a pretty good alternative.
The Satellite 6.1/6.2 documentation specifically recommends that /var/lib/pulp/ be on low-latency storage (and based on experience, I think it should be considered a strong recommendation). The specifically recommend agains NFS.
Based on that, I would guess that the order, from best to worst, would be: EBS (make sure you use LVM for easy expansion later, if needed), EFS, and (in a distant 3rd place), S3-over-FUSE. But a definite answer would probably require benchmarking each option (using actual Satellite 6.x tasks like "re-sync all RHEL6 and RHEL7 base repos" or "publish new Content View containing RHEL6-server & -optional repos"--not synthetic benchmarks like iometer). If your Satellite environment is small enough (only a dozen or two clients, one or a very few Organizations, relatively few Content Views that need to be re-published), the difference may not matter. For a large/complex environment, I suspect you would want EBS on SSD.
Possible penny-pinching option: use EBS for /var/lib/pulp, but mount /var/lib/pulp/content/ from cheaper/slower storage. The docs don't admit this, but that is the subdirectory where the bulky and less latency-sensitive package files live (I had a working Satellite 6.0.x or 6.1 beta system with /var/lib/pulp/content mounted over NFS, from a not-very-fast server). Other parts of /var/lib/pulp/ will generate insane amounts of I/O when e.g. publishing Content Views, and should stay on the lowest-latency storage available.
If he's thinking S3, that generally means an EC2 solution. In that context, best options for bandwidth- or latency-sensitive workloads tend to be (in descending order):
- instance storage
- EBS on privisioned-IOPS (io1)
- EBS on SSD (gp2)
Thing to remember with EBS is that:
- io1 volumes only achieve high performance as you scale the volume size. At lower volume-sizes, a given volume isn't especially performant over other EBS types.
- gp2, while notionally high-performance, doesn't guarantee a given performance-profile over time
Instance-storage is very performant (why some applications markedly benefit from the I3 instance-types). Primary down-side is that it's ephemeral: if you stop your instance, data on those volumes goes poof.
With respect to NFS-suitable components, EFS isn't necessarily "slow". Its performance is tied to the type and size of the EC2 instance acting as an NFS client. If you're using an m4-generation instance-type, the networking is between 1Gbps and 20Gbps. The speed-variability depends on which drivers you're using and the sizing of the instance (m4.large you're looking at a skosh over 1Gbps sustained throughput; m4.4xlarge you're at just about 10Gbps of sustained throughput). Another thing to remember about EFS is that, unlike instance-storage or EBS volumes, data is automatically replicated to all AZs in a region. So, if you have an instance-failure or even an AZ failure, quick recovery is easy to configure.
S3's technically a network-based storage-method. While the endpoint/S3 control-plane is designed to be performant, the overhead of the HTTP-based actions encapsulated within an user-space driver means you're just as network-bound, performance-wise, as NFS ...you just tack on some further overhead for the HTTP-based transfers and the user-space storage-API translations.
All in all, probably worth breaking out the storage benchmarking tools, though. =)
With respect to LVM for easy expansion... That's more true for use-cases where you're concatenating EBSes rather than using an EBS elastic volume. This is particularly so for a RHEL6-based host with the standard 2.6 kernel. Within the overall EBS elastic volume context, "easy" expandability can be had by using unpartitioned block devices. Basically, make the OS rescan the block device so that it's aware of the EBS's new geometry then use your chosen filesystem's resize options to grow to the new volume-size. With LVM active on a device, the expanded EBS elastic volume is kept in opened state. That opened state can interfere with the ability to re-read the device's new geometry.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
