Are you looking for ways to deliver your applications more quickly and with less effort? Do you need to move more services to the cloud?
Red Hat has these same issues and goals. By using our own product OpenShift, we have been able to shorten our release cycle times and support our developers better.
We built and deployed our own OpenShift Enterprise Platform-as-a-Service (PaaS) offering and are running it in Amazon Web Services (AWS). This has allowed our teams to have more control over creating and developing their own applications in the cloud. OpenShift's automated workflows and tooling help our developers access what they need, when they need it.
We hope our success story can become your success story with OpenShift.
Case Study: IT OpenShift Enterprise
In 2013, Red Hat IT launched a new internal PaaS deployment, available to all associates and based on OpenShift Enterprise.
Before the launch of IT’s OpenShift Enterprise offering, IT did not distinguish much between major enterprise applications and smaller applications that do not require strict enterprise management and change control. Plus our release engineering tooling was falling behind the modern standards.
While management processes are valuable in many cases, our Red Hat associates were asking for a self-service, application-deployment platform for lighter workloads. At the same time, IT wanted to:
- shorten our release cycle times
- support developers
- expand our services to the cloud
Because we needed multiple application security zones and platform customizations (for some enterprise applications), we could not move into OpenShift Online.
IT's Cloud Enablement team (in collaboration with OpenShift Engineering, IT Operations, and Global Support Services (GSS)) built and deployed a 90-day, proof-of-concept OpenShift Engineering PaaS offering, running in AWS, within 5 weeks. Developers and teams began creating and managing their own applications immediately and moved us towards our goal of running more services in the cloud.
At the end of the proof-of-concept phase, IT delivered additional releases on the platform. IT also made it a production service, expanded the number of available security zones, and added other new features.
The IT OpenShift service currently supports 3 major security zones, each with infrastructure to support 3 different application sizes (small, medium, and large districts). These zones are spread across multiple AWS regions and accounts to ensure higher availability and security. The security zones offer different options depending on the intended customer base and application security requirements, including:
- internal/intranet only
- externally accessible
- IT development
The OpenShift Enterprise PaaS provides a high level of availability through load-balanced broker services, MongoDB replica sets and redundant components at every level. Individual application owners can now deliver application high-availablilty within the platform.
IT team members included both associates who had previously worked on OpenShift Online and associates who had never before touched OpenShift. To facilitate cooperation, the IT team worked closely with the OpenShift Engineering teams, some of whom were former IT team members.
This unique setup allowed for a rapid flow of information between IT and Engineering. The teams met on a regular basis while implementing the project. This collaborative environment made it easier to discuss and deliver feature requests, bugs, and patches.
As we began to form our OpenShift Enterprise support team for our external customers, GSS was able to work closely with IT and use this effort as a training ground for our support associates. Our internal partnership allowed Red Hat to serve our current and future customers even better.
OpenShift Enterprise 2.2
Red Hat® Enterprise Linux® 6.6
Red Hat Cloud Access for AWS
31 Amazon Elastic Compute Cloud (EC2) nodes in AWS for Openshift infrastructure (brokers + app nodes) | 7 EC2 nodes in AWS for supporting Openshift infrastructure | 1038 Red Hat associates logged into the system
1093 Applications deployed | 618 Domains | 11 Bugs opened with Engineering | 5 Feature requests opened with Engineering
- Access Labs, https://access.redhat.com/labs/
- Partner Enablement Training for JBoss BPMS6 and FSW
- SalesForce.com connector to sync our Partner Center Salesforce data to a SaaS (Software-as-a-Service) Learning Management System
- Conference room reservation mobile app
And many others . . . .
- Red Hat IT’s standard Nagios active monitoring, plus 5 custom scripts to monitor broker services.
- Red Hat IT’s standard Graphite (for data collection and trending)
- Red Hat IT’s standard MongoDB monitoring tools
- AWS CloudFormation Stacks (for provisioning EC2 infrastructure across availability zones in 3 Virtual Private Clouds (VPCs))
- Custom administrative scripts to manage new dynamic cloud services
The Lessons Learned
- Agile (Scrum) methodologies are effective for managing an OpenShift implementation.
- Building services in the cloud (AWS) is significantly different from building them in a datacenter. But with proper tooling and understanding, the cloud model offers many opportunities.
- Working directly with GSS and Engineering helped us make this project more visible throughout the company.
- Mojo can provide a good place to manage documentation and provide a support portal for our users.
- Developers waited to do most major user application development until the Production PaaS service was available. Users want to know their platform will be supported.
- A flexible and stable PaaS platform is now available to all Red Hat associates.
- OpenShift Enterprise allows us to customize the platform as needed.
- Developers can now code, test, and deploy applications directly into our internal OpenShift Enterprise system with minimal paperwork.
- We now have more services running in the cloud and more experience with the cloud model.
- We will soon be moving to OpenShift Enterprise v3.0, with Docker and Kubernetes technologies built in.
- We are also working to integrate Red Hat Atomic into our PaaS offerings.
Subject Matter Experts: Tom Benninger and Andrew Butcher
Product Owner: Hope Lynch, email@example.com (please copy on correspondence)Posted: 2015-05-15T19:53:33+00:00
Docker is a new technology within the past few years, but many know that linux containers have been around longer than that. With the excitement around containers and Docker, the Customer Portal team at Red Hat wanted to take a look at how we could leverage Docker to deploy our application. We hope to always iterate on how code gets released to our customers, and containers seemed like a logical step to speeding up our application delivery.
The method of defining our installed Drupal modules centered around RPM packaging. This process of bundling Drupal modules inside of RPMs made sense for aligning ourselves with the release processes of other workstreams. The issue with this process for Drupal is managing many RPMs (we use over 100 Drupal modules on the Customer Portal). As our developers began to take over more of the operational aspects of Drupal releases, the process of the RPM packaging didn't flow well with the rest of coding that our developers were maintaining. Packaging a Drupal module inside of an RPM adds that extra abstraction layer of versioning as well, so our developers didn't always know exactly which version with patches of a module may be installed.
Along with RPM packaging, there wasn't a unified way in which code was installed per environment. For our close-to-production and production environments, this centered around versioned RPM updates via puppet. However, for our CI environment, in order to speed up the application delivery there, contributed Drupal module installs had to be done manually by the developer (mainly to support integration testing in case that module needed to be removed quickly). Developer environments were also different in that they didn't always have a packaged RPM for delivering code, but instead the custom code repository would need to be directly accessible to the developer.
Delivering a Docker image
You can probably see how unwieldy juggling over one hundred modules with custom code can become. Our Drupal team wanted to deliver one artifact that defined the entire release of our application. Docker doesn't solve this simply by shoving all the code inside a container, but it does allow for an agreement to be established as to what our Drupal developers are responsible for delivering. The host that runs this container can have the tools it needs to run Drupal with the appropriate monitoring. The application developer can define the entirety of the Drupal application, all the way from installing Drupal core to supplying our custom code. Imagine drawing a circle around all the parts that existed on the host that define our Drupal app. This is essentially how our Docker image was formed.
Below is the Dockerfile we use to deploy Drupal inside a container. You might notice one important piece that is missing: the database! It is assumed that your database host exists outside this container and can be connected to from this application (it could even be another running container on the host).
Also note, this is mostly an image used for testing and spinning up environments quickly right now, but as we vet our continuous delivery process the structure of this file would likely change.
This is some basic start configuration. We use the rhel7 Docker image as a base for running Drupal. A list of published Red Hat containers can be found here.
# Docker - Customer Portal Drupal # # VERSION dev # Use RHEL 7 image as the base image for this build. # Depends on subscription-manager correctly being setup on the RHEL 7 host VM that is building this image # With a correctly setup RHEL 7 host with subscriptions, those will be fed into the docker image build and yum repos # will become available FROM rhel7:latest MAINTAINER Ben Pritchett
Here we install yum packages and drush. The yum cache is cleared on the same line as the install in order to save space between image snapshots.
# Install all the necessary packages for Drupal and our application. Immediately yum update and yum clean all in this step # to save space in the image RUN yum -y --enablerepo rhel-7-server-optional-rpms install tar wget git httpd php python-setuptools vim php-dom php-gd memcached php-pecl-memcache mc gcc make php-mysql mod_ssl php-soap hostname rsyslog php-mbstring; yum -y update; yum clean all # Still need drush installed RUN pear channel-discover pear.drush.org && pear install drush/drush
Supervisord can be used to manage several processes at once, since the Docker container goes away when the last foreground process exits. Also we have some file permissions changes to the appropriate process users.
# Install supervisord (since this image runs without systemd) RUN easy_install supervisor RUN chown -R apache:apache /usr/sbin/httpd RUN chown -R memcached:memcached /usr/bin/memcached RUN chown -R apache:apache /var/log/httpd
With Docker, you could run a configuration management tool inside the container. However this generally increases the scope of what your app needs. These PHP config options are the same for all environments, so they are simply changed in place here. A tool like Augeus would be helpful for making these line edits to configuration files that don't change often.
# we run Drupal with a memory_limit of 512M RUN sed -i "s/memory_limit = 128M/memory_limit = 512M/" /etc/php.ini # we run Drupal with an increased file size upload limit as well RUN sed -i "s/upload_max_filesize = 2M/upload_max_filesize = 100M/" /etc/php.ini RUN sed -i "s/post_max_size = 8M/post_max_size = 100M/" /etc/php.ini # we comment out this rsyslog config because of a known bug (https://bugzilla.redhat.com/1088021) RUN sed -i "s/$OmitLocalLogging on/#$OmitLocalLogging on/" /etc/rsyslog.conf
Here we add the makefile for our Drupal environment. This is the definition of where Drupal needs to be installed and with what modules/themes.
# Uses the drush make file in this Docker repo to correctly install all the modules we need # https://www.drupal.org/project/drush_make ADD drupal.make /tmp/drupal.make
Here we add some Drush config, and Registry Rebuild tool (which we have gotten a lot of value out of).
# Add a drushrc file to point to default site ADD drushrc.php /etc/drush/drushrc.php # Install registry rebuild tool. This is helpful when your Drupal registry gets # broken from moving modules around RUN drush @none dl registry_rebuild --nocolor
Finally, we install Drupal using drush.make. This takes a bit of time while all the modules are downloaded from Drupal.org.
# Install Drupal via drush.make. RUN rm -rf /var/www/html ; drush make /tmp/drupal.make /var/www/html --nocolor;
Some file permissions changes occur here. Notice we don't have a settings.php file; that will be added with the running Docker container.
# Do some miscellaneous cleanup of the Drupal file system. If certain files are volume linked into the container (via -v at runtime) # some of these files will get overwritten inside the container RUN chmod 664 /var/www/html/sites/default && mkdir -p /var/www/html/sites/default/files/tmp && mkdir /var/www/html/sites/default/private && chmod 775 /var/www/html/sites/default/files && chmod 775 /var/www/html/sites/default/files/tmp && chmod 775 /var/www/html/sites/default/private RUN chown -R apache:apache /var/www/html
Here we add our custom code, and link it into the appropriate location on the filesystem. Host name is filtered in this example ($INTERNAL_GIT_REPO). Another way to approach this would be to house the Dockerfile with our custom code to centralize things a bit.
# Pull in our custom code for the Customer Portal Drupal application RUN git clone $INTERNAL_GIT_REPO /opt/drupal-custom # Put our custom code in the appropriate places on disk RUN ln -s /opt/drupal-custom/all/themes/kcs /var/www/html/sites/all/themes/kcs RUN ln -s /opt/drupal-custom/all/modules/custom /var/www/html/sites/all/modules/custom RUN ln -s /opt/drupal-custom/all/modules/features /var/www/html/sites/all/modules/features RUN rm -rf /var/www/html/sites/all/libraries RUN ln -s /opt/drupal-custom/all/libraries /var/www/html/sites/all/
Here's an example of installing a module outside of drush.make. I believe due to drush.make's issues with git submodules, this was done in a custom manner.
# get version 0.8.0 of raven-php. This is used for integration with Sentry RUN git clone https://github.com/getsentry/raven-php.git /opt/raven; cd /opt/raven; git checkout d4b741736125f2b892e07903cd40450b53479290 RUN ln -s /opt/raven /var/www/html/sites/all/libraries/raven
Add all the configuration files that don't make sense to make one line changes for. Again, these can be added here since they do not change with environments.
# Add all our config files from the Docker build repo ADD supervisord /etc/supervisord.conf ADD drupal.conf /etc/httpd/conf.d/site.conf ADD ssl_extras.conf /etc/httpd/conf.d/ssl.conf ADD docker-start.sh /docker-start.sh ADD drupal-rsyslog.conf /etc/rsyslog.d/drupal.conf
The user for this Docker image is root, but supervisord handles running different processes as their appropriate users (apache, memcached, etc).
The docker-start.sh script handles the database configuration once the Drupal container is running.
CMD ["/bin/bash", "/docker-start.sh"]
Docker start script
Once the Docker image is built that contains all the Drupal code we'll want to run, the container will be started up on the appropriate environment. Notice that environment-specific configuration was left out of the Docker image building process, and for this reason we know that this image should be deployable to any of our environments (assuming it gets the right configuration). With a configuration tool like puppet or ansible, we can provide the correct settings.php file to each host before our Docker container is deployed, and "configure" the container on startup with a command similar to below:
/usr/bin/docker run -td -p 80:80 -p 11211:11211 -e DRUPAL_START=1 -e DOMAIN=ci -v /var/log/httpd:/var/log/httpd -v /opt/settings.php:/var/www/html/sites/default/settings.php:ro drupal-custom:latest
A summary of some of the arguments:
* -p 80:80 -p 11211:11211 (open/map the correct ports for apache and memcached)
* -e DRUPAL_START=1 (apply the database configuration stored in Drupal code, described in the docker-start.sh script)
* -e DOMAIN=ci (let this container know it belongs in the ci domain)
* -v /var/log/httpd:/var/log/httpd (write apache logs inside the container to the host. In this way, we always have our logs stored between container restarts)
* -v /opt/settings.php:/var/www/html/sites/default/settings.php:ro (Let the container know its configuration from what exists on the host)
When the container starts up, the assumed action to take is to start up supervisord, and apply database configuration if it was requested with DRUPAL_START. Mainly this involves running some drush helper commands, like features revert, applying module updates to the database, and ensuring our list of installed modules is correct with the Master module.
#!/bin/bash if [[ ! -z $DRUPAL_START ]]; then supervisord -c /etc/supervisord.conf & sleep 20 cd /var/www/html && drush cc all status=$? if [[ $status != 0 ]]; then drush rr fi cd /var/www/html && drush en master -y cd /var/www/html && drush master-execute --no-uninstall --scope=$DOMAIN -y && drush fra -y && drush updb -y && drush cc all status=$? if [[ $status != 0 ]]; then echo "Drupal release errored out on database config commands, please see output for more detail" exit 1 fi echo "Finished applying updates for Drupal database" kill $(jobs -p) sleep 20 fi supervisord -n -c /etc/supervisord.conf
That's it! On your host, you should now be able to access the Drupal environment over port 80, assuming that the database connection within settings.php is correct. Depending on whether DRUPAL_START was set, the environment may take some time to configure itself against a current database.
The entirety of this example Dockerfile can be found here.
Next blog post we'll talk about using this container approach with Jenkins to automate the delivery pipeline.Posted: 2015-04-08T20:55:23+00:00
Today, we are excited to announce the addition of a new Red Hat Certification Catalog to the numerous resources within the Customer Portal.
Red Hat collaborates with hundreds of companies that develop hardware, devices, plug-ins, software applications, and services that are tested, supported, and certified to run on Red Hat technologies. Empowered with the new Red Hat Certification Catalog, you can explore a wide variety of partner solutions to:
- Ensure your Red Hat solution is running on tested, verified and supported hardware
- Find third-party software solutions tested specifically on the Red Hat platform
- Choose a Red Hat certified public cloud provider to run your Red Hat technologies
Our ecosystem of thousands of certified partner solutions continues to expand with each new release of Red Hat products, and the new catalog has been designed with you (our customers) in mind. Using the new Red Hat Certification Catalog, you can find hardware and cloud providers that are certified to run our latest product releases such as Red Hat Enterprise Linux 7 or Red Hat Linux OpenStack Platform 5. Furthermore, you can use the new catalog to find a public cloud provider certified to offer the latest version of Red Hat Enterprise Linux within a specific region or supporting a specific language.
Check out the new Red Hat Certification Catalog for yourself today, available via Certification in the navigation, to be equipped with the best combination of Red Hat technologies and certified partner solutions.Posted: 2014-07-14T21:00:01+00:00
We're excited to announce that for the fourth consecutive year, the Red Hat Customer Portal has been named one of the industry’s "Ten Best Web Support Sites” by the Association of Support Professionals (ASP). ASP is an international membership organization for customer support managers and professionals.
The "Ten Best Web Support Sites" competition is an awards program that showcases excellence in online service and support, and this year we were honored alongside other technology industry leaders including Cisco, Google Adwords, IBM, McAfee, Microsoft Surface Support, and PTC. Winners were selected based on performance criteria including overall usability, design and navigation, knowledgebase and search implementation, interactive features, and customer experience.
Since receiving our ASP award last year, we've been working diligently to improve our user experience even more. Armed with both customer and ASP feedback, we have enhanced our portal navigation and added features to better help customers find what they need, how and when they need it. The Customer Portal team's focus on analytics has brought rapid development and change. Using email and website statistics, A/B testing, and customer feedback, we are doing everything we can to better understand how our subscribers are using the portal.
While we're honored to receive the ASP award again, we're committed to continued innovation and improvement to give our customers the best experience. That commitment is evidenced by our most recent addition, the Certification Catalog, which unifies Red Hat certifications into a single platform and allows users to find certified hardware, certified third-party software solutions, and certified public cloud providers for their specific environment.
Learn more about the ASP award:
- ASP's 2014 press release: http://www.asponline.com/14announcement.html
- Red Hat's press blog: http://www.redhat.com/en/about/blog/red-hat-customer-portal-named-one-ten-best-web-support-sites-2014
With the upcoming release of Red Hat Enterprise Linux 7 on Tuesday June 10th, we've improved several things for our customers to provide the best experience yet.
Two new ways to download a product- A product page download section, and a revamped downloads area:
A refined getting started guide, using icons to help prep and set expectations for a basic deployment of RHEL 7
Curated content for user tasks to help customers accomplish their most important tasks
A redesigned look and feel to the docs, including removal of the html-single and epub format
A draft of the page can be seen here: https://access.redhat.com/site/products/red-hat-enterprise-linux-7. At launch, this page will replace the existing RHEL product page at this URL https://access.redhat.com/site/products/red-hat-enterprise-linux
There are two new ways to download RHEL now:
The product page tab for Downloads
A new downloads page leading to a new application:
As of today, all versions of Red Hat Enterprise Linux are available here.
New Get Started:
We've also introduced a new getting started guide for RHEL 7, with simplified requirements and steps.
We've also tried to reorganize our content by some of the most common tasks our customers read about. By looking at documentation trends, we've organized our resources based on what our customers have shown us is high priority. From there, we worked to display and organize different types of content to help them quickly learn about all the benefits available in RHEL 7.
We've also changed how we handle documentation in a few ways. In order to improve upon our documentation by adding comments, and future enhancements, we've removed the html-single and epub formats. We did this because we really want to take this opportunity to focus on the html version and do what we can do improve that experience in the future.
Additionally we've set PDF documentation to require a login. While this may be an adjustment for some users, we believe that we'll be able to offer a much better experience with future improvements focusing on html.
We're really excited about these changes, and would love to hear back about what we can do to make your experience even better.Posted: 2014-06-09T20:33:19+00:00
We're excited to announce the new ability to comment on new documentation within the Customer Portal. This is live and available today and we'll begin to roll this out for future documentation releases as well. For an example see the Red Hat Enterprise Linux 7 Release Notes.
Several months ago we introduced a survey for our documentation to gather feedback from the people who use it the most. With this most recent release we wanted to take that a step further, as we felt it was really important to source feedback from customers as quickly as possible. If there is something wrong or confusing, we want to know. Having the ability to do this in the moment allows us to really see what people might be struggling with, and offers an immediacy we don't always receive with surveys. We've received good feedback in the past on our solutions, articles, and videos this way and extending our community efforts to documentation made perfect sense.
We really hope that people find this new feedback method for documentation helpful. Our documentation teams will be reviewing and responding to comments directly and are excited to get feedback from the people who best help us measure our successes and improve in the future.Posted: 2014-05-27T16:59:17+00:00
At the Summit? Stop by the Customer Experience booth and do the following:
- Talk to Red Hat Support Engineers about your technical issues, upcoming products, or anything else you can think of!
- Enter our tech challenge for a chance to win a pair of Bose QC-15 noise canceling headphones. Interested in further details? Swing by the booth!
Not sure where our booth is? From the main entry point to the Moscone Center South on Howard Street, simply walk in and instead of getting on the escalators, look for the stairwells that are between the escalators. Our stairwell is right across from the Cool Stuff Store -- walk down the stairs and you won't be able to miss us!Posted: 2014-04-15T22:36:45+00:00
Sitting here at the Moscone Center in San Francisco, one can sense a palpable excitement for the upcoming Red Hat Summit! Keynotes begin tonight and the customer experience booth opens at 5pm. Come on by and take our skills test to see how you do.
Not in San Francisco? Don't worry, head on over to http://www.redhat.com/summit/ and live stream the keynotes tonight starting at 6:30pm PDT / 9:30pm EDT / 1:30am UTC. Too late in the evening for you? The keynotes will be recorded and those are usually posted to http://www.redhat.com/summit/ once they are available.Posted: 2014-04-14T20:21:07+00:00
The latest release of the Customer Portal offers a few new changes with regard to content and our navigational layout that we’re really excited about.
We recently introduced new product pages that serve as centralized locations for you to find all available resources in relation to a product. These pages offer direct access to our knowledge, documentation, videos, discussions, and more. We're happy to announce that we now offer pages for all of Red Hat's currently supported products.
We've also worked to provide product specific Get Started guides, and we will be releasing even more of these in the coming months. These guides are designed to offer succinct, basic instructions for customers who are new to their product.
For more advanced users, check out the recommended content on the Overview page. You can see that we've highlighted some reference architectures, where available. These guides offer in-depth, deployment-ready use cases and are a great resource for advanced deployments. For more papers like that, be sure to check out all our other reference architectures.
Our navigation has been slightly tweaked with this release as well. Now when selecting the Products view, you can easily see all our products broken out into categories. This should help you find products more easily based on technology categories. For those who prefer an alphabetical listing, we’ve offered a new view that can be selected from the drop-down located at access.redhat.com/site/products.
We’ve also made some slight changes to the Support tab. We took feedback that documentation was difficult to find, so we’ve placed that immediately below the Support drop-down. Additionally, we’ve consolidated Support Policies and Support Programs into a single category.
We're really excited about these changes because we believe these new pages and our updated navigation will enrich your online experience. We would love to hear your feedback on these changes, so please leave us a comment to let us know what you think!Posted: 2014-04-04T13:50:34+00:00
With this latest site release for the Red Hat Customer Portal, we're adding a new blog feature that we're very excited about. These blogs give us the ability to provide a unique channel of information and interaction to you, our customers. You'll have the opportunity to stay up to date on our various products and services, and we encourage you to share your ideas and suggestions with our various Red Hat contributors.
The initial set of Customer Portal blogs covers areas such as security, performance, our new Access Labs area, and, of course, the Customer Portal itself. Over time, we'll introduce more blogs and contributors to widen the scope and reach of the information that you want to hear.
Here are some initial blog posts that you might find interesting:
Introducing Access Labs! by Andrew Hecox: With exciting new developments to enhance your success with our products, our Access Labs are a new way for Red Hat engineers to quickly deliver tools to help improve performance, troubleshoot issues, identify security problems, and assist with any other issue we see customers experiencing in their IT environments.
The Right Performance Tool for the Task by Will Cohen: Will writes about his
Performance Measurement Cookbook and his efforts to make it easier for people to measure and understand machine performance. Rather than being focused on the tools themselves, the cookbook tasks (or recipes) are focused on how to get the data that you need to better understand what is happening on your system.
New SELinux Feature: File Name Transitions by Dan Walsh: In this blog post, Dan highlights the new Red Hat Enterprise Linux 7 feature to make labeling files easier for users and administrators, resolving one of the biggest issues with SELinux where initial creation of content by users and administrators can sometimes get the wrong label. The goal is to prevent the accidental mislabeling of file objects. Dan will be talking about many of Red Hat's security-related topics in our new Red Hat Security blog.
Upcoming Improvements to Customer Portal Navigation by Kyle Benson: Changes are coming to the main Customer Portal navigation menu to improve your experience in accessing the information, software, and services that are important to you.
We encourage you to follow these blogs to keep informed about what's going on here at Red Hat, including our ongoing efforts to exceed your expectations with our products and services. Log in today and join the conversation, and please share your ideas regarding what you would like to see us discuss in these new blogs!Posted: 2014-04-03T22:10:28+00:00