Minishift and the Enterprise: Disconnected Image Registry

Posted: May 31st, 2018 | Author: | Filed under: Technology | Tags: , , , | No Comments »

Security continues to be a priority in most organizations. Any breach may result in intellectual or financial losses. Reducing access to external systems by internal resources is one was to limit the threat potential. One such method is to place a middleman, or proxy, between internal and external resources to govern the types of traffic. Considerations for how the Container Development Kit (CDK) can traverse proxy servers were covered in a prior blog. However, many organizations are further reducing the need for communicating with remote systems and placing all resources within their infrastructure. Systems operating in a manner where access to external resources is completely restricted is known as running in a disconnected environment. OpenShift supports operating in a disconnected environment and cluster operators can take steps to prepare for normal operation. A full discussion on managing OpenShift in a disconnected environment is beyond the scope of this discussion, but can be found here. While there are several areas the must be accounted for when operating in a disconnected environment, having access to the container images that reside in external image registries is essential. The CDK, like the full platform is driven by container images sourced from external locations. Fortunately, the CDK does contain the functionality to specify an alternate location for which images that control the execution can originate from.

OpenShift’s container images are stored by default in the Red Hat Container Catalog (RHCC). Many organizations operate their own container registry internally for providing content either from remote locations or created in house. Common registry examples in use include a standalone docker registry (docker distribution), Sonatype Nexus, JFrog Artifactory and Red Hat Quay. Since the same container images that are used by OpenShift Container Platform are used by the CDK, organizations can serve them using an internal registry and satisfy both sets of consumers. One requirement that must be adhered to is that the name of the image repository, name and tag must match the source from the Red Hat Container Catalog (it can differ, however several manual changes would then be required).

export MINISHIFT_ENABLE_EXPERIMENTAL=y

With experimental features enabled, the CDK can now be started. For this example, let’s assume that there is an image registry located at registry.mycorp.com which has been seeded with the images to support OpenShift. Execute the following command to utilize the CDK with images sourced from this internal registry:

minishift start --insecure-registry registry.mycorp.com --docker-opt add-registry=registry.mycorp.com --docker-opt block-registry=registry.access.redhat.com --extra-clusterup-flags --image=registry.mycorp.com/openshift3/ose

Note: Concepts from both prior blogs on Proxies and Registration can also be used when running in a fully disconnected environment.

Note: Due to a regression in version 3.4 of the CDK, the –extra-clusterup-flags parameter is not accepted

Phew, that was a long command. Let’s take a moment to break it down.

  • minishift start

This is the primary command and subcommand used to start the CDK

  • –insecure-registry registry.mycorp.com

While the registry may be served using trusted SSL certificates, many organizations have their own Certificate Authority instead of leveraging a public CA, such as Comodo. Since the VM running the CDK only trusts certificates from public CA’s, this will allow docker to be able to communicate with the registry

  • –docker-opt add-registry= registry.mycorp.com

Many OpenShift components do not include the registry portion of the image and instead rely on the configuration of the underlying Docker daemon to provide a default set of registries to use. Both the OpenShift Container Platform and the Container Development Kit have the RHCC configured by default. By specifying the location of the internal registry, the CDK will be able to reference it when images are specified without the value of the registry.

  • –docker-opt block-registry=registry.access.redhat.com

To ensure images are only being sourced from the corporate registry not the default location (RHCC), the CDK VM can be configured to place a restriction at the docker daemon level.

  • –extra-clusterup-flags –image= registry.mycorp.com/openshift3/ose

OpenShift in the context of the CDK as of OpenShift version 3.9 utilizes the same image as containerized installation and contains all of the necessary logic to manage an OpenShift cluster. Under the covers of the CDK, the “oc cluster up” utility is leveraged to deploy OpenShift. By default, “oc cluster up” references the full path of the image, including registry. This experimental feature flag allows this value to be overridden with the location of the image from the enterprise registry.

The CDK will now start by pulling the container image and once this image is started, all dependent images by the platform will be retrieve. After the CDK has started fully, verify all running images are using the enterprise container registry.

First, check the names of the images currently running at a Docker level using the minishift ssh command:

minishift ssh "docker images --format '{{.Repository}}:{{.Tag}}'"

The final component that requires modification to support leveraging an enterprise registry is to update all of the ImageStreams that are populated in OpenShift. By default, they reference images from the RHCC. The Ansible based OpenShift installer does contain logic to update ImageStreams if the location differs from the RHCC. Unfortunately, the CDK does not contain this logic. Fortunately, this issue can be corrected with only a few commands.

First, make sure you are logged into OpenShift as a user with `cluster-admin` rights. By default, the `admin` user contains these privileges.

oc login -u admin

Similar to all other accounts in the CDK, any password can be specified.

Next replace the RHCC with the location of the enterprise registry for all ImageStreams by executing the following command:

oc get is -n openshift -o json | sed -e 's|registry.access.redhat.com|registry.mycorp.com|g' | oc replace -n openshift -f-

Make sure to replace registry.mycorp.com with the address of the enterprise registry.

With the ImageStreams now utilizing all of the enterprise registry as the source, reimport all of the ImageStreams:

for x in `oc get is -n openshift -o name`; do oc import-image $x -n openshift --all --insecure=true; done

After the command completes, all ImageStreams will be updated.

At this point the CDK is fully functional with images being referenced from the enterprise registry, thus enabling productivity in environments where security is a high priority.


Red Hat Summit 2018 Labs: A Recap

Posted: May 21st, 2018 | Author: | Filed under: Uncategorized | Tags: , | No Comments »

Red Hat Summit week is one of those weeks that I regularly circle on the calendar. Not only does it afford the opportunity to connect with my fellow Red Hat colleagues, but in addition, I am able to reconnect with some of my former customers. In reality, the entire three days that comprises Summit (plus OpenShift Commons Gathering which is held the day prior) is one big blur that consists of non-stop activity, but most importantly, a whole lot of fun. Part of the fun is the ability to showcase some of the latest and greatest technology thorough the use of hands-on labs that attendees are able to take advantage of. A full overview of these lab sessions were discussed in a prior post.

There are several challenges when it comes to hands-on labs at Red Hat Summit:

  • Lab sessions are longer than breakout sessions. The time commitment required may result in missing other sessions of interest.
  • Some prior knowledge may be required in order to fully appreciate the content

Regardless of the reason, one of the key goals of being an open source advocate is for materials to be made available publicly. Not only do I preach this sentiment, but this is also expressed by Red Hat as a whole. Fortunately, this year, all of the lab content from Summit is available on GitHub at the following location:

https://github.com/RedHatOfficial/rhsummitlabs-2018

Included in the repository are the lab guides that the actual attendees at Summit leveraged. Several of the guides also include steps for creating the supporting environment.

As for the lab sessions themselves, they were a mix of angst, frustration, excitement and finally relief. As anyone who has previously presented at a conference can attest to, if anything can happen, it will happen. Each of my lab sessions were in the 4pm-6pm timeslot. The timing of the lab can be dangerous as exhaustion from the day starts to take hold as well as being the one impediment holding attendees back from happy hours and evening events. Nevertheless, each of the sessions were packed to the brim.

Develop IoT solutions with containers and serverless patterns

The first day of Summit featured my first lab combining the Internet of Things (IoT), Containers (OpenShift) and Serverless (Functions as a Service [FaaS] using OpenWhisk). While things kicked off great, things were about to go downhill quickly. One of the first steps in the lab was for attendees to clone the Git repository containing the lab material. Unfortunately, the speed of the cloning action hovered around 5kb/s. For anyone who has worked previously in the container space, this meant trouble especially when larger container images needed to be retrieved later on. Fears came to reality as attendees struggled through the next set of tasks and we quickly realized that a successful completion of the lab may not be attainable. The entire lab environment was hosted in a cloud environment which removed the majority of the constraints on the infrastructure at Summit. However, it was communicated to the various team who were running labs during this session time that the cloud provider was having technical faults communicating with their external Internet Service Provider which resulted in the slow, but almost unusable connection to the public internet. After 45 minutes of waiting and hoping the issue would be resolved, attention turned to giving attendees the best experience possible given the constraints.

Fortunately, the associated lab guide provides a high level overview of Functions as a Services and the key OpenWhisk concepts that were to be introduced. While attendees were a little disappointed they would not be able to have hands on experience during this session, they left with the understanding of where to find the lab material, but most importantly how to create an environment representative of the lab themselves.

You too can learn how to set up and complete the lab in your own environment by utilizing the following set of assets:

Managing Your OpenShift Cluster From Installation and Beyond

The second day of Red Hat Summit featured the next evolution in the “Managing OpenShift from Installation and Beyond” series. Even with the technical challenges faced the prior day, there were several factors that provided some form of assurance that this lab would go smoother:

  • This was actually the second opportunity at Red Hat Summit to execute this lab. The team completed a lab session the day prior to Summit kicking off to a few select individuals. Any outstanding issues or enhancements were made after this session so that attendees of Red Hat Summit proper could have the best experience possible.
  • The lab was hosted in Amazon Web Services (AWS). If a similar issue in the environment occurred, this lab session would be the least of Amazon’s concern 🙂

As anticipated, the lab went off without any issues and attendees were able to fully immerse themselves into the ways that automation using Ansible and Ansible Tower can install and manage the OpenShift Container Platform. This lab was advantageous as it covered a variety of topics ranging from Ansible Tower, the Prometheus ecosystem with visualization support from Grafana along with building and deploying Ansible Playbook Bundles.

Those interested in reviewing the guide or standing up the environment itself can refer to the lab assets:

While Red Hat Summit 2018 has come to a close, eyes are already on to next year’s event back on the east coast of the US in Boston May 7-9 2019. Hope to see you there as well!


Hands on Labs at Red Hat Summit 2018

Posted: May 4th, 2018 | Author: | Filed under: Uncategorized | Tags: , , , | No Comments »

Red Hat Summit once again returns to the city by the bay, San Francisco, California, the week of May 7th. Not only is the event returning to Silicon Valley, but I will be reprising my role as hands on lab magician to offer Red Hat Summit attendees with the opportunity to take advantage of some of the latest and greatest technology available. Last year, I had the fortune of working with two of the smartest groups of individuals to develop and deliver labs focused of two hot topics in the industry: The installation and management of the OpenShift Container Platform along with the Internet of Things (IoT). This year, an entire new set of labs have been developed to demonstrate new and exciting ways to utilize the technology.

Develop IoT solutions with containers and serverless patterns

Tuesday is the first full day of Red Hat Summit and Ishu Verma (@IoT_Ishu), Technical Marketing Manager for IoT solutions at Red Hat and I showcase how the Internet of Things can be applied to one of the hottest trends in the industry: serverless technologies. In this session, attendees will be introduced to the principles along with the benefits of serverless technology and deploy Apache OpenWhisk on top of the OpenShift Container Platform. Once deployed, a variety of OpenWhisk concepts and patterns will be leveraged to demonstrate how metrics sent from IoT devices can best be utilized. These patterns include the basics of OpenWhisk tasks, such as creating actions, triggers and rules, along with methods for ingesting IoT data onto the platform. This session is ideal for anyone interested in working with serverless technologies on an open platform using real world IoT use cases.

Date: Tuesday May 8
Time: 4:00pm-6:00pm
Location: Moscone South – Room 158
Session Link: https://agenda.summit.redhat.com/SessionDetail.aspx?id=153502

Managing your OpenShift cluster from installation and beyond

In this session, Scott Collier (@collier_s), Vinny Valdez (@VinnyValdez) and Brett Thurber (LinkedIn) from Red Hat and myself will not only showcase how to install the OpenShift Container Platform using automation tools such as Ansible and Ansible Tower, but we will cover key “day two operations” concepts that ensure that the entire OpenShift Container Platform ecosystem remains healthy. This includes leveraging the Prometheus ecosystem of services to monitor the platform along with visualization support using Grafana. In addition, attendees will learn how to build their own Ansible Playbook Bundles to deploy an application that plays an integral part in managing the lifecycle of cluster management events. You do not want to miss this session!

Date: Thursday May 9
Time: 4:00pm-6:00pm
Location:Moscone South – Room 157
Session Link: https://agenda.summit.redhat.com/SessionDetail.aspx?id=154276


Minishift and the Enterprise: Registration

Posted: April 28th, 2018 | Author: | Filed under: Technology | Tags: , | No Comments »

One of the many hallmarks of Open Source Software is the ability for anyone in the community to freely contribute to a software project. This open model provides an opportunity to garner insight into the direction of a project from a larger pool of resources in contrast to a closed sourced model where software may be regulated by a single organization or group. Many enterprises also see the value of Open Source Software to power their most critical systems. However, enterprises must be cognizant that Open Source Software from the community may not have the integrity that they have been accustomed to when using software obtained directly from a vendor. Red Hat, as a leader of Open Source Software solutions, provides a subscription model that can be used to meet the quality and support requirements necessary by any organization. A subscription includes fully tested and hardened software, patches, and customer support. Once a subscription has been purchased, licensed software must be registered to activate the necessary included features.

The Container Development Kit (CDK) is the supported version of the upstream minishift project, and given that the software package is built on top of a Red Hat Enterprise Linux base, a valid subscription associated with a Red Hat account is required to access the entire featureset provided by the CDK. To enable the development on Red Hat’s ecosystem of tools, a no-cost developer subscription is available through the Red Hat Developer program and includes an entitlement to Red Hat Enterprise Linux along with a suite of development tools that are regularly updated with the latest enhancements and features. Information about the Red Hat Developer Subscription along with the steps to create an account can be found at the Red Hat Developer Website.

Once a Red Hat Developer account has been obtained, the configuration of associating the account within the CDK can be completed. These steps were detailed in the prior post, Minishift and the Enterprise: Installation.

While the Red Hat Developer subscription is a great way for developers to take advantage of enterprise Linux software, many organizations frown upon the use of personal licenses operating within the organization, especially on company owned machines. The CDK is configured to automatically register and associate subscriptions against Red Hat’s hosted subscription management infrastructure. Accounts for developers can be created within the Red Hat Customer Portal for use with the CDK. As described in the post Minishift and the Enterprise: Proxies, subscription-manager, the tool within RHEL for tracking and managing subscriptions, is automatically configured to traverse a corporate proxy server to the public internet when this option is enabled. This feature, as previously mentioned, is useful as most enterprises employ some form of barrier between the end user and external network.

Unfortunately, many enterprises do not use Red Hat’s hosted subscription management system to register machines on their network and instead leverage Red Hat Satellite within their internal network. The CDK, as of version 3.4, is only able to register subscriptions against Red Hat automatically as part of normal startup. Fortunately, there are methods in which the user can configure the CDK to register against a satellite server instead of Red Hat. These options include:

  1. Executing commands to facilitate the registration process
  2. Leveraging an add-on which streamlines the registration process

Regardless of the method utilized, the CDK should be instructed to not attempt to register the machine during startup. This is accomplished by passing the –skip-registration parameter when executing the minishift start command as shown below:

minishift start --skip-registration

Even though the RHEL machine within the CDK is not registered, the majority of the functionality will remain unaffected. The key exception is managing software packages using the yum utility. Since RHEL based images inherit subscription and repository information from the host they are running on, operations both on the host machine as well as within a container making use of yum will fail due to the lack of valid subscriptions. This is primarily noticeable at image build time as it typically involves the installation of packages using yum.

The RHEL machine within the CDK can be registered manually in a similar fashion to any other RHEL machine using the subscription-manager utility. To gain access to a prompt within the CDK, the minishift ssh command can be used.

minishift ssh

By default, an ssh session is established within the CDK using the “docker” user. Since subscription-manager requires root privileges, access must be elevated using the sudo command. Execute the following command to elevate to the root user:

sudo su -

With access to root privileges, the machine can now be registered to Red Hat using the subscription-manager register command. Either a username/password or activation key/organization combination can be used as follows:

subscription-manager register --username=<username> --password=<password>

Or:

subscription-manager register --org=<organization> --activationkey=<activation_key>

In either case, adding the –auto-attach parameter to each command will attach a subscription automatically to the new registration.

To subscribe the CDK against an instance of Red Hat Satellite instead of Red Hat’s hosted infrastructure, many of the same commands can be reused. An additional step is required to first download the bundle containing the certificates for the Satellite server so that the CDK can communicate securely to facilitate the registration process. Execute the following command to install the certificates into the CDK:

rpm -Uvh http://<satellite_server>/pub/katello-ca-consumer-latest.noarch.rpm

Now use subscription-manager to complete the registration process using the –org and –activationkey parameters:

subscription-manager register --org=<organization> --activationkey=<activation_key> --auto-attach

To validate the CDK is properly subscribed, let’s start a new container and attempt to install a package using yum by executing the following command within the CDK:

docker run -it --rm rhel:7.4 yum install -y dos2unix

If the above command succeeded, the CDK is properly registered and subscribed.


Installing the Ansible Playbook Bundle CLI Using Homebrew

Posted: April 15th, 2018 | Author: | Filed under: Technology | 1 Comment »

An Ansible Playbook Bundle is a lightweight application definition that utilizes a short-lived container to choreograph the deployment of applications. By leveraging the power of the Ansible configuration management tool within the Open Service Broker API specification, it has never been easier to orchestrate even the most complex application. For developers looking to make use of this framework, a command line tool called apb is available to facilitate the creation, building and publishing of Ansible Playbook Bundles. The tool can be installed from an rpm package, built from source, or even run within a container itself.

A challenge presents itself for developers who are striving to jump right into APB development, but may be on an Operating System that either does not support rpm packages, or do not want to install an extra set of tools to build the application from source. For those running OSX, Homebrew has become the defacto package manager akin to yum for certain Linux distributions. Installing software packages which, previously proved to be a challenge, can be accomplished with a simple command, like brew install node. Now, developers can take advantage of the ease of use and wide adoption of Homebrew to install the apb CLI!

First, ensure Homebrew is installed on the local machine. Full instructions can be found at the Homebrew website.

Within Homebrew, software packages are organized into Formulas. The majority of the packages that most users leverage are in the core set of formulas. However, formulas can exist outside of the core and be referenced from external sources, known as Taps. apb resides as a formula that is brought into Homebrew on a machine using a tap.

Installation

Add the tap containing the apb formula:

brew tap sabre1041/openshift

With the tap installed, install the apb CLI:

brew install apb

After the process completes, the tool will be installed.

To verify the installation, execute the following command:

apb version

If a response was returned, the installation of the apb CLI was successful.

With only a few short commands, developers of Ansible Playbook Bundles within the OSX ecosystem can quickly begin to create powerful applications.


Minishift and the Enterprise: Proxies

Posted: March 8th, 2018 | Author: | Filed under: Technology | Tags: , , , | 3 Comments »

Cross Transport

In part 1 of this series, an overview was provided on how to install and configure Minishift/Container Development Kit (CDK) for use in an enterprise environment. During this discussion, the primary concern focused on the preparation of the host with the proper prerequisite software and to confirm the user executing the tool has the appropriate access to execute the required commands for operation. For those who have spent any amount of time in an enterprise environment, one of the primary challenges is how to manage resources that have a dependency on communicating with external resources.

Most organizations have some form of security or filtering layer that all ingress and egress traffic must traverse through before reaching the intended destination. These types of systems are commonplace and take the form of a software or hardware proxy and are implemented for security and compliance reasons. Computers on a corporate network are preconfigured to redirect all traffic (typically at an Operating System level) through the proxy before being allowed on the public Internet.

Unfortunately for developers (and in reality all users), proxy servers tend to cause the most distress and can negatively affect their productivity. Users must be cognizant of the transport mechanisms involved in any tool they are working with and any reliance on external systems. In addition, they must have the knowledge and understanding of whether the tool supports proxying connections, and if so, how to complete the configuration. With an understanding of the vital role proxies play in an enterprise setting, the remainder of this discussion will provide an overview on the support Minishift has for proxy servers and the various ways the various included components can be configured.

Determine required external resources

Minishift is one of those tools that makes requests to external resources. Over the course of normal operation, several different external resources are accessed by the various components within not only Minishift’s architecture, but also the underlying OpenShift ecosystem. When working a proxy in any corporate environment, one of the first questions before determining how to configure the tool for proxy support is to pinpoint the list of resources that the tool will need access to and if access is currently allowed. This step is akin to the process that was completed in the prior blog post for determining any prerequisite software Minishift needed. In addition, having a list prepared upfront aids in any discussion with security teams or as part of a site exemption process.

The following is a high level list of sites that Minishift requires access to for minimal operation.

  • Docker Images (From DockerHub)
    • docker.io – Authentication and registry
    • cloudfront.net – Image content is served through the CloudFront CDN
  • Red Hat Container Catalog
    • registry.access.redhat.com
    • access.redhat.com
  • Red Hat Subscription Manager (CDK)
    • subscription.rhsm.redhat.com
  • Build assets
    • If external resources are being leveraged such as source code from github.com or build artifacts from the Maven public repository.

OpenShift and Proxies

With an understanding of the different types of external resources that are accessed during the normal operation of Minishift, how are they used by the various included components? I would be flat out lying if I were to say that the proxy configuration in OpenShift and its components are straightforward. In fact, there is an entire document dedicated to detailing the components that have proxy support and the mechanisms in which they can be configured. These include:

  • Master and node services
  • Docker
  • Image builds
  • Usage in running containers

As you can see, the number of components is not trivial and each has their own set of configuration files and rules that can be applied. At times, it can be dizzying.

Configuring Minishift for Proxies

Fortunately, Minishift takes away the majority of the pain involved with managing proxies by supporting the auto configuration of many of the items presented in the prior section through a set of startup flags and options. Since proxies are typically targeted at managing HTTP traffic, there are three primary considerations:

  • How to manage insecure traffic (HTTP)
  • How to manage secure traffic (HTTPS)
  • Which type of traffic should not be routed through the proxy

Most operating systems provide support for these items through a set of configurable variables:

  • http_proxy
  • https_proxy
  • no_proxy
  • (Along with their all uppercase counterparts).

Likewise, proxy support in Minishift is facilitated by making use of these types of variables. Let’s see how to configure Minishift to make use of a proxy.

The most declarative method for managing proxy connections is to specify the options as parameters when starting Minishift with the minishift start command. As indicated previously, three command line options are available: The –http-proxy option is used for insecure communication while –https-proxy for secure communication. Finally, –no-proxy allows for a list of addresses to be provided which will not make use of a proxy.

For example, to start Minishift and to configure the instance to make use of a proxy server listening at http://corporate-proxy:8080 and to avoid using the proxy for a resource located at internal-resource.mycorp.com, the following command can be executed:

minishift start --http-proxy=http://corporate-proxy:8080 --https-proxy=http://corporate-proxy:8080 --no-proxy=internal-resource.mycorp.com

Note: If authentication is required to communicate with the proxy, it can be provided using the format “http://<user>:<password>@<ip_addr>:<port>”

You can verify Minishift is being configured with proxy support when the following appears during startup:

-- Setting proxy information ... .OK

Instead of specifying the proxy parameters each time Minishift is started, the values can be persistently set using the minishift config command as shown below using the aforementioned example values:

minishift config set http-proxy http://corporate-proxy:8080
minishift config set https-proxy http://corporate-proxy:8080
minishift config set no-proxy internal-resource.mycorp.com

So What Did Minishift Configure?

While an overview was provided that described the various ways proxies can be configured in OpenShift, which areas in particular did Minishift implement with regards to proxies?

  • Preflight checks to verify connectivity prior to launch
  • Host level operating system environment variables
  • Subscription manager to support registering the CDK
  • Docker daemon of the underlying host running the containerized version of OpenShift
  • Global build defaults to support accessing source code and build dependencies

If Only It Were That Easy….

Recall previously where I had mentioned that proxies cause the most heartache of all other obstacles in an enterprise environment? Well, even after Minishift is configured to make use of a proxy server, there are a number of gotcha’s that trip up even the most seasoned developer. Let’s discuss some of the areas that have been known to cause issues when working with Minishift and proxies.

Running Containers

Even though the docker daemon of the underlying host is configured with proxy support, these values are not inherited by the underlying containers. Instead, the container (or more accurately Pod), must be defined with values which the included components can make use of. These typically take the form of the aforementioned proxy values. The following is an example of how to define proxy values within a pod specification.

containers:
- env:
  - name: "HTTP_PROXY"
    value: "http://corporate-proxy:8080”
  - name: "HTTPS_PROXY"
    value: "http://corporate-proxy:8080”
  - name: "NO_PROXY"
    value: "internal-resource.mycorp.com”

As previously mentioned, it is up to the underlying application on how and whether it supports proxies.

Command Line

The most common method when interacting with Minishift is through the executable using the command line. Once Minishift is up and running, focus shifts to the running OpenShift cluster and management through the Command Line Interface (even though the web console is also available). While the majority of CLI commands specifically target the OpenShift cluster, there are several commands that have the potential to fail within an environment containing proxies. As part of the “oc new-app” and “oc new-build” commands, the target images and source code repositories are inspected in order to satisfy the actions of the command. If the target location requires the traversal through the proxy server and the proxy details are not within the command line environment, the query and subsequent commands will fail.

Fortunately, since most organizations configure proxies at an Operating System level, these values are inherited within the command line session. If they are not defined, set the HTTP_PROXY and HTTPS_PROXY environment variables accordingly. However, this does cause a problem with communicating with OpenShift along with Minishift itself. If the IP address of Minishift is not specified in the list of NO_PROXY values, traffic intended for Minishift will instead be bound for the proxy and in most cases terminate at that point as the proxy has no knowledge of the networking on an individual developers machine.

To add Minishift to the list of addresses that should bypass proxies, utilize the following steps:

Find the IP address of Minishift:

minishift ip

Take the address provided by the previous command and add it to the list of addresses:

export NO_PROXY=$NO_PROXY,<minishift ip>

OpenShift CLI commands should once again return valid results

Final thoughts

As has been described, Minishift provides a number of solutions for utilizing the tool within an environment that contains a proxy server. While the majority of the most common scenarios were addressed within this article, it is almost certain that there will be other situations depending on the environment for which the tool is deployed within. Fortunately, as a result of its core architecture for supporting proxies, Minishift has the capabilities to be leveraged successfully in an enterprise environment.


Minishift and the Enterprise: Installation

Posted: February 14th, 2018 | Author: | Filed under: Technology | 3 Comments »

SoftwareAs technology has continued to advance, the steps for developing and deploying applications has followed suit. What was once a fairly standard process that involved only a finite number of components has evolved dramatically in recent years with the introduction of new toolsets and frameworks. While adding a new software package on an individual’s local machine may seem like a trivial task, the concerns are multiplied within an enterprise setting. What is the origin of the component? Why is it needed? What is the update cycle? Who supports the tool? These are all valid questions and a common occurrence in any organization practicing diligent IT security and compliance methodologies.

While a full-blown installation of OpenShift is not practical on a typical developer machine, Minishift offers developers the opportunity to take advantage of managing advanced container workloads while accelerating developer productivity without all of the complexities of the operational support. As organizations embrace OpenShift, they see the benefits of giving their developers the freedom of running an entire containerized version platform within an individual machine. These advantages include eliminating the reliance on a shared OpenShift environment as well as reducing the overall resource utilization of said cluster. As a result, costs are reduced and the opportunity for adding additional services to the environment can be attained.

So you want to run OpenShift locally

Based on prior experience, these concerns are typically encountered before organizations can adopt Minishift:

  • Describe the system requirements
  • Provide a list of dependencies

This type of information gives IT administrators, desktop support personnel, and compliance teams the necessary knowledge to fully understand the items they may be ultimately responsible for managing.

The system requirements and dependencies directly correlate to the type of distributions that will be provided.

Given the context of this series is focused on an enterprise setting, the remainder of these discussions will focus on the Container Development Kit (CDK), the downstream, productized version of Minishift. While many of the same concerns are found in the upstream project, there are specific items that are only found in the CDK.

The Container Development Kit is available using two installation paths:

  • A standalone executable
  • As part of the Red Hat Development Suite: A packaged set of tools to support container development, which includes OpenJDK, Red Hat JBoss Developer Studio, VirtualBox, and the Container Development Kit.

In most organizations that I have worked with, they choose to adopt the standalone executable instead of the Red Hat Development Suite. This decision is a result of organizations wanting to manage a succinct number of tools instead of a full suite. The suite is great for developers working outside of an enterprise context, as it provides a guided installation of the CDK. Also, there may already be a comparable tool that is packaged with the suite already within the enterprise.

Regardless of the installation approach, the primary requirement to support the CDK is the presence of a supported hypervisor as it runs within a virtual machine.

Several popular options are available:

The choices are directly dependent on the target Operating System, which in most organizations is either Windows or OSX. Even though I do work for a Linux company, I fully acknowledge the fact that most developers will rarely use Linux as their primary host in an enterprise setting.

Some organizations do wince at the prospect of having to support virtualization tools as they typically do require an increased system resource profile, such as RAM and CPU. Security teams also are concerned about any implementations that may attempt to compromise existing security features implemented both within the network as well as on the machine itself. In addition, administrative rights are required for the installation and in some cases the operation of hypervisor tooling.

Fortunately, thanks to the advent of development approaches leveraging virtual machines along with tools to support automated environment configuration, such as Vagrant, some organizations already have hypervisors present on developer machines and can forgo working through this process. This greatly simplifies the potential hurdles for adoption within the enterprise.

Setup and configuration

Once all of the hurdles of installing the prerequisite software have been completed, the CDK itself can be installed and configured. Organizations typically block the individual downloading of executable files for security purposes, so other mechanisms must be utilized. In many cases, approved software is made available through a software catalog. Otherwise, the executable for the CDK can be hosted in an artifact repository, such as Sonatype Nexus or JFrog Artifactory, for users to download.

When using a manual installation process and not initiated by a corporate installation tool that may automate many of the installation and configuration processes, users can either reference the executable file directly or place the file on the PATH and make it available within any terminal session. In the case of a Windows system, users will typically not be able to modify the system PATH variable as it is shared among multiple users of the machine. Instead, they can modify the user PATH variable that only affects their own individual account.

The next step is to setup the CDK itself by using the setup-cdk minishift command as shown below:

minishift setup-cdk

Setting up the CDK extracts the ISO file used to boot the virtual machine and prepares the system for use. All of these files are copied to the .minishift folder inside the users’ HOME directory ($HOME/.minishift). The location of this folder can be changed either by passing the –minishift-home flag or setting the MINISHIFT_HOME environment variable to a suitable location. In many environments, a home directory is located on a network share so that files can easily be retrieved across multiple systems as a user traverses different machines in the organization. However, a performance loss will be incurred during the operation of the CDK as resources are retrieved from a remote location. In addition, if for some reason a user is not connected to the corporate network, they will not have access to the required files to operate the CDK. When selecting an alternate folder location, users must be cognizant of the folder they are intending to target and whether they have write permissions.

The final step prior to starting the CDK is to specify credentials for registering the included copy of Red Hat Enterprise Linux (RHEL) to the Red Hat Network (RHN). Registration is required in order for containers running within OpenShift to take advantage of yum packages, as yum utilizes the subscriptions of the underlying host. A developer subscription can be obtained for free at https://developers.redhat.com or may be available through an existing enterprise license. While a properly subscribed machine is not required for normal operation, it will result in reduced functionality.

Credentials can be stored persistently using the MINISHIFT_USERNAME and MINISHIFT_PASSWORD environment variables, specified using the –username and –password flags during startup, or specified as a configuration item managed by the minishift config command. In addition to managing usernames and passwords, a wide range of options can be specified using the minishift config command, such as the hypervisor driver along with the amount of CPU or RAM allocated for the virtual machine. An example of how to set the virtual machine driver is shown below:

minishift config set vm-driver virtualbox

All that is left to do is to start the CDK along with any optional parameters:

minishift start

The Container Development Kit will provision a new virtual machine, register the machine, and complete a number of provisioning steps including the following:

  • Downloading the containerized version of OpenShift
  • Allocating a set of PersistentVolumes for making use of persistent storage
  • Applying several add-ons which extend the functionality of the base OpenShift configuration
  • Creating an admin and developer account for interacting with the platform

Once the CDK has started, it will present a URL to access the web console along with credentials for the developer account.

A baseline installation of the Container Development Kit in an enterprise environment is fairly straightforward once all of the prerequisites have been fulfilled. However, there are situations where access to external resources may not be readily available. In the next post in this series, we will illustrate how to use the Container Development Kit in an environment that requires the use of a proxy server to access external assets.


Minishift and the Enterprise

Posted: February 6th, 2018 | Author: | Filed under: Uncategorized | No Comments »

MinishiftThe Enterprise and the downstream Red Hat Container Development Kit (CDK) gives users the power of running a cloud environment backed by the OpenShift Container Platform on their local machines. It not only gives operations focused individuals the ability to test out features to better understand the platform, but also accelerates developers’ ability to produce world-class software that can thrive within cloud environments. However, their experiences working with the tool can vary greatly depending on the environment they are operating within. For those who happen to work within an enterprise setting, their experience may be fraught with pitfalls and perils. Access restrictions, policies and limitations are the norm. Even the most simplistic task can be a challenge.

To showcase how to overcome common challenges experienced by users running Minishift in a constrained setting, I will demonstrate how to resolve barriers such as proxy servers and offline image registries in a series titled “Minishift and the Enterprise”.


Automating the Template for New OpenShift Projects

Posted: January 22nd, 2018 | Author: | Filed under: Technology | Tags: , , | No Comments »

Project templateSeparation of concerns is one of the fundamental principles in a multitenant environment. It ensures that multiple users can coexist without knowledge or access to each others resources. While OpenShift supports multitenancy through a number of different mechanisms, the use of projects provide users with their own separate workspace with the cluster. Built on top of Kubernetes namespaces, projects in OpenShift not only allows for a single user, but an entire community of users the ability to effectively collaborate within the cloud. One of the unique features of projects is that when they are initially created, they are preconfigured with a set of service accounts and permissions (roles/role bindings) that allow for users to be immediately productive. While the initial set of customized resources within a project provides a suitable baseline for end users, cluster administrators may want to enforce additional policies and configurations whenever a new project is created. Fortunately, OpenShift provides the capability of defining a default project template that will be instantiated whenever a new project is created. The following steps describe how to create a new default project template:

  1. Export the default project template to a file
  2. Make the appropriate modifications
  3. Add the template to the cluster
  4. Edit the master-config.yaml configuration on each master to designate the template that should be used as the default project template

In most of the clusters that I regularly deploy, I have automated these actions as a post installation step (obviously driven through Ansible). Even with the additional automation, it still presents another dependency that must be maintained when installing a new OpenShift environment. Fortunately starting with OpenShift 3.7.14, native support for configuring the default project template is available within the installation. Similar to the other options that are available during an installation of OpenShift, all that is required is defining a few Ansible variables. The rest of this post describes how to automate the configuration of a default project template through the OpenShift installer.

Before specifying the mechanisms necessary to configure the installer, let’s define a common use case of how the default project template could be modified. To protect the stability of the cluster, ResourceQuotas and LimitRanges can be applied to any new project to define the and restrict the amount of resources that can be consumed by running applications. While these metrics can vary from cluster to cluster, a middle of the road set of resources can be defined as follows:

ResourceQuotas

Name CPU Memory Scope
quota 2 12Gi NonTerminating
burst-quota 4 16

LimitRanges

Name Pod Max Pod Min Container Max Container Min Container Default
limits CPU: 200m Memory: 2Gi CPU: 10m Memory: 128Mi CPU: 200m Memory: 2Gi CPU: 20m Memory: 256Mi CPU: 50 Memory: 256Mi

The openshift_project_request_template Ansible role is ultimately responsible for implementing many of the steps that would normally need to be manually configured. This role makes use of an Ansible library called yedit that allows for the modification of YAML files. As the role executes, it exports the default project template and modifies the file with customized values. Any modification to the default project template must be defined by the variable openshift_project_request_template_edits which takes in an array of modifications that should be executed by yedit. This variable, similar to other variables that allow for the customization of the OpenShift installation are defined within the [OSEv3:vars] section of the Ansible inventory. Formatting the variable in a manner in which the module can properly process the changes is the most complex aspect of the customization process.

Each array value must define a set of values to perform the desired results of customizing the default project template. First is a key within the yaml file that should be modified. For those familiar with OpenShift templates, there are two primary key properties: objects which define the resources that are applied to OpenShift and parameters which allow for values to be dynamically injected at runtime. Since default project templates do not support additional parameter values, the only the object key can be modified. Since the desired outcome of the execution process is to append the existing list of objects, yedit has a module option called `append` that when set to True, will append the newly defined values with the existing values. Finally, the newly created object itself must be defined. Each additional resource to be added to the default project template must be separated into its own array object. In our use case, three objects will be created as shown below:

 

openshift_project_request_template_edits:
  - key: objects
    action: append
    value:
      apiVersion: v1
      kind: ResourceQuota
      metadata:
        annotations:
          openshift.io/quota-tier: Medium
        labels:
          quota-tier: Medium
        name: quota
      spec:
        hard:
          cpu: "2"
          memory: 12Gi
        scopes:
        - NotTerminating
  - key: objects
    action: append
    value:
      apiVersion: v1
      kind: ResourceQuota
      metadata:
        annotations:
          openshift.io/quota-tier: Medium
        labels:
          quota-tier: Medium
        name: burst-quota
      spec:
        hard:
          cpu: "4"
          memory: 16Gi
  - key: objects
    action: append
    value:
      apiVersion: v1
      kind: LimitRange
      metadata:
        annotations:
          openshift.io/quota-tier: Medium
        labels:
          quota-tier: Medium
        name: limits
      spec:
        limits:
        - max:
            cpu: 200m
            memory: 2Gi
          min:
            cpu: 10m
            memory: 128Mi
          type: Pod
        - default:
            cpu: 50m
            memory: 256Mi
          max:
            cpu: 200m
            memory: 2Gi
          min:
            cpu: 20m
            memory: 256Mi
          type: Container

 

Once the openshift_project_request_template_edits variable has been defined, a few more variables must also be defined. First, openshift_project_request_template_name should be defined in order to specify the name of the template that should be created and then openshift_project_request_template_namespace specifies the namespace (or project) in which the template will be created. To create a template called project-request in the default namespace, the following variables can be specified:

 

openshift_project_request_template_namespace: default
openshift_project_request_template_name: project-request

 

Next, the OpenShift installation must be instructed to execute the logic in the openshift_project_request_template through the use of the openshift_project_request_template_manage boolean variable which should be set to True as shown below:

 

openshift_project_request_template_manage: True

 

The final step is to define the name along with the namespace in the format namespace/name that will be added to the projectRequestTemplate field of the master-config.yaml file. This is defined by the osm_project_request_template as shown below:

 

osm_project_request_template: default/project-request

 

After the four (4) variables have been added, execute the installer against a new or existing OpenShift cluster.

Once the installer has completed, verify the expected results by logging in to the OpenShift environment and creating a new project. Create a new project called test-project-template.

 

oc new-project test-project-template

 

Verify the ResourceQuotas are present:

 

$ oc get resourcequotas

NAME          AGE
burst-quota   1m
quota         1m

 

Now verify the LimitRange was created:

 

$ oc get limitrange

NAME      AGE
limits    1m

 

With just a few Ansible variables, the OpenShift installer streamlined the steps necessary to configure the default project template for an OpenShift cluster.


An Update to Visualizing the OpenShift Rest API

Posted: October 22nd, 2017 | Author: | Filed under: Technology | No Comments »

Swagger Logo

As microservices architectures continue to proliferate, there becomes an ever-increasing need for insight into the options that may be provided by each service. This is important as the primary mechanism between each service is through lightweight request protocols that are typically exposed through an Application Programming Interface (API). Several tools have been developed to address concerns within this space and Swagger, an ecosystem of API development and testing tools, is one of the most popular options available.

Prior to the release of OpenShift version 3 in the summer of 2015, I designed a version of the Swagger User Interface (UI) frontend based on a concept from OpenShift lead engineer Clayton Coleman which enabled users to easily visualize and interact the OpenShift REST API (A blog entry can be found here). At that time, only two endpoints were exposed:

Fast forward two years and Kubernetes, OpenShift and Swagger have come a long way. In the fall of 2015, it was announced that an open governance model would be created around the Swagger specification, which would solidify Swagger as the standard for describing REST API’s. Version 2.0 of Swagger represented the first release under the new model. The initial implementation of Swagger within Kubernetes and OpenShift made use of the older 1.2 version of the Swagger specification. Fortunately, a few months later, a 2.0 compatible specification was released to support the standardized model for describing the API.

Today, the OpenShift Swagger UI site has been updated that takes advantage of both the OpenShift provided Swagger 2.0 compatible specification along with the recently released Swagger 3.0 UI. No longer is the UI restricted to just the /api and /oapi endpoints, but all of the API’s exposed, including the new API groups, are available.

Swagger OpenShift

The repository can be found below:

https://github.com/sabre1041/openshift-api-swagger

The repository contains a simple HTML along with a few assets. It can be run as is from a local machine or deployed to a web server. Similar to the prior version, OpenShift must be configured to allow cross site JavaScript requests (CORS). Once OpenShift has been properly configured, all that is needed by the end user is the address of the OpenShift master and an OAuth token of a user already authenticated to OpenShift for methods requiring authentication.

Enjoy!