Dynamic Jenkins Slave Provisioning on OpenShift

Posted: January 31st, 2016 | Author: sabre1041 | Filed under: Technology | 1 Comment »

Configuring the OpenShift Environment

With an understanding of the process facilitated by the Kubernetes plugin, lets create the environment within OpenShift. First, download the GitHub repository containing the necessary resources to your local machine.

git clone https://github.com/sabre1041/ose-jenkins-cluster

Next, create a new project in OpenShift either in the web console or on the command line using the oc client called jenkins

oc new-project jenkins

Enter the directory containing the Git repository cloned previously and add the two templates to the newly created project

oc create -f support/jenkins-cluster-persistent-template.json,support/jenkins-cluster-ephemeral-template.json

Now that the templates are now available in the jenkins project within OpenShift, let’s take a moment to review the contents of the templates. For the most part, both templates are identical with the exception of the jenkins-cluster-persistent template which allows for the configurations of the Jenkins server to survive a pod destruction as the data is saved to permanent storage. The templates also utilize parameters that can be customized for a particular environment. While we will not detail the full list of template parameters (a complete list and description can be found in the repository README.md file), several of them do merit further discussion as their values dictate the amount of manual intervention necessary once the instance has been created or how the cluster will perform in the environment.

First, the service account parameter (SERVICE_ACCOUNT) used to specify the name of the service account that should be applied to the Jenkins master pod. Every pod in OpenShift has a service account automatically mounted as a volume located at /var/run/secrets/kubernetes.io/serviceaccount/.  Without any customization, the default project service account is used that injects its OAuth token and OpenShift Certificate Authority (CA) file which can be used to communicate with the OpenShift API. As discussed previously, the Kubernetes plugin leverages the API to communicate the actions of creating and destroying slave instances. Unfortunately, the default service account does not have the requisite permissions to perform all of the functions required by the plugin. To be able to use the default service account, execute the following command which will provide the requisite permissions

oc policy add-role-to-user system:deployer system:serviceaccount:jenkins:default -n jenkins

This command added the system defined deployer role which has the rights to manipulate pods within a project.

Alternatively, a separate service account could be specified instead of the default service account. Let’s demonstrate creating a new service account for this purpose called jenkins. A file in the support folder of the repository called jenkins-sa.json contains the definition of the service account. Add the new Jenkins service account to OpenShift by running the following command from the root directory of the git repository:

oc create -f support/jenkins-sa.json

Once again, add the system defined deployer role to the Jenkins service account:

oc policy add-role-to-user system:deployer system:serviceaccount:jenkins:jenkins -n jenkins

The next template parameter of note is the Slave Recurrence interval. This value is used internally by Jenkins to determine how ofter the master should check to see if it should deploy new slave instances to take on pending jobs. You may wonder why there should be a need for such a configuration. If a there is a pending job, then a new slave should be allocated. However, if Jenkins did abide by this paradigm, there is a high likelihood that the slave pool will overcommit on the number of resources consumed in the environment and actually slow down the time it takes for jobs to complete. Consider that it may take 15-30 seconds for a slave to fully come online. If an existing slave was concluding running an existing job and finished just after a new slave was provisioned, it would not only be unable to take on the job, but additional resources would be consumed creating the new slave and the time would be wasted for it to fully come online. Instead, a mathematical calculation is performed at each recurrence interval to determine if the overall system load requires the creation of additional slave instances. The default recurrence value in Jenkins is 1000, but the template defaults to 500. By lowering the default recurrence value, Jenkins will increase the frequency that it will determine whether to provision additional slave instances.

With an understanding of the template parameters, instantiate the template either in the web console or using the CLI. To instantiate the template using the CLI, run the following command in the jenkins project by specifying the template name and the service account to launch the jenkins master:

oc new-app --template=jenkins-cluster-ephemeral -p JENKINS_SERVICE_ACCOUNT=jenkins

Note: If you added permissions to the default service account instead of creating a new service account, the parameter option can be omitted

Once the template has been instantiated,  new Docker builds of the Jenkins master and slave components will be started. Once the builds complete and the deployments succeed, both the master and slave deployments can be seen on the OpenShift project overview page.

---