Jenkins for PHP projects
Stephan Hochdörfer // October 20 2016
About me
Stephan Hochdörfer
Head of Technology , bitExpert AG (Mannheim, Germany)
S.Hochdoerfer@bitExpert.de
@shochdoerfer
#PHP, #DevOps, #Automation
#phpugffm, #phpugmrn, #unKonf
Our story: Diversity
A lot of different projects with a lot of technology stacks. In need of a flexible solution. 4 years ago we migrated to Jenkins and this is our story and the lessons learned.
Our Jenkins infrastructure grew from a rather simple master - slave setup into a low two-digit number of buildnodes with a lot of differences (which sometimes are still hard to track).
Back in the "old" days...
Before using Jenkins we were using Trac + bitten.
...and today: Jenkins
Jenkins Use Cases
Running continuous builds
Running nightly builds
Running integration tests
Triggering Satis builds
Gitlab builds
Continuous builds needs to run fast, just PHPUnit, CodeSniffer, PMD and such
Nightly builds check more things, e.g. run a full sencha app build
Integration builds test everything from the frontend up to the database
Satis is used for hosting our private Composer packages
PHP Projects with Jenkins
Template for Jenkins Jobs
Jenkins Project Overview
Jenkins Plugin: Custom Job Icon Plugin, Green Balls Plugin
Mostly used to better "organize" (categorize) our jobs
Icon Config Management
Jenkins Job Icon Config
Config File Management
Jenkins Plugin: Config File Provider Plugin
Jenkins Node Overview
Buildnodes for "unit tests" vs. Integrationnodes for "integration tests". Every job can run on a buildnode while just one particular job will tun on the integration build servers.
Code is checked out directly on the integration build servers, not copied around! Was easier this way.
Not easy to keep an overview, but there`s a solution to that...
Jenkins Node Config
The Jenkins law: If a job can run on multiple nodes Jenkins will always pick the node which has at least 1 of the required tools missing and thus the job will fail! Always."
Jenkins Node Environment
Did not want to whitelist the buildnodes in our Satis instance. Decided to configure username and password here.
Continuous build
« [...] environments automatically rebuild the project whenever changes are checked in - often several times a day - and provide more immediate feedback » - Wikipedia
Our typical (PHP) toolchain
Composer, Phing, PHPUnit,
PHP_CodeSniffer, PDepend, PHPCPD
Project structure
/ tmp/ myproject
|- build
|--- coverage
|--- logs
|--- pdepend
|- docs
|- src
|--- Vendor
|----- MyProject
|- tests
|--- Vendor
|----- MyProject
|- vendor
|- web
|- build.xml
|- composer.json
|- composer.lock
|- phpdoc.dist.xml
|- phpmd.xml
|- phpunit.xml.dist
Basic Phing build.xml
<?xml version ="1.0" ?>
<project name ="ci" default ="ci:help" basedir ="." >
<target name ="-ci:clean"
depends ="-init"
hidden ="true" >
<delete dir ="${phing.dir}/build/logs" quiet ="true" />
</target>
<target name ="-ci:prepare"
depends ="-init, -ci:clean"
hidden ="true" >
<mkdir dir ="${phing.dir}/build/logs" />
</target>
<target name ="ci:build"
depends ="-init, -ci:prepare, ci:pdepend, ci:phpunit,
ci:phpcpd, ci:phpmd, ci:phpcs" />
</project>
Phing is Ant for PHP on steriods.
For most languages / tools we have a ci:build task which is triggered by Jenkins. The only exception are Puppet builds which are named ci_build. Consistency is very important for us!
Jenkins Job Config
Label Usage
Old: Jobs run on buildslave01 or buildslave02 or...
New: Jobs need a buildslave capable of providing featureA, featureB...
Define Git repository
Polling vs. Push notifications
For a long time we used the polling feature but it broke last year. Had to switch to polling instead
Polling vs. Push notifications
Minor difference! Schedule information should not be needed, but we still have it configured...
Polling vs. Push notifications
#!/bin/bash
# https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin
repo= "https://gitlab.loc/customer-1/project-1.git"
curl - s - o / dev/ null "https://jenkins.loc/jenkins/git/notifyCommit?url=${repo}" 2 >& 1
The Git plugin will scan all the jobs that:
Have Build Triggers > Poll SCM enabled. No polling Schedule is required.
Are configured to build the repository at the specified URL
Optionally you are able to pass the branch names or the SHA-1 commit ID to run the build against
Environment configuration
Jenkins Plugin: Mask Passwords Plugin
Push config files to node
Jenkins Plugin: Config File Provider Plugin
We could have used Puppet or Chef for that but for me it made more sense to let Jenkins do the job as it is somehow part of the build process.
Validate composer.json
A couple of months back the validation command changed a bit: It will run a checksum test against
composer.json. Any small change, e.g. typo in the description needs a composer update!
Installing dependencies
$COMPOSER_USER and $COMPOSER_PASS are populated from the node environment settings we defined before.
Jenkins Plugin: Mask Passwords Plugin
Running Phing
Will run PHP_CodeSniffer, PHPUnit, PDepend, ...
We do not use the Phing Jenkins plugin as the plugin is only capable of running a globally installed version of Phing. We like local dependencies and for that need to use the Shell commands.
Installing npm dependencies
Running Grunt
Grunt ist for Javascript what Phing is for PHP (but without the steriods)
Will run jscs, jshint, scss-lint
Process the build results
Checkstyle, PMD, Duplicate code results, Clover PHP Coverage Report, JUnit Report
Clover PHP Coverage Report
Checkstyle, PMD, Duplicate code results, Clover PHP Coverage Report, JUnit Report
Email notifications
Jenkins Plugin: Email-ext plugin
Email notifications
For the continious builds we send failure emails just on the 1st failure, but nightly builds or integration builds we send mails on every failure.
Trigger HHVM / PHP7 builds
For libraries we want to make sure that they are compatible with hhvm or the in-the-works version 7 of PHP.
Execute HHVM build
The Phing task will execute PHPUnit via the PHP interpreter which we do not want. Simple bugfix for the problem.
Trigger Satis build
Running nightly builds
« A nightly build is a neutral build that takes place automatically. These typically take place when no one is likely to be working in the office [...] » - Wikipedia
Runs the same tasks as the continous builds and some more, e.g. "compiling" js apps, or duing security checks.
Security issues?
Check composer.lock file
$ php security-checker.phar security: check ./ composer.lock
Security Check Report
~~~~~~~~~~~~~~~~~~~~~
Checked file : composer.lock
[ OK]
0 packages have known vulnerabilities
Security Checker Phing task
Automate with Jenkins
Running integration tests
« [...] is the phase in software testing in which individual software modules are combined and tested as a group. » - Wikipedia
-> Code is checked out directly on the integration build servers, not copied around!
Running Phing
Tests the service layer interaction with database access.
Running Grunt
Tests the frontend with Selenium connecting to the local instance.
Triggering Satis builds
Satis is a ultra-lightweight, static file-based version of packagist.
Installing Satis
$ composer.phar create- project composer/ satis -- stability= dev
{
"name" : "bitExpert Satis repo" ,
"homepage" : "https://satis.loc" ,
"repositories" : [
{ "type" : "vcs" , "url" : "/srv/repos/project-1.git" },
{ "type" : "vcs" , "url" : "/srv/repos/project-2.git" }
],
"require-all" : true ,
"archive" : {
"directory" : "dist" ,
"format" : "zip" ,
"skip-dev" : false
}
}
Git, Satis and Jenkis master run on the same box that`s why we use the local file paths to our git repos. No need to clone a remote.
Running Satis
$ php bin/ satis build config.json web/
$ php bin/ satis build config.json web/ customername/ project1
Jenkins Satis Job Config
Define Job Param (Package)
Jenkins Plugin: Parameterized Trigger Plugin
Define Job Param (Recipient)
Jenkins Plugin: Parameterized Trigger Plugin
Running Satis build
Rewrite repo urls
Git, Satis and Jenkis master run on the same box, that`s why we need to rewrite the paths here to be able to run composer with --prefer-source to clone the source repo.
Cleanup old files
Notify users after build
Jenkins Plugin: Email-ext plugin
Running the build
Console output
Satis repository
Run Composer
$ composer.phar require customername/ project1
Using version ^1.0 for customername/ project1
./ composer.json has been created
Loading composer repositories with package information
Updating dependencies ( including require- dev)
- Installing customername/ project1 ( 1.0.0 )
Downloading: 100 %
Writing lock file
Generating autoload files
Gitlab builds
2 approaches: Webhook calling the notify url or Jenkins Gitlab Plugin
Gitlab Jenkins Webhook
Similar as seen before: Call the notifiy url with the repository url.
This mechanism will not show build status for each commit or merge request in Gitlab!
Jenkins Gitlab Plugin Config
Jenkins Plugin: Gitlab Plugin
Jenkins Job Build Trigger
"Set build description to build cause" -> which commit triggered the build in Jenkins?
"Accept merge request on success" -> no need to manually click on the MR button in Gitlab
Gitlab Jenkins Webhook
If you plan to use forked repositories, you will need to enable the GitLab CI integration on each
fork.
Gitlab Commit Build running
Gitlab Commit Build result
Gitlab Merge Request
Gitlab Merge Request
Gitlab Merge Request
Jenkins Plugins Reference