The Setup: Composer,
Phing, Vagrant, Jenkins

Stephan Hochdörfer // 27.02.2014

About me

  • Stephan Hochdörfer
  • Head of IT, bitExpert AG (Mannheim, Germany)
  • S.Hochdoerfer@bitExpert.de
  • @shochdoerfer

  • #PHP, #DevOps, #Automation

Our situation (~2 years ago)

Goal: Improve the situation

The challenge


  • How to deal with dependencies?
  • How to deal with database
    schema changes?
  • How to isolate the
    development environment?
  • How to hide the complexitiy?

As simple as possible





git clone && vagrant up


Vagrant benefits



  • Environment per project
  • Versionable
  • Shared across the team
  • No more: "Works on my
    machine!"

Vagrantfile

# Vagrant 1.3.1 or later!
Vagrant.configure("2") do |config|
    config.vm.define "box" do |box_cfg|
        box_cfg.vm.box      = 'debian-wheezy-64'
        box_cfg.vm.box_url  = 'https://myhost.loc/Wheezy64.box'
        box_cfg.vm.hostname = "my-host"

        box_cfg.vm.synced_folder ".", "/vagrant", owner: "vagrant", group: "vagrant"
    end
end

Run the virtual machine

$> vagrant up
Bringing machine 'box' up with 'virtualbox' provider...
[box] Importing base box 'debian-wheezy-64'...
[box] Matching MAC address for NAT networking...
[box] Setting the name of the VM...
[box] Clearing any previously set forwarded ports...
[box] Creating shared folders metadata...
[box] Clearing any previously set network interfaces...
[box] Preparing network interfaces based on configuration...
[box] Forwarding ports...
[box] -- 22 => 2222 (adapter 1)
[box] Booting VM...
[box] Waiting for machine to boot. This may take a few minutes...
[box] Machine booted and ready!
[box] Setting hostname...
[box] Mounting shared folders...
[box] -- /vagrant

Vagrantbox.es

Vagrant plugins


  • vagrant-hostmanager
  • vagrant-cachier
  • vagrant-librarian-puppet
  • Provider plugins: AWS,
    Openstack, Rackspace, ...
  • ...



Simple Puppet script

$default_packages = [ 'expect', 'curl', 'git']
package { $default_packages :
    ensure => present,
}

host { 'myhost.loc':
    ip     => '127.0.1.1'
}

Running Puppet

$> puppet apply init.pp
notice: /Stage[main]//Package[curl]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Package[git]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Package[expect]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Host[myhost.loc]/ensure: created
notice: Finished catalog run in 1.86 seconds

Directory layout

/home/shochdoerfer/Projects/sample-project
   |-build
   |-config
   |-puppet
   |---hieradata
   |-----common.yaml
   |---manifests
   |-----init.pp
   |---modules
   |---src
   |-----sample-project
   |-------manifests
   |-------templates
   |---Puppetfile
   |-shell
   |---bootstrap.sh
   |-src
   |-tests
   |-webroot

librarian-puppet





You can use librarian-puppet to manage the
puppet modules your infrastructure depends on.

Puppetfile

forge "http://forge.puppetlabs.com"

# external dependencies
mod 'example42/apache', '2.1.2'
mod 'example42/mysql', '2.1.0'
mod 'example42/php', '2.0.9'
mod 'puppetlabs/apt', '1.3.0'
mod 'puppetlabs/stdlib', '4.1.0'

# copy local module in the modules directory
mod 'sample-project',
  :path => './src/sample-project'
$> librarian-puppet install --verbose

Vagrant Provisioning

# Vagrant 1.3.1 or later!
Vagrant.configure("2") do |config|
    config.vm.define "box" do |box_cfg|
        box_cfg.vm.box      = 'debian-wheezy-64'
        box_cfg.vm.box_url  = 'https://myhost.loc/Wheezy64.box'
        box_cfg.vm.hostname = "my-host"

        box_cfg.vm.synced_folder ".", "/vagrant", owner: "vagrant", group: "vagrant"
        
        box_cfg.vm.provision "shell" do |shell|
          shell.path = 'shell/bootstrap.sh'
          shell.args = "-d /vagrant/puppet/ -i init.pp -h /vagrant/puppet/hiera.yaml"
        end
    end
end

bootstrap.sh

while getopts d:i:h flag; do
  case $flag in
    d) PUPPET_DIR=$OPTARG; ;;
    i) PUPPET_FILE=$OPTARG; ;;
    h) HIERA_CONFIG=$OPTARG; ;;
    ?) exit; ;;
  esac
done

if [ `gem query --local | grep librarian-puppet | wc -l` -eq 0 ]; then
  gem install librarian-puppet
  cd /vagrant/puppet/ && librarian-puppet install --clean
else
  cd /vagrant/puppet/ && librarian-puppet update
fi

puppet apply --verbose --debug --hiera_config $HIERA_CONFIG --modulepath=$PUPPET_DIR/modules/ $PUPPET_DIR/manifests/$PUPPET_FILE



Install Composer

$> curl -sS https://getcomposer.org/installer | php

Running Composer

{
    "require": {
        "monolog/monolog": "1.0.*",
        "swiftmailer/swiftmailer": "v5.0.3"
    }
}
$> composer.phar install
Loading composer repositories with package information
Installing dependencies (including require-dev)
  - Installing swiftmailer/swiftmailer (v5.0.3)
    Downloading: 100%

  - Installing monolog/monolog (1.0.2)
    Downloading: 100%

Writing lock file
Generating autoload files

Composer via Puppet

$default_packages = [ 'curl', 'git', 'php5-cli' ]
package { $default_packages: ensure => present, }

exec { 'composer.phar':
  command => 'curl -sS https://getcomposer.org/installer | php',
  cwd     => '/usr/local/bin',
  creates => '/usr/local/bin/composer.phar',
  timeout => 0,
  require => [Package['php5-cli'], Package['curl']]
}

exec { 'composer-run':
  command     => 'composer.phar install --dev --prefer-dist',
  cwd         => '/vagrant/',
  environment => 'COMPOSER_HOME=/root/',
  timeout     => 0,
  require     => [Package['git'], Package['php5-cli'], Exec['composer.phar']]
}

Publish your packages

Private packages?





Satis is a ultra-lightweight, static
file-based version of packagist.

Install Satis

$> composer.phar create-project composer/satis --stability=dev

Running Satis

{
    "name": "My own repo",
    "homepage": "http://satis.mydomain.loc",
    "repositories": [
        { "type": "vcs", "url": "http://git.mydomain.loc/repo1.git" },
        { "type": "vcs", "url": "http://git.mydomain.loc/repo2.git" }
    ],
    "require-all": true,
    "archive": {
        "directory": "dist",
        "format": "zip",
        "skip-dev": false
    }
}
$> php bin/satis build config.json web/

Generated Satis Repo

Configure custom repository

{
    "require": {
        "monolog/monolog": "1.0.*",
        "swiftmailer/swiftmailer": "v5.0.3"
    },
    "repositories" : [{
	    "type" : "composer",
	    "url" : "http://satis.mydomain.loc/"
    }],
}

Dealing with private repos





What about repo`s with an
HTTP Basic Auth protection?

Using "expect"

#!/bin/sh
expect -c " 
set timeout -1 
spawn composer.phar update --dev --prefer-dist
expect { 
    -re { Username: } { 
	send \"myusername\r\" 
	exp_continue 
    } 
    -re { Password: } { 
	send \"mypassword\r\" 
	exp_continue 
    } 
    -re {^Generating autoload files } 
}"

Using "expect"

# Vagrant 1.3.1 or later!
Vagrant.configure("2") do |config|
    config.vm.define "box" do |box_cfg|
        box_cfg.vm.box      = 'debian-wheezy-64'
        box_cfg.vm.box_url  = 'https://myhost.loc/Wheezy64.box'
        box_cfg.vm.hostname = "my-host"

        box_cfg.vm.synced_folder ".", "/vagrant", owner: "vagrant", group: "vagrant"
        
        box_cfg.vm.provision "shell" do |shell|
	  USER=ENV['COMPOSER_USER']
	  PASS=ENV['COMPOSER_PASS']

          shell.path = 'shell/bootstrap.sh'
          shell.args = "-d /vagrant/puppet/ -i init.pp -h /vagrant/puppet/hiera.yaml -u USER -p PASS"
        end
    end
end

Using "expect"

exec { 'composer-run':
  command => "expect -c '
    set timeout -1
    spawn composer.phar install --dev --prefer-dist
    expect {
      -re { Username: } {
	send \"${::user}\\r\"
	exp_continue
      }
      -re { Password: } {
	send \"${::pass}\\r\"
	exp_continue
      }
      -re {^Generating autoload files }
    }'",
  cwd     => '/vagrant',
  timeout => 0,
  require => [Package['expect'], Package['git'], Package['php5-cli'], File['composer.phar']]
}

Install tools with Composer

{
    "require": {
        "monolog/monolog": "1.0.*",
        "swiftmailer/swiftmailer": "v5.0.3"
    },
    "require-dev" : {
	"phpunit/phpunit" : "3.7.*@stable",
	"phing/phing" : "2.5.*@stable",
	"squizlabs/php_codesniffer" : "1.4.*@stable"
    },
    "repositories" : [{
	    "type" : "composer",
	    "url" : "http://satis.mydomain.loc/"
    }],
}
$> ./vendor/bin/phpunit
$> ./vendor/bin/phing
$> ./vendor/bin/phpcs




Domain-specific language

A glue for 3rd party tools

Why Phing?


  • Runs everywhere where
    PHP runs
  • No additional depencendies
    needed (e.g. Java, ...)
  • More than 120 predefined
    tasks to choose from
  • Easy to extend by writing
    custom tasks in PHP

Distinct Target Naming

<?xml version="1.0"?>
<project name="myproject" default="ci:run-tests">

    <target name="app:clean-cache">
    </target>

    <target name="app:create-cache">
    </target>

    <target name="db:migrate">
    </target>

    <target name="js:minifiy">
    </target>

    <target name="ci:lint">
    </target>

    <target name="ci:run-tests">
    </target>
</project>

Import Targets

<?xml version="1.0"?>
<project name="myproject" default="app:run">

    <!--
    The following target namespaces exist:
    db:*  - Database specific targets
    app:* - Application specific tasks
    ci:*  - CI server specific tasks
    -->
    <import file="build/build.db.xml" />
    <import file="build/build.app.xml" />
    <import file="build/build.ci.xml" />
</project>

Import Targets (Composer)

<?xml version="1.0"?>
<project name="myproject" default="app:run">

    <!--
    The following target namespaces exist:
    lib1:*  - Targets imported from lib1
    lib2:*  - Targets imported from lib2
    app:*   - Local application targets
    -->
    <import file="vendor/vendor1/lib1/build/build.xml" />
    <import file="vendor/vendor2/lib2/build/build.xml" />
    <import file="build/build.app.xml" />
</project>

Phing via Puppet

# Running Liquibase via Phing
exec { 'phing-db-update':
  command   => './vendor/bin/phing db:update',
  cwd       => '/vagrant/',
  require   => [Exec['composer-run'], Package['postgresql-server-9.1']],
  logoutput => true
}

# Generate cache files
exec { 'phing-generate-cache':
  command   => './vendor/bin/phing app:warm-cache',
  cwd       => '/vagrant',
  require   => Exec['composer-run'],
  logoutput => true
}





Jenkins project configuration

Jenkins project configuration

Jenkins project configuration








Thank you!