From Vagrant
to Production

Stephan Hochdörfer // 18.11.2015

About me

• Stephan Hochdörfer
• Head of Technology, bitExpert AG
  (Mannheim, Germany)
• S.Hochdoerfer@bitExpert.de
• @shochdoerfer
  
  
• #PHP, #DevOps, #Automation, #unKonf

Our situation (~4 years ago)

Goal: Improve the situation

The challenge

• How to deal with dependencies?
• How to deal with database
  schema changes?
• How to isolate the
  development environment?
• How to hide the complexitiy?

As simple as possible

git clone && vagrant up

Vagrant benefits

• Environment per project
• Versionable
• Shared across the team
• No more: "Works on my
  machine!"

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.require_version ">= 1.6.0"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    config.vm.define "wheezy" do |wheezy|
        wheezy.vm.box      = 'debian-wheezy-64'
        wheezy.vm.box_url  = 'http://vagrantboxes.loc/wheezy64.box'
        wheezy.vm.hostname = 'myhost'

        wheezy.vm.synced_folder ".", "/vagrant", owner: "www-data", 
        group: "vagrant"
    end
end

Run the virtual machine

$> vagrant up

Bringing machine 'box' up with 'virtualbox' provider...
[box] Importing base box 'debian-wheezy-64'...
[box] Matching MAC address for NAT networking...
[box] Setting the name of the VM...
[box] Clearing any previously set forwarded ports...
[box] Creating shared folders metadata...
[box] Clearing any previously set network interfaces...
[box] Preparing network interfaces based on configuration...
[box] Forwarding ports...
[box] -- 22 => 2222 (adapter 1)
[box] Booting VM...
[box] Waiting for machine to boot. This may take a few minutes...
[box] Machine booted and ready!
[box] Setting hostname...
[box] Mounting shared folders...
[box] -- /vagrant

Vagrantbox.es

Vagrant Provider

• Virtualbox
• VMware
• Hyper-V
• Docker
• KVM
• ...

Vagrant Plugins

• vagrant-cachier
• vagrant-hostmanager
• vagrant-librarian-puppet
• vagrant-dns
• ...

Vagrant Cachier Plugin

$ vagrant plugin install vagrant-cachier

VAGRANTFILE_API_VERSION = "2"

Vagrant.require_version ">= 1.6.0"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    if Vagrant.has_plugin?("vagrant-cachier")
       config.cache.enable :apt
       config.cache.enable :apt_lists
       config.cache.enable :composer
    end

    config.vm.define "wheezy" do |wheezy|
        wheezy.vm.box      = 'debian-wheezy-64'
        wheezy.vm.box_url  = 'http://vagrantboxes.loc/wheezy64.box'
        wheezy.vm.hostname = 'myhost'

        wheezy.vm.synced_folder ".", "/vagrant", owner: "www-data", 
        group: "vagrant"
    end
end

PuPHPet

Simple Puppet script

$default_packages = [ 'expect', 'curl', 'git']
package { $default_packages :
    ensure => present,
}

host { 'myhost.loc':
    ip     => '127.0.1.1'
}

Running Puppet

$> puppet apply init.pp

notice: /Stage[main]//Package[curl]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Package[git]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Package[expect]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Host[myhost.loc]/ensure: created
notice: Finished catalog run in 1.86 seconds

Directory layout

/home/shochdoerfer/Projects/sample-project
   |-build
   |-config
   |-puppet
   |---hieradata
   |-----common.yaml
   |---manifests
   |-----init.pp
   |---modules
   |---src
   |-----sample-project
   |-------manifests
   |-------templates
   |---Puppetfile
   |-shell
   |---bootstrap.sh
   |-src
   |-tests
   |-webroot

librarian-puppet

You can use librarian-puppet to manage the
puppet modules your infrastructure depends on.

Puppetfile

forge "https://forgeapi.puppetlabs.com"

# external puppet modules that should be downloaded
mod 'example42-apache', '2.1.7'
mod 'example42-php', '2.0.18'
mod 'puppetlabs-apt', '1.3.0'
mod 'puppetlabs-stdlib', '4.1.0'
mod 'puppetlabs-postgresql', '3.3.3'

# copy local module in the modules directory
mod 'bitexpert-project',
  :path => './src/sample-project'

Running librarian-puppet

$> gem install librarian-puppet

$> librarian-puppet install

$> librarian-puppet update

Vagrant Provisioning

VAGRANTFILE_API_VERSION = "2"

Vagrant.require_version ">= 1.6.0"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    config.vm.define "wheezy" do |wheezy|
        wheezy.vm.box      = 'debian-wheezy-64'
        wheezy.vm.box_url  = 'http://vagrantboxes.loc/wheezy64.box'
        wheezy.vm.hostname = 'myhost'

        wheezy.vm.synced_folder ".", "/vagrant", owner: "www-data", 
        group: "vagrant"

        wheezy.vm.provision "shell" do |shell|
          shell.path = 'shell/bootstrap.sh'
          shell.args = "-d /vagrant/puppet/ -i init.pp"
        end
    end
end

bootstrap.sh

#!/usr/bin/env bash

while getopts d:i:h flag; do
  case $flag in
    d) PUPPET_DIR=$OPTARG; ;;
    i) PUPPET_FILE=$OPTARG; ;;
    h) HIERA_CONFIG=$OPTARG; ;;
    ?) exit; ;;
  esac
done

if [ `gem query --local | grep librarian-puppet | wc -l` -eq 0 ]; then
  gem install librarian-puppet
  cd $PUPPET_DIR && librarian-puppet install --clean
else
  cd $PUPPET_DIR && librarian-puppet update
fi

puppet apply --verbose --debug --hiera_config $HIERA_CONFIG --modulepath=$PUPPET_DIR/modules/ $PUPPET_DIR/manifests/$PUPPET_FILE

Install Composer

$> curl -sS https://getcomposer.org/installer | php

Running Composer

{
    "require": {
        "monolog/monolog": "1.0.*",
        "swiftmailer/swiftmailer": "v5.0.3"
    }
}

$> composer.phar install

Loading composer repositories with package information
Installing dependencies (including require-dev)
  - Installing swiftmailer/swiftmailer (v5.0.3)
    Downloading: 100%

  - Installing monolog/monolog (1.0.2)
    Downloading: 100%

Writing lock file
Generating autoload files

Composer via Puppet

$default_packages = [ 'curl', 'git', 'php5-cli' ]
package { $default_packages: ensure => present, }

exec { 'composer':
  command => 'curl -sS https://getcomposer.org/installer | php',
  cwd     => '/usr/local/bin',
  creates => '/usr/local/bin/composer.phar',
  timeout => 0,
  require => [Package['php5-cli'], Package['curl']]
}

exec { 'composer-run':
  command     => 'composer.phar install --prefer-dist',
  cwd         => '/vagrant/',
  environment => 'COMPOSER_HOME=/root/',
  timeout     => 0,
  require     => [Package['git'], Package['php5-cli'], Exec['composer']]
}

Publish your packages

Private packages?

Satis is a ultra-lightweight, static
file-based version of packagist.

Install Satis

$> composer.phar create-project composer/satis --stability=dev

Running Satis

{
    "name": "My own repo",
    "homepage": "http://satis.mydomain.loc",
    "repositories": [
        { "type": "vcs", "url": "http://git.mydomain.loc/repo1.git" },
        { "type": "vcs", "url": "http://git.mydomain.loc/repo2.git" }
    ],
    "require-all": true,
    "archive": {
        "directory": "dist",
        "format": "zip",
        "skip-dev": false
    }
}

$> php bin/satis build config.json web/

Generated Satis Repo

Configure custom repository

{
    "require": {
        "monolog/monolog": "1.0.*",
        "swiftmailer/swiftmailer": "v5.0.3"
    },
    "repositories" : [{
	    "type" : "composer",
	    "url" : "http://satis.mydomain.loc/"
    }],
}

$> composer.phar install

Dealing with private repos

HTTP Basic Auth

$> composer.phar install
Loading composer repositories with package information
    Authentication required (satis.mydomain.loc):
      Username: myuser
      Password:
Do you want to store credentials for satis.mydomain.loc in 
~/.composer/auth.json ? [Yn]

HTTP Basic Auth (auth.json)

{
    "http-basic": {
        "satis.mydomain.loc": {
            "username": "myuser",
            "password": "mypassword"
        },
        "git.mydomain.loc": {
            "username": "mygituser",
            "password": "mypassword"
        },
        "svn.mydomain.loc": {
            "username": "mysvnuser",
            "password": "mypassword"
        }
    }
}

Using "expect"

#!/bin/sh
expect -c " 
set timeout -1 
spawn /usr/local/bin/composer.phar install --prefer-dist
expect { 
    -re { Username: } { 
	send \"myusername\r\" 
	exp_continue 
    } 
    -re { Password: } { 
	send \"mypassword\r\" 
	exp_continue 
    } 
    -re {^Generating autoload files } 
}"

Using "expect"

VAGRANTFILE_API_VERSION = "2"

Vagrant.require_version ">= 1.6.0"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    config.vm.define "wheezy" do |wheezy|
        wheezy.vm.box      = 'debian-wheezy-64'
        wheezy.vm.box_url  = 'http://vagrantboxes.loc/wheezy64.box'
        wheezy.vm.hostname = 'myhost'
        wheezy.vm.synced_folder ".", "/vagrant", owner: "www-data", 
        group: "vagrant"
        wheezy.vm.provision "shell" do |shell|
          USER=ENV['COMPOSER_USER']
          PASS=ENV['COMPOSER_PASS']

          shell.path = 'shell/bootstrap.sh'
          shell.args = "-d /vagrant/puppet/ -i init.pp -u USER -p PASS"
        end
    end
end

Using "expect"

exec { 'composer-run':
  command => "expect -c '
    set timeout -1
    spawn /usr/local/bin/composer.phar install --prefer-dist
    expect {
      -re { Username: } {
	send \"${::user}\\r\"
	exp_continue
      }
      -re { Password: } {
	send \"${::pass}\\r\"
	exp_continue
      }
      -re {^Generating autoload files }
    }'",
  cwd     => '/vagrant',
  timeout => 0,
  environment => 'COMPOSER_HOME=/root/',
  require => [Package['expect'], Package['git'], Package['php5-cli'], 
  File['composer']]
}

Install tools with Composer

{
    "require": {
        "monolog/monolog": "1.0.*",
        "swiftmailer/swiftmailer": "v5.0.3"
    },
    "require-dev" : {
	"phpunit/phpunit" : "4.3.*@stable",
	"squizlabs/php_codesniffer" : "1.4.*@stable"
    },
    "repositories" : [{
	    "type" : "composer",
	    "url" : "http://satis.mydomain.loc/"
    }],
}

$> ./vendor/bin/phpunit
$> ./vendor/bin/phpcs

Install tools with Composer

# https://github.com/dflydev/just-run-phpunit
PHPUNIT_PATH=vendor/bin:bin

function phpunit {
    TEST_PATHS=($PHPUNIT_PATH)
    for TEST_PATH in "${TEST_PATHS[@]}"
    do
        if [ -x "${TEST_PATH}/phpunit" ]
        then
            "${TEST_PATH}/phpunit" "$@"; return $?
        fi
    done

    PHPUNIT_ON_PATH="$(type -P phpunit )"
    if [ -n "${PHPUNIT_ON_PATH}" ]; then
        "${PHPUNIT_ON_PATH}" "$@"; return $?
    else
        echo "phpunit not found!"; return 127
    fi
}

Domain-specific language

A glue for 3rd party tools

Why Phing?

• Runs everywhere where
  PHP runs
• No additional depencendies
  needed (e.g. Java, ...)
• More than 120 predefined
  tasks to choose from
• Easy to extend by writing
  custom tasks in PHP

Install Phing

{
    "require-dev" : {
	"phing/phing" : "2.5.*@stable"
    },
    "repositories" : [{
	    "type" : "composer",
	    "url" : "http://satis.mydomain.loc/"
    }],
}

$> ./vendor/bin/phing -v
Phing 2.5.0

build.properties

phpunit.path=vendor/bin/phpunit
phpunit.junit.log=build/logs/junit.xml
phpunit.coverage.clover=build/logs/clover.xml
phpunit.coverage.html=build/coverage

phpcs.path=vendor/bin/phpcs
phpcs.ignore=/Phing/
phpcs.log=build/logs/checkstyle.xml

deploy.dev.ssh.user=
deploy.dev.ssh.host=dev.mydomain.loc
deploy.dev.tmp.dir=/deploy/
deploy.dev.install.dir=/srv

local.properties

deploy.dev.ssh.user=myuser

Using Properties Files

<?xml version="1.0"?>
<project name="myproject" default="hello">

    <target name="hello" depends="init">
	<echo msg="Hello ${deploy.dev.ssh.user}" />
    </target>

    <target name="init" depends="prop, local-prop">
	<!-- some more init logic -->
    </target>

    <target name="prop">
	<echo message="Loading default build.properties"/>

	<property file="build.properties" />
    </target>

Using Properties Files

    <target name="local-prop" if="local-prop.exists"
	depends="local-prop-check">

	<echo message="Loading custom properties!"/>
	<property file="local.properties" override="true"/>
    </target>

    <target name="local-prop-check">
	<available file="local.properties"
	    property="local-prop.exists" />
    </target>
</project>

Using Properties Files

$> phing
Buildfile: /tmp/myproject/build.xml

myproject > prop:
     [echo] Loading default build.properties
 [property] Loading /tmp/myproject/build.properties

myproject > local-prop-check:

myproject > local-prop:

myproject > init:

myproject > hello:
     [echo] Hello

BUILD FINISHED

Total time: 0.1383 seconds

Using Properties Files

$> phing
Buildfile: /tmp/myproject/build.xml

myproject > prop:
     [echo] Loading default build.properties
 [property] Loading /tmp/myproject/build.properties

myproject > local-prop-check:

myproject > local-prop:
     [echo] Loading custom properties!
 [property] Loading /tmp/myproject/local.properties

myproject > init:

myproject > hello:
     [echo] Hello myuser

BUILD FINISHED

Total time: 0.0493 seconds

Phing via Puppet

# Running Liquibase via Phing
exec { 'phing-db-update':
  command   => './vendor/bin/phing db:update',
  cwd       => '/vagrant/',
  require   => [Exec['composer-run'], Package['postgresql-server-9.3']],
  logoutput => true
}

# Generate cache files
exec { 'phing-generate-cache':
  command   => './vendor/bin/phing app:warm-cache',
  cwd       => '/vagrant',
  require   => Exec['composer-run'],
  logoutput => true
}

Liquibase

Liquibase Changeset

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">

    <changeSet id="1" author="shochdoerfer">
        <createTable tableName="person">
            <column name="id" type="int" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="firstname" type="varchar(50)"/>
            <column name="lastname" type="varchar(50)">
                <constraints nullable="false"/>
            </column>
            <column name="state" type="char(2)"/>
        </createTable>
    </changeSet>
</databaseChangeLog>

Liquibase Changeset

--liquibase formatted sql

--changeset shochdoerfer:1
create table person (
  id int not null primary key,
  firstname varchar(80),
  lastname varchar(80) not null,
  state varchar(2)
);

Liquibase Filesystem layout

/home/shochdoerfer/Projects/sample-project
   |-build
   |-config
   |-db
   |---dev
   |-----changesets
   |-------141001-01-initial-structure.xml
   |-------141005-02-offer-number.xml
   |-------141005-02-statistics-table.xml
   |-----liquibase.xml
   |---prod   
   |-puppet
   |-shell
   |-src
   |-tests
   |-webroot

Running Liquibase

$> ./vendor/bin/phing db:update

$> liquibase --driver=com.mysql.jdbc.Driver \
     --classpath=/opt/liquibase/mysql-connector-java-5.1.21-bin.jar \
     --changeLogFile=db/dev/liquibase.xml \
     --url="jdbc:mysql://localhost/mydatabase" \
     --username=myuser \
     --password=mypass \
     migrate

Buildserver Integration

Configure Node

Configure Node Environment

Configure Job

Copy configuration files

Validate composer.json

Install Dependencies

Run the build

Deploy the build

<?xml version="1.0"?>
<project name="myproject" default="-init">

    <target name="app:deploy-dev" depends="-init, app:build">
	<echo message="Deploying to ${deploy.dev.ssh.host}"/>
	<exec executable="rsync" passthru="true" checkreturn="true">
	    <arg value="--recursive" />
	    <arg value="--exclude=.git" />
	    <arg value="--delete" />
	    <arg value="-l" />
	    <arg value="." />
	    <arg value="${deploy.dev.deploydir}" />
	</exec>

	<echo message="Running deploy script..."/>
	<exec executable="ssh" passthru="false" checkreturn="true">
	    <arg value="${deploy.dev.ssh}" />
	    <arg value="${deploy.dev.deploydir}/shell/deploy.sh" />
	</exec>
    </target>
</project>

Thank you! Questions?