From Vagrant to Production

From Vagrant
to Production

Stephan Hochdörfer // February 19 2016

About me

Stephan Hochdörfer
Head of Technology, bitExpert AG
(Mannheim, Germany)
S.Hochdoerfer@bitExpert.de
@shochdoerfer
#PHP, #DevOps, #Automation
#phpugffm, #phpugmrn, #unKonf

Our situation (~5 years ago)

Goal: Improve the situation

The challenge

How to deal with dependencies?
How to deal with database
schema changes?
How to isolate the
development environment?
How to hide the complexitiy?

As simple as possible

git clone && vagrant up

Vagrant benefits

Environment per project
Versionable
Shared across the team
No more: "Works on my
machine!"

Vagrantfile

VAGRANTFILE_API_VERSION = "2"

Vagrant.require_version ">= 1.8.0"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    config.vm.define "jessie" do |jessie|
        jessie.vm.box      = 'debian-jessie-64'
        jessie.vm.box_url  = 'http://vagrantboxes.loc/jessie64.box'
        jessie.vm.hostname = 'myhost'

        jessie.vm.synced_folder ".", "/vagrant", owner: "www-data",
        group: "vagrant"
    end
end

Run the virtual machine

$> vagrant up

Bringing machine 'box' up with 'virtualbox' provider...
[box] Importing base box 'debian-jessie-64'...
[box] Matching MAC address for NAT networking...
[box] Setting the name of the VM...
[box] Clearing any previously set forwarded ports...
[box] Creating shared folders metadata...
[box] Clearing any previously set network interfaces...
[box] Preparing network interfaces based on configuration...
[box] Forwarding ports...
[box] -- 22 => 2222 (adapter 1)
[box] Booting VM...
[box] Waiting for machine to boot. This may take a few minutes...
[box] Machine booted and ready!
[box] Setting hostname...
[box] Mounting shared folders...
[box] -- /vagrant

Vagrantbox.es

Vagrant Provider

Virtualbox
VMware
Hyper-V
Docker
KVM
...

Vagrant Plugins

vagrant-cachier
vagrant-hostmanager
vagrant-librarian-puppet
vagrant-dns
...

Vagrant Cachier Plugin

$ vagrant plugin install vagrant-cachier

VAGRANTFILE_API_VERSION = "2"
Vagrant.require_version ">= 1.8.0"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    if Vagrant.has_plugin?("vagrant-cachier")
       config.cache.enable :apt
       config.cache.enable :apt_lists
       config.cache.enable :composer
    end

    config.vm.define "jessie" do |jessie|
        jessie.vm.box      = 'debian-jessie-64'
        jessie.vm.box_url  = 'http://vagrantboxes.loc/jessie64.box'
        jessie.vm.hostname = 'myhost'

        jessie.vm.synced_folder ".", "/vagrant", owner: "www-data",
        group: "vagrant"
    end
end

PuPHPet

Simple Puppet script

$default_packages = [ 'expect', 'curl', 'git']
package { $default_packages :
    ensure => present,
}

host { 'myhost.loc':
    ip     => '127.0.1.1'
}

Running Puppet

$> puppet apply init.pp

notice: /Stage[main]//Package[curl]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Package[git]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Package[expect]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]//Host[myhost.loc]/ensure: created
notice: Finished catalog run in 1.86 seconds

Directory layout

/home/shochdoerfer/Projects/sample-project
   |-build
   |-config
   |-puppet
   |---hieradata
   |-----common.yaml
   |---manifests
   |-----init.pp
   |---modules
   |---src
   |-----sample-project
   |-------manifests
   |-------templates
   |---Puppetfile
   |-shell
   |---bootstrap.sh
   |-src
   |-tests
   |-webroot

librarian-puppet

You can use librarian-puppet to manage the
puppet modules your infrastructure depends on.

Puppetfile

forge "https://forgeapi.puppetlabs.com"

# external puppet modules that should be downloaded
mod 'example42-apache', '2.1.7'
mod 'example42-php', '2.0.18'
mod 'puppetlabs-apt', '1.3.0'
mod 'puppetlabs-stdlib', '4.1.0'
mod 'puppetlabs-postgresql', '3.3.3'

# copy local module in the modules directory
mod 'customer1-project1',
  :path => './src/sample-project'

Running librarian-puppet

$> gem install librarian-puppet

$> librarian-puppet install

$> librarian-puppet update

Vagrant Provisioning

VAGRANTFILE_API_VERSION = "2"

Vagrant.require_version ">= 1.8.0"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    config.vm.define "jessie" do |jessie|
        jessie.vm.box      = 'debian-jessie-64'
        jessie.vm.box_url  = 'http://vagrantboxes.loc/jessie64.box'
        jessie.vm.hostname = 'myhost'

        jessie.vm.synced_folder ".", "/vagrant", owner: "www-data",
        group: "vagrant"

        jessie.vm.provision "shell" do |shell|
          shell.path = 'shell/bootstrap.sh'
          shell.args = "-d /vagrant/puppet/ -i init.pp"
        end
    end
end

bootstrap.sh

#!/usr/bin/env bash

while getopts d:i:h flag; do
  case $flag in
    d) PUPPET_DIR=$OPTARG; ;;
    i) PUPPET_FILE=$OPTARG; ;;
    h) HIERA_CONFIG=$OPTARG; ;;
    ?) exit; ;;
  esac
done

if [ `gem query --local | grep librarian-puppet | wc -l` -eq 0 ]; then
  gem install librarian-puppet
  cd $PUPPET_DIR && librarian-puppet install --clean
else
  cd $PUPPET_DIR && librarian-puppet update
fi

puppet apply --verbose --debug --hiera_config $HIERA_CONFIG --modulepath=$PUPPET_DIR/modules/ $PUPPET_DIR/manifests/$PUPPET_FILE

Composer via Puppet

$default_packages = [ 'curl', 'git', 'php7-cli' ]
package { $default_packages: ensure => present, }

exec { 'composer':
  command => 'curl -sS https://getcomposer.org/installer | php',
  cwd     => '/usr/local/bin',
  creates => '/usr/local/bin/composer.phar',
  timeout => 0,
  require => [Package['php7-cli'], Package['curl']]
}

exec { 'composer-run':
  command     => 'composer.phar install',
  cwd         => '/vagrant/',
  environment => 'COMPOSER_HOME=/root/',
  timeout     => 0,
  require  => [Package['git'], Package['php7-cli'], Exec['composer']]
}

Publish your packages

Private packages?

Satis is a ultra-lightweight, static
file-based version of packagist.

Toran acts as a proxy for
Packagist and GitHub.

Install Satis

$> composer.phar create-project composer/satis --stability=dev

Running Satis

{
    "name": "My own repo",
    "homepage": "https://satis.loc",
    "repositories": [
        { "type": "vcs", "url": "https://git.loc/repo1.git" },
        { "type": "vcs", "url": "https://git.loc/repo2.git" }
    ],
    "require-all": true,
    "archive": {
        "directory": "dist",
        "format": "zip",
        "skip-dev": false
    }
}

$> php bin/satis build config.json web/

Generated Satis Repo

Dealing with private repos

HTTP Basic Auth

$> composer.phar install
Loading composer repositories with package information
    Authentication required (satis.loc):
      Username: myuser
      Password:
Do you want to store credentials for satis.loc in 
~/.composer/auth.json ? [Yn]

HTTP Basic Auth (auth.json)

{
    "http-basic": {
        "satis.loc": {
            "username": "myuser",
            "password": "mypassword"
        },
        "git.loc": {
            "username": "mygituser",
            "password": "mypassword"
        }
    }
}

expect

« [...] is a program that "talks" to other interactive
programs according to a script. » - expect(1)

Composer via expect

#!/bin/sh
expect -c " 
set timeout -1 
spawn /usr/local/bin/composer.phar install
expect { 
    -re { Username: } { 
	send \"myusername\r\"
	exp_continue 
    } 
    -re { Password: } { 
	send \"mypassword\r\"
	exp_continue 
    } 
    -re {^Generating autoload files } 
}"

expect Vagrant configuration

VAGRANTFILE_API_VERSION = "2"

Vagrant.require_version ">= 1.8.0"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

    config.vm.define "jessie" do |jessie|
        jessie.vm.box      = 'debian-jessie-64'
        jessie.vm.box_url  = 'http://vagrantboxes.loc/jessie64.box'
        jessie.vm.hostname = 'myhost'
        jessie.vm.synced_folder ".", "/vagrant", owner: "www-data",
        group: "vagrant"

        jessie.vm.provision "shell" do |shell|
          USER=ENV['COMPOSER_USER']
          PASS=ENV['COMPOSER_PASS']

          shell.path ='shell/bootstrap.sh'
          shell.args ="-d /vagrant/puppet/ -i init.pp -u USER -p PASS"
        end
    end
end

expect bootstrap.sh

#!/usr/bin/env bash

# [...]

if [ `gem query --local | grep librarian-puppet | wc -l` -eq 0 ]; then
  gem install librarian-puppet
  cd $PUPPET_DIR && librarian-puppet install --clean
else
  cd $PUPPET_DIR && librarian-puppet update
fi

export FACTER_user="${COMPOSER_USER}"
export FACTER_pass="${COMPOSET_PASS}"

puppet apply --verbose --debug --hiera_config $HIERA_CONFIG --modulepath=$PUPPET_DIR/modules/ $PUPPET_DIR/manifests/$PUPPET_FILE

Running expect in Puppet

exec { 'composer-run':
  command => "expect -c '
    set timeout -1
    spawn /usr/local/bin/composer.phar install
    expect {
      -re { Username: } {
	send \"${::user}\\r\"
	exp_continue
      }
      -re { Password: } {
	send \"${::pass}\\r\"
	exp_continue
      }
      -re {^Generating autoload files }
    }'",
  cwd     => '/vagrant',
  timeout => 0,
  environment => 'COMPOSER_HOME=/root/',
  require => [Package['expect'], Package['git'], Package['php7-cli'],
  File['composer']]
}

Running expect in Puppet

exec { 'composer-run':
  command => "expect -c '
    set timeout -1
    spawn /usr/local/bin/composer.phar dumpautoload -o --no-dev
    expect {
      -re { Username: } {
	send \"${::user}\\r\"
	exp_continue
      }
      -re { Password: } {
	send \"${::pass}\\r\"
	exp_continue
      }
      -re {^Generating autoload files }
    }'",
  cwd     => '/vagrant',
  timeout => 0,
  environment => 'COMPOSER_HOME=/root/',
  require => [Package['expect'], Package['git'], Package['php7-cli'],
  File['composer']]
}

Domain-specific language

A glue for 3rd party tools

Why Phing?

Runs everywhere where
PHP runs
No additional depencendies
needed (e.g. Java, ...)
More than 120 predefined
tasks to choose from
Easy to extend by writing
custom tasks in PHP

Install Phing

{
    "require-dev" : {
	"phing/phing" : "2.5.*@stable"
    },
    "repositories" : [{
	    "type" : "composer",
	    "url" : "https://satis.loc/"
    }],
}

$> ./vendor/bin/phing -v
Phing 2.5.0

build.properties

phpunit.path=vendor/bin/phpunit
phpunit.junit.log=build/logs/junit.xml
phpunit.coverage.clover=build/logs/clover.xml
phpunit.coverage.html=build/coverage

phpcs.path=vendor/bin/phpcs
phpcs.ignore=/Phing/
phpcs.log=build/logs/checkstyle.xml

deploy.dev.ssh.user=
deploy.dev.ssh.host=dev.loc
deploy.dev.tmp.dir=/deploy/
deploy.dev.install.dir=/srv

local.properties

deploy.dev.ssh.user=myuser

Using Properties Files

<?xml version="1.0"?>
<project name="myproject" default="hello">

    <target name="hello" depends="init">
	<echo msg="Hello ${deploy.dev.ssh.user}" />
    </target>

    <target name="init" depends="prop, local-prop">
	<!-- some more init logic -->
    </target>

    <target name="prop">
	<echo message="Loading default build.properties"/>

	<property file="build.properties" />
    </target>

Using Properties Files

    <target name="local-prop" if="local-prop.exists"
	depends="local-prop-check">

	<echo message="Loading custom properties!"/>
	<property file="local.properties" override="true"/>
    </target>

    <target name="local-prop-check">
	<available file="local.properties"
	    property="local-prop.exists" />
    </target>
</project>

Using Properties Files

$> phing
Buildfile: /tmp/myproject/build.xml

myproject > prop:
     [echo] Loading default build.properties
 [property] Loading /tmp/myproject/build.properties

myproject > local-prop-check:

myproject > local-prop:

myproject > init:

myproject > hello:
     [echo] Hello

BUILD FINISHED

Total time: 0.1383 seconds

Using Properties Files

$> phing
Buildfile: /tmp/myproject/build.xml

myproject > prop:
     [echo] Loading default build.properties
 [property] Loading /tmp/myproject/build.properties

myproject > local-prop-check:

myproject > local-prop:
     [echo] Loading custom properties!
 [property] Loading /tmp/myproject/local.properties

myproject > init:

myproject > hello:
     [echo] Hello myuser

BUILD FINISHED

Total time: 0.0493 seconds

Running Phing via Puppet

# Running Liquibase via Phing
exec { 'phing-db-update':
  command   => './vendor/bin/phing db:update',
  cwd       => '/vagrant/',
  require   => [Exec['composer-run'], Package['postgresql-server-9.3']],
  logoutput => true
}

# Generate cache files
exec { 'phing-generate-cache':
  command   => './vendor/bin/phing app:warm-cache',
  cwd       => '/vagrant',
  require   => Exec['composer-run'],
  logoutput => true
}

Liquibase

Liquibase Changeset

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.1.xsd">

    <changeSet id="1" author="shochdoerfer">
        <createTable tableName="person">
            <column name="id" type="int" autoIncrement="true">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="firstname" type="varchar(50)"/>
            <column name="lastname" type="varchar(50)">
                <constraints nullable="false"/>
            </column>
            <column name="state" type="char(2)"/>
        </createTable>
    </changeSet>
</databaseChangeLog>

Liquibase Changeset

--liquibase formatted sql

--changeset shochdoerfer:1
create table person (
  id int not null primary key,
  firstname varchar(80),
  lastname varchar(80) not null,
  state varchar(2)
);

Liquibase Filesystem layout

/home/shochdoerfer/Projects/sample-project
   |-build
   |-config
   |-db
   |---dev
   |-----changesets
   |-------141001-01-initial-structure.xml
   |-------141005-01-offer-number.xml
   |-------141005-02-statistics-table.xml
   |-----liquibase.xml
   |---prod   
   |-puppet
   |-shell
   |-src
   |-tests
   |-webroot

Running Liquibase

$> ./vendor/bin/phing db:update

$> liquibase --driver=com.mysql.jdbc.Driver \
     --classpath=/opt/liquibase/mysql-connector-java-5.1.21-bin.jar \
     --changeLogFile=db/dev/liquibase.xml \
     --url="jdbc:mysql://localhost/mydatabase" \
     --username=myuser \
     --password=mypass \
     migrate

Buildserver Integration

Gitlab Webhook

Jenkins buildnodes

Configure build node

Configure build node

Configure job

Bind job to host

Copy configuration files

Validate composer.json

Running security checks

Running Puppet via Jenkins

Running the build

Running the build

Deploy the build

<?xml version="1.0"?>
<project name="myproject" default="-init">

    <target name="app:deploy-dev" depends="-init, app:build">
	<echo message="Deploying to ${deploy.dev.ssh.host}"/>
	<exec executable="rsync" passthru="true" checkreturn="true">
	    <arg value="--recursive" />
	    <arg value="--exclude=.git" />
	    <arg value="--delete" />
	    <arg value="-l" />
	    <arg value="." />
	    <arg value="${deploy.dev.deploydir}" />
	</exec>

	<echo message="Running deploy script..."/>
	<exec executable="ssh" passthru="false" checkreturn="true">
	    <arg value="${deploy.dev.ssh}" />
	    <arg value="${deploy.dev.deploydir}/shell/deploy.sh" />
	</exec>
    </target>
</project>

Thank you! Questions?

Do not forget to rate the talk:
https://joind.in/talk/a0502

From Vagrantto Production

About me

Our situation (~5 years ago)

Goal: Improve the situation

The challenge

As simple as possible

Vagrant benefits

Vagrantfile

Run the virtual machine

Vagrantbox.es

Vagrant Provider

Vagrant Plugins

Vagrant Cachier Plugin

PuPHPet

Simple Puppet script

Running Puppet

Directory layout

librarian-puppet

Puppetfile

Running librarian-puppet

Vagrant Provisioning

bootstrap.sh

Composer via Puppet

Publish your packages

Private packages?

Install Satis

Running Satis

Generated Satis Repo

Dealing with private repos

HTTP Basic Auth

HTTP Basic Auth (auth.json)

expect

Composer via expect

expect Vagrant configuration

expect bootstrap.sh

Running expect in Puppet

Running expect in Puppet

Domain-specific language

A glue for 3rd party tools

Why Phing?

Install Phing

build.properties

local.properties

Using Properties Files

Using Properties Files

Using Properties Files

Using Properties Files

Running Phing via Puppet

Liquibase

Liquibase Changeset

Liquibase Changeset

Liquibase Filesystem layout

Running Liquibase

Buildserver Integration

Gitlab Webhook

Jenkins buildnodes

Configure build node

Configure build node

Configure job

Bind job to host

Copy configuration files

Validate composer.json

Running security checks

Running Puppet via Jenkins

Running the build

Running the build

Deploy the build

From Vagrant
to Production