Create the Template

The next step is to create a template yaml file with the definitions to deploy the HaProxy Load Balancer, Tomcat Server and Wildbook Application, and MySQL database server.

Refer to the Template Syntax section in this document for reference.

Create a new template file

Using your preferred text editor create a new template yaml file. Add the following statements, feel free to substitute values as appropriate.

template_name: "Tomcat Wildbook Three Server Stack using Chef"
template_author: "Gary Forghetti"
template_version: "1.00"
description: "Deploys a Tomcat Wildbook Application 3 server stack using Chef"
long_description: "Deploys a 3 server stack consisting of HaProxy Load Balancer, a Tomcat Server running the Wildbook Application and a MySQL database server, using Chef."

imports:
  - base_types.yaml                                                   # Import the DCM base type definitions that are referenced in this template

Define a custom node type

In this tutorial a custom base server node type will be created. The custom base server node type will be used to define the three server nodes in the node_templates section of the template. This will reduce the amount of definitions required. The three server nodes will inherit the properties of the custom base server node type. Refer to the Custom node types section in this document for more details.

Add the statements to the yaml file to define the custom base server node type named base_vm.

node_types:

  ##################################################################################################################################################
  # This defines a custom node type which will be referenced (inherited from) the two server nodes types below in this template
  ##################################################################################################################################################
  base_vm:
    type: dcm.nodes.Server
    properties:                                                                # Retrieve these server launch properties from the "inputs"
      cloud: { get_input: [account_region_zone_selector, cloud] }              # Retrieve cloud from the AccountRegionSelector
      cloudAccountId: { get_input: [account_region_zone_selector, accountId] } # Retrieve the cloud account ID from the AccountRegionSelector
      region: { get_input: [account_region_zone_selector, region] }            # Retrieve region from the AccountRegionSelector
      zone: { get_input: [account_region_zone_selector, zone] }                # Retrieve zone from the AccountRegionSelector
      platform: { get_input: [product_selector, platform] }                    # Retrieve platform from the Product selector
      architecture: { get_input: [product_selector, architecture] }            # Retrieve architecture from the Product selector
      product: { get_input: [product_selector, product] }                      # Retrieve product from the Product selector
      image: { get_input: [product_selector, image] }                          # Retrieve machine from the Product selector
      serverProductId: { get_input: [product_selector, serverProductId] }      # Retrieve server product id from the Product selector
      startupScript:  |
        #!/bin/bash
        ##########################################################################################################################################
        # Install curl on Debian/Ubuntu if necessary
        ##########################################################################################################################################
        which apt-get && apt-get update -qq && (which curl || apt-get install curl -y)

        ##########################################################################################################################################
        # Install curl on RHEL/CentOS if necessary
        ##########################################################################################################################################
        which yum && yum -q makecache && (which curl || yum install curl -y)

        ##########################################################################################################################################
        # Install the Dell Cloud Manager agent, point it to the Dell Cloud Manager server and configure it to startup at boot time.
        ##########################################################################################################################################
        export DCM_URL=${dcm.callback.url}

        ##########################################################################################################################################
        # The -Z argument configures the Dell Cloud Manager agent and allows it to accept unknown certificates.
        # The -o argument will install the latest Chef Client 
        # This is only recommended for testing and should not be done in a production environment.
        ##########################################################################################################################################
        curl -L --retry 10 https://linux-stable-agent.enstratius.com/installer.sh | bash -s - --url $DCM_URL -o --on-boot -Z

        ##########################################################################################################################################
        # Start the Dell Cloud Manager agent.
        ##########################################################################################################################################
        /etc/init.d/dcm-agent start

Define topology_template and inputs

topology_template:

  inputs:

Define AccountRegionSelector

The next step is to define a AccountRegionSelector to allow the Dell Cloud Manager console end user the ability to select the cloud, region and data center for launching the server in the stack.

  ################################################################################################################################################
  # This defines the AccountRegionSelector which allows the user to select the Cloud, Region and Datacenter
  ################################################################################################################################################
  account_region_zone_selector:                                     # Define the section for the Cloud, Region and Datacenter selection boxes
    type: dcm.inputs.AccountRegionSelector                          # Input type is dcm.inputs.accountRegionSelector 
    properties:
      regions:                                                      # Define the Cloud and Regions
        "Amazon":                                                   # Amazon
          "us-east-1": [ ]                                          # All data centers for the us-east-1 region
          "us-west-1": ["us-west-1a", "us-west-1c"]
          "us-west-2": ["us-west-2a", "us-west-2b", "us-west-2c"]
          "eu-west-1": ["eu-west-1a", "eu-west-1c"]

Define Product Selector

The next step is to define a Product selector to allow the Dell Cloud Manager console end user the ability to select the server product for launching the server(s) in the stack. This example will use the Amazon cloud and a subset of the regions and server products. An Ubuntu trusty 14.04 machine image will be defined in the template to be used for creating the server.

Note

If you do not have access to the Amazon cloud or wish to use a different cloud provider’s cloud that Dell Cloud Manager supports, you will need to make the necessary changes below for the cloud, region(s), image(s) and products.

Add the statements to the yaml file to define the inputs product_selector. The inputs: statement must be indented inside the topology_template: statement. Refer to the Product Selector section in this document for reference.

    ################################################################################################################################################
    # This defines the Product selector which allows the user to select the server product size
    ################################################################################################################################################        
    product_selector:                                                 # Define the product selector so the user can select the cloud and region
      type: dcm.inputs.Product                                        # Input type is dcm.inputs.Product
      properties:
        accountRegionSelector: account_region_zone_selector           # This connects the AccountRegionSelector to the Product selector
        platform: UNIX                                                # Virtual machine images are Linux
        architecture: I64                                             # 64 bit images
        productMappings:
          "Amazon":                                                   # Amazon cloud
            "us-east-1":                                              # us-east-1 Region
              image: "ami-c4edc0d3"                                   # Machine image identifier for an Ubuntu image in this region
              products: ['t1.micro', 'm1.small', 'm1.medium']         # The virtual machine product sizes for this region
            "us-west-1":                                              # us-west-1 Region
              image: "ami-e7035687"                                   # Machine image identifier for an Ubuntu image in this region
              products: ['t1.micro', 'm1.small', 'm1.medium']         # The virtual machine product sizes for this region
            "us-west-2":                                              # us-west-2 Region
              image: "ami-578c2f37"                                   # Machine image identifier for an Ubuntu image in this region
              products: ['t1.micro', 'm1.small', 'm1.medium']         # The virtual machine product sizes for this region
            "eu-west-1":                                              # eu-west-1 Region
              image: "ami-e6a1f795"                                   # Machine image identifier for an Ubuntu image in this region
              products: ['t1.micro', 'm1.small', 'm1.medium']         # The virtual machine product sizes for this region

Define outputs

In this tutorial 3 outputs will be defined and displayed on the Stack Overview page.

  • The Wildbook application User Id
  • The Wildbook application password
  • The Web URL of the HaProxy which will redirect to the Wildbook application

Add the statements to the yaml file to define the outputs. The outputs: statement must be indented inside the topology_template: statement and aligned with the inputs: statement. Refer to the Template Outputs section in this document for reference.

##################################################################################################################################################
# This defines the three outputs to appear on the DCM console Stack overview page (Wildbook userid, password and URL)
##################################################################################################################################################
outputs:                                                        # Define outputs
  application_group:                                            # Create a Group
    type: dcm.outputs.DisplayGroup                              # It's a displayGroup
    properties:
      displayName: "Tomcat Server Information"                  # Set the display name for the group containing the outputs which appears on the Stack Overview page

  user_id:                                                      # Define another output for the Tomcat Wildbook user id
    type: string                                                # It's a string output
    description: Wildbook user id                               # Set the description for the string output
    value: tomcat                                               # Set the value to the constant string "tomcat"
    properties:
      displayName: User id                                      # Set the display name (label) for the string output to "User Id"
      displayGroup: application_group                           # Place this output in the displayGroup named application_group

  password:                                                     # Define another output
    type: password                                              # It's a password output
    description: Wildbook password                              # Set the description for the string output
    value: tomcat123                                            # Set the value to the constant string "tomcat123"
    properties:
      displayName: Password                                     # Set the display name (label) for the string output to "Password"
      displayGroup: application_group                           # Place this output in the displayGroup named application_group

  link:                                                         # Define another output
    type: dcm.outputs.Uri                                       # It's a URI output
    # Set the initial value for the HaProxy Wildbook Application URL
    value: http://xxxxxxxxxxxxx
    properties:
      host: {get_attribute: [lb_vm, publicIpAddress]}           # Set the hostname/ipaddress of the URI to the public IP address of the HaProxy server
      displayName: Wildbook Application URL                     # Set the display name (label) for the URI output
      displayGroup: application_group                           # Place this output in the displayGroup named application_group

Define the node templates

The next step is to define 9 node_templates.

  1. A node_template to create a server for the the HaProxy Load Balancer server.
  2. A node_template to create a server for the the MySQL database server.
  3. A node_template to create a server for the Tomcat Wildbook Application.
  4. A node_template to create a firewall for the HaProxy Load Balancer server.
  5. A node_template to create a firewall for the MySQL database server.
  6. A node_template to create a firewall for the Tomcat Server.
  7. A node_template to install the HaProxy Load Balancer server using Chef.
  8. A node_template to install the MySQL database server using Chef.
  9. A node_template to install the Tomcat Server abd Wildbook Application using Chef

Define the HaProxy LB Server

Add the statements to define the HaProxy Load Balancer server. In this example the server name will be hardcoded to be wildbook-lb-vm. The remaining server properties will be retrieved from the corresponding properties derived from the product_selector and the Dell Cloud Manager console end user’s Launch Blueprint selections. Refer to the Server node section in this document for reference.

node_templates:

  ##################################################################################################################################################
  # This node_template defines a virtual machine node named "lb_vm" which will host the HaProxy Load Balancer
  ##################################################################################################################################################
  lb_vm:
    type: base_vm                                                 # Define lb server and inherit the properties from the custom base server node base_vm
    properties:
      name: "wildbook-lb-vm"                                      # !!!! Do not change this name, the chef recipe requires this name !!!!!!
    requirements:
      - firewall: lb_vm_firewall_rules                            # This virtual machine node has a requirement on the lb_vm firewall
        relationship_type: tosca.relationships.DependsOn

Define the MySQL Database Server

Add the statements to define the MySQL database server. In this example the server name will be hardcoded to be wildbook-mysql-vm. The remaining server properties will be retrieved from the corresponding properties derived from the product_selector and the Dell Cloud Manager console end user’s Launch Blueprint selections.

Refer to the Server node section in this document for reference.

##################################################################################################################################################
# This node_template defines a virtual machine node named "db_vm" which will host the MySQL database server
##################################################################################################################################################
db_vm:
  type: base_vm                                            # Define db server and inherit the properties from the custom base server node base_vm
  properties:
    name: wildbook-mysql-vm                                # !!!! Do not change this name, the chef recipe requires this name !!!!!!
  requirements:                                            # This virtual machine node has a requirement on the db_vm firewall
    - firewall: db_vm_firewall_rules
      relationship_type: tosca.relationships.DependsOn

Define the Tomcat Server

Add the statements to define the Tomcat server. In this example the server name will be hardcoded to be wildbook-web-vm. The remaining server properties will be retrieved from the corresponding properties derived from the product_selector and the Dell Cloud Manager console end user’s Launch Blueprint selections.

Refer to the Server node section in this document for reference.

##################################################################################################################################################
# This node_template defines a virtual machine node named "web_vm" which will host the Tomcat Server running the Wildbook Application
##################################################################################################################################################
web_vm:
  type: base_vm                                                 # Define web server and inherit the properties from the custom base server node base_vm
  properties:
    name: wildbook-web-vm                                       # !!!! Do not change this name, the chef recipe requires this name !!!!!!
  requirements:                                                 # This virtual machine node has a requirement on the web_vm firewall and the MySQL Server
    - firewall: web_vm_firewall_rules
      relationship_type: tosca.relationships.DependsOn
    - db: db_vm
      relationship_type: tosca.relationships.ConnectsTo

Define the lb_vm Firewall

Add the statements to define the firewall lb_vm_firewall_rules. Define a rule to open port 80 (HTTP).

Refer to the FirewallGroup node section in this document for reference.

###################################################################################################################################################
# This node_template defines a firewall which opens the HaProxy port 80
###################################################################################################################################################
lb_vm_firewall_rules:
  type: dcm.nodes.FirewallGroup                                              # This is a firewall
  properties:                                                                # Retrieve the cloud properties from the "inputs"
    name: "fw-lb-server"                                                     # Define a name for the firewall (it will get prefixed with the stack name)
    cloud: { get_input: [account_region_zone_selector, cloud] }              # Retrieve the cloud from the AccountRegionSelector
    cloudAccountId: { get_input: [account_region_zone_selector, accountId] } # Retrieve the cloud account ID from the AccountRegionSelector
    region: { get_input: [account_region_zone_selector, region] }            # Retrieve the region from the AccountRegionSelector
    zone: { get_input: [account_region_zone_selector, zone] }                # Retrieve the zone from the AccountRegionSelector
    rules:
      - remote_ip_prefix: 0.0.0.0/0
        port: 80                                                             # Define a rule to open port 80  (HTTP)

Define the db_vm Firewall

Add the statements to define the firewall db_vm_firewall_rules. Define a rule to open port 3306 (MySQL).

Refer to the FirewallGroup node section in this document for reference.

###################################################################################################################################################
# This node_template defines a firewall which opens the MySQL server port 3306
###################################################################################################################################################
db_vm_firewall_rules:
  type: dcm.nodes.FirewallGroup                                              # This is a firewall
  properties:                                                                # Retrieve the cloud properties from the "inputs"
    name: "fw-mysql-server"                                                  # Define a name for the firewall (it will get prefixed with the stack name)
    cloud: { get_input: [account_region_zone_selector, cloud] }              # Retrieve the cloud from the AccountRegionSelector
    cloudAccountId: { get_input: [account_region_zone_selector, accountId] } # Retrieve the cloud account ID from the AccountRegionSelector
    region: { get_input: [account_region_zone_selector, region] }            # Retrieve the region from the AccountRegionSelector
    zone: { get_input: [account_region_zone_selector, zone] }                # Retrieve the zone from the AccountRegionSelector
    rules:
      - remote_ip_prefix: 0.0.0.0/0
        port: 3306                                                           # Define a rule to open port 3306  (MySQL)

Define the web_vm Firewall

Add the statements to define the firewall web_vm_firewall_rules. Define a rule to open port 8080 (HTTP Tomcat).

Refer to the FirewallGroup node section in this document for reference.

###################################################################################################################################################
# This node_template defines a firewall which opens the HTTP port 8080 for the Tomcat server
###################################################################################################################################################
web_vm_firewall_rules:
  type: dcm.nodes.FirewallGroup                                              # This is a firewall
  properties:                                                                # Retrieve the cloud properties from the "inputs"
    name: "fw-tomcat-server"                                                 # Define a name for the firewall (it will get prefixed with the stack name)
    cloud: { get_input: [account_region_zone_selector, cloud] }              # Retrieve the cloud from the AccountRegionSelector
    cloudAccountId: { get_input: [account_region_zone_selector, accountId] } # Retrieve the cloud account ID from the AccountRegionSelector
    region: { get_input: [account_region_zone_selector, region] }            # Retrieve the region from the AccountRegionSelector
    zone: { get_input: [account_region_zone_selector, zone] }                # Retrieve the zone from the AccountRegionSelector
    rules:
      - remote_ip_prefix: 0.0.0.0/0
        port: 8080                                                           # Define a rule to open port 8080  (HTTP)

Define Chef for HaProxy

Add the statements to define the Chef node to install the HaProxy Load Balancer server.

Attention

The chefServer: value must exactly match the name of the Chef Account you configured in Dell Cloud Manager.

Refer to the Chef section in this document for reference.

###################################################################################################################################################
# This node_template will result in Chef installing the HaProxy Load Balancer on the launched virtual machine named "lb_vm"
###################################################################################################################################################
chef_lb:
  type: dcm.nodes.Chef                                            # This is a Chef node
  properties:
    chefServer: "ACME Chef"                                       # Name of the Chef Server Account defined to the Dell Cloud Manager Server
    environment: "_default"                                       # Chef environment
    roles: ['haproxy-wildbook']                                   # Chef role which contains the cookbooks and recipes to install HaProxy LB
  requirements:
    - host: lb_vm                                                 # This requires (is "HostedOn") the Server defined in the template named "lb_vm"
      relationship_type: tosca.relationships.HostedOn
    - web_relationship: chef_tomcat                               # It also requires ("DependsOn") the Tomcat web server to be created
      relationship_type: tosca.relationships.DependsOn

Define Chef for MySQL

Add the statements to define the Chef node to install the MySQL database server.

Attention

The chefServer: value must exactly match the name of the Chef Account you configured in Dell Cloud Manager.

Refer to the Chef section in this document for reference.

###################################################################################################################################################
# This node_template will result in Chef installing the MySQL server on the launched virtual machine named "db_vm"
###################################################################################################################################################
chef_mysql:
  type: dcm.nodes.Chef                                            # This is a Chef node
  properties:
    chefServer: "ACME Chef"                                       # Name of the Chef Server Account defined to the Dell Cloud Manager Server
    environment: "_default"                                       # Chef environment
    roles: ['mysql-wildbook']                                     # Chef role which contains the cookbooks and recipes to install MySQL Server
  requirements:
    - host: db_vm                                                 # This requires (is "HostedOn") the Server defined in the template named "db_vm"
      relationship_type: tosca.relationships.HostedOn

Define Chef for Tomcat

Add the statements to define the Chef node to install the Tomcat Server and Wildbook Application.

Attention

The chefServer: value must exactly match the name of the Chef Account you configured in Dell Cloud Manager.

Refer to the Chef section in this document for reference.

###################################################################################################################################################
# This node_template will result in Chef installing the Tomcat Server on the launched virtual machine named "web_vm"
###################################################################################################################################################
chef_tomcat:
  type: dcm.nodes.Chef                                            # This is a Chef node
  properties:
    chefServer: "ACME Chef"                                       # Name of the Chef Server Account defined to the Dell Cloud Manager Server
    environment: "_default"                                       # Chef environment
    roles: ['tomcat-wildbook']                                    # Chef role which contains the cookbooks and recipes to install Tomcat
  requirements:
    - host: web_vm                                                # This requires (is "HostedOn") the Server defined in the template named "web_vm"
      relationship_type: tosca.relationships.HostedOn
    - db_relationship: chef_mysql                                 # It also requires ("DependsOn") the database server to be created
      relationship_type: tosca.relationships.DependsOn

Define Auto Scaling Policy

In this tutorial 2 auto scaling policies will be defined to auto scale the Tomcat Server. Refer to the Auto scaling section in this document for reference.

  • A scale up will be performed when the last 3 consecutive periods of cpu idle time are less than 20%.
  • A scale down will be performed when the last 2 consecutive periods of cpu idle time are greate than 80%.

Add the following statments. Make sure the groups: statement is aligned exactly with the topology_template statement (same indentation). The members: statement must specify the node_template name of the Tomcat Server. In this example that name was web_vm.

  ###################################################################################################################################################
  # This group defines the auto scaling and auto healing policies for the Apache Web server (web_vm)
  ###################################################################################################################################################
  groups:
    scaling_and_healing_group:
      members: [web_vm]                            # The node_template with the statement named "web_vm" is in this group
      properties:
        instances: 1                               # Launch 1 server when the stack is started
        minInstances: 1                            # The minimum number of servers
        maxInstances: 5                            # The maximum number of servers
        coolDown: 300                              # The number of seconds to wait before performing a auto scale or auto repair operation.

      policies:
        # This scale up policy will perform a scale up when the last 3 periods of idle time reported are < 20%
        scale_up_on_cpu:
          type: dcm.policy.types.BasicPolicy
          actions: [scale_up]                      # The scale_up action is defined in the actions: section of the template
          measurement: cpu_idle_time               # The cpu_idle_time label is defined in the measurements: section of the template
          criterion: less_than_20                  # The less_than_20 is a label defined in the criteria: section of the template

        # This scale down policy will perform a scale down when the last 2 periods of idle time reported are > 80%
        scale_down_on_cpu:
          type: dcm.policy.types.BasicPolicy
          actions: [scale_down]                    # The scale_down action is defined in the actions: section of the template
          measurement: cpu_idle_time               # The cpu_idle_time label is defined in the measurements: section of the template
          criterion: more_than_80                  # The more_than_80 is a label defined in the criteria: section of the template

      # Actions define the details of the various actions that could be taken.
      actions:
        scale_up:                                  # The scale_up action label is referenced in the policy in the actions[] statement
          type: dcm.policy.action.ScaleUpGroup
          properties:
            instances: 1                           # The number of resources to scale up on a scale_up action
            changeType: ADD                        # Add resource(s) on a scale_up action
            recordTask: RESOURCES_CHANGE           # Check to see if the scale up action would actually create a new resource, 
                                                   # or a scale-down operation would actually destroy a resource before creating a task to perform the action.

        scale_down:                                # The scale_down action label is referenced in the policy in the actions[] statement
          type: dcm.policy.action.ScaleDownGroup
          properties:
            instances: 1                           # The number of resources to scale down on a scale_down action 
            changeType: REMOVE                     # Remove resource(s) on a scale_down action
            recordTask: ALWAYS                     # Always attempt to perform a scale up or scale down action when the policy determines the actions should occur.
                                                   # Do not check the minInstances and maxInstances limits beforehand.

          type: dcm.policy.action.ReplaceResource                                           

      # Measurements define the "measurements" used to determine when to perform the actions
      measurements:
        cpu_idle_time:                             # The cpu_idle_time measurements label is referenced in the policy in a measurements: statement
        # The Dell Cloud Manager agent will collect and store the last 15 samples of cpu %idle time measured in 30-second intervals
          type: dcm.policy.measurement.CpuIdle
          properties:
            period: 30                             # Take a measurement every 30 seconds
            count: 15                              # Take 15 measurements

      # Criteria specify the "criteria" which is used along with the measurements to determine when to perform the actions
      criteria:
        less_than_20:                              # The less_than_20 criteria label is referenced in the policy in a criterion: statement
          type: dcm.policy.criteria.SeriesLessThan
          properties:
            count: 3                               # 3 consecutive periods 
            threshold: 20                          # Threshold is 20% 

        more_than_80:                              # The more_than_80 criteria label is referenced in the policy in a criterion: statement 
          type: dcm.policy.criteria.SeriesMoreThan
          properties: 
            count: 2                               # 2 consecutive periods
            threshold: 80                          # Threshold is 80%

Define Auto Healing Policy

Now lets add an auto healing policy to the template.

Add the following statements inside the policy: section and align them with the two scaling policies.

repair_on_status:
  type: dcm.policy.types.BasicPolicy
  measurement: cloud_reported_status
  criterion: check_fails
  actions: [repair]

Add the following statements inside the actions: section and align it with the two scaling actions.

repair:                                    # This auto healing action will terminate a degraded resource and then create a new resource
  type: dcm.policy.action.ReplaceResource

Add the following statements inside the measurements: section and align them with the two scaling measurements.

cloud_reported_status:
  type: dcm.policy.measurement.ResourceActive

resource_status:
  type: dcm.policy.measurement.ResourceStatus

Lastly add the following statements inside the criteria: section and align them with the two scaling criteria.

check_fails:                               # This is the criteria for auto healing
  type: dcm.policy.criteria.False          # "False" tells the policy to perform the action in the policy if the measurement result matches "False".