Ansible

Directory Layout

inventories
  dev
    eu-lab-central-1   # inventory file for lab-eu dev servers
      group_vars/
        all.yaml       # here we assign variables to all hosts
      spc-scheduler.yaml  # here we assign hosts and variables to particular groups
      hosts.yaml       # dynamically generated inventory file (do not edit)
  stg
    eu-lab-central-1   # inventory file for lab-eu stg servers
      group_vars/
  ppd
    eu-lab-central-1   # inventory file for lab-eu ppd servers
      group_vars/
  prd
    eu-central-1   # inventory file for lab-eu prd servers
      group_vars/

roles/
   common/            # this hierarchy represents a "role"
      tasks/        
        main.yaml      # tasks file can include smaller files if warranted
      handlers/      
        main.yaml      # handlers file
      templates/      # files for use with the template resource
        ntp.conf.j2   # templates end in .j2
      files/          
        bar.txt       # files for use with the copy resource
        foo.sh        # script files for use with the script resource
      vars/         
        main.yaml      # variables associated with this role
      defaults/      
        main.yaml      # default lower priority variables for this role
      meta/         
        main.yaml      # role dependencies

   computed/          
   ec2_metadata/
   iam/
   nbs/

site.yaml              # master playbook
computed.yaml          # playbook for computed tier
kraken.yaml            # playbook for kraken tier
infra.yaml             # playbook for infra tier
platform.yaml          # playbook for platform tier

• inventories
  • layout : ${env}/${region}/
  • group_vars
  • all.yaml : Assign global variables for all hosts
  • spc-scheduler.yaml

    Define Host groups for specific service (ex. spc-scheduler)

    Group variables for the service are in here

• roles
  • There are two places this can be done with variables defined in either the ‘vars’ or the ‘defaults’ directory of the role. (See Scoping variables for more information)
• playbook
  • site.yaml : Import a playbook that defines our entire infrastructure
  • computed.yaml/kraken.yaml/infra.yaml/platform.yaml : Map the configuration of the specific group to the roles
  • We dont use variables in the playbook itself

Variable precedence

Order of precedence

• If multiple variables of the same name are defined in different places, they get overwritten in a certain order.
• Here is the order of precedence from least to greatest (the last listed variables winning prioritization)

	role defaults
	inventory vars
	inventory group_vars
	inventory host_vars
	playbook group_vars
	playbook host_vars
	host facts
	play vars
	play vars_prompt
	play vars_files
	registered vars
	set_facts
	role and include vars
	block vars (only for tasks in block)
	task vars (only for the task)
	extra vars (always win precedence)

Scoping variables

• The variables used by ‘spc-anible’ are below.

  roles/*/defaults/main.yaml < group_vars/all < group_vars/* < host_vars/* < roles/*/vars/main.yaml
  

  1. roles/*/defaults/main.yaml
     • Variables that require deafult values within roles are roles/*/defaults/main.yaml
     • Since it is the most default in Ansible, it can be overriding by everything else.
  2. group_vars/all
     • Variables for all groups are stored in group_vars/all. Top-level variable
  3. group_vars/*
     • Variables corresponding to a specific group can be saved as group_vars/region.
     • Overriding variables of parent group including all
  4. host_vars/*
     • Variables corresponding to a specific host can be saved as host_vars/xxx.yyy.zzz.
     • Overriding the upper two
  5. roles/*/vars/main.yaml
     • Use roles/*/vars/main.yaml if you want to fix the variable value to be used within role
     • No overriding by external values! Except -e option
     • Do not put variables that will allow overriding.

• We don’t use variables in the playbook itself

Role dependencies

• Role dependencies allow you to automatically pull in other roles when using a role.
• Dependent roles are always executed before the roles that depend on them.
• Also, they are only executed once. If two roles state the same one as their dependency, it is only executed the first time. But when using allow_duplicates: true, it is executed the several time.

case 1) allow_duplicates: false

• roles/example1/meta/main.yaml

---
dependencies:
  - role: example_dep
    vars:
      some_parameter: 1

• roles/example2/meta/main.yaml ``` — dependencies:
  • role: example_dep vars: some_parameter: 2 ```
• roles/example_dep/meta/main.yaml --- allow_duplicates: false
• the execution order would be the following: example_dep -> example1 -> example2

case 2) allow_duplicates: true

• roles/example1/meta/main.yaml

---
dependencies:
  - role: example_dep
    vars:
      some_parameter: 1

• roles/example2/gmeta/main.yaml ``` — dependencies:
  • role: example_dep vars: some_parameter: 2 ```
• roles/example_dep/meta/main.yaml --- allow_duplicates: true
• the execution order would be the following: example_dep -> example1 -> example_dep -> example2

Execution strategy with tags

• A tag is an attribute that you can set to an ansible structure (plays, roles, tasks)

• When you apply tags attributes to structures other than tasks, ansible processes the tag attribute to apply only to the tasks they contain. Applying tags anywhere other than tasks is just a convenience so you don’t have to tag tasks individually.

• Role-based execution

  • To execute the whole task of the specific role, add tags to roles inside the play

• Task-based execution

  • To execute a specific task, add tags to tasks inside the roles/*/tasks/main.yaml

• Sample of playbook command

  • ansible-playbook ${playbook} -i ${inventory} -e ${extra_variables} -t ${tags}

  ansible-playbook platform.yaml -i inventories/dev/lab-eu-central-1 -e ci_version=0.0.1 -t example 
  ansible-playbook platform.yaml -i inventories/dev/lab-eu-central-1 -e ci_version=0.0.1 -t example_start
  ansible-playbook platform.yaml -i inventories/dev/lab-eu-central-1 -e ci_version=0.0.1 -t example_install,example_start 

Best Practices

Groups

Do define the group for your service based on the primitive groups

• Primitive groups are defined in hosts.yaml
  • for node type
  • infra, platform, compute
  • for node type by az:
  • form: ${node_type}_az_${az}
  • infra_az_a, infra_az_b, infra_az_c
  • platform_az_a, platform_az_b, platform_az_c
  • compute_az_a, compute_az_b, compute_az_c
  • for each node for infra/platform
  • form: ${node_type}_az_${az}_${index}
  • infra_az_a_1, infra_az_b_1, infra_az_b_1, …
  • platform_az_a_1, platform_az_b_1, platform_az_c_1, …
  • not defined for compute hosts

Don’t edit the primitive group file - hosts.yaml

• hosts.yaml file will be dynamically generated whenever the inventory is changed on the Serengeti Platform
• If you modify the primitive group, it will affect to all user and it will be overwritten in the next inventory generation

Variables

Don’t use host variables as much as possible

• Using host variables tend to make it hard for automation the provisioning process especially if the value can’t be evaluated automatically
• If you need to define some variables which need to be changed per each host, we strongly recommend to consider implementing the centralized configuration management mechanism instead of using host variables
• If it is unavoidable using host variables, you need to define them in the single file host_vars/{ip_address}.yaml for the visibility

Don’t define variables for the target host ip address

• Use domain or inventory_hostname instead of defining variables

Do define group variables and host groups together

• Inventory file for your group: inventories/${env}/${region}/${your_group}.yaml
• It is good for the visibility because we can find all information for the group and the group-scope variable in a single group file
• We recommend that we keep only all.yaml file in inventories/${env}/${region}/group_vars to maintain the variables for all groups

Don’t define any credentials in plain text

• Encrypt the credentials with ansible vault at least for any password

Playbook

Don’t write tasks in the playbook

• Playbooks should do nothing more than include a list of roles except where pre_tasks and post_tasks are required
  • Hide your task code under the roles to form clean, reusable abstractions
• Reuse the existing roles and playbooks for the larger-scoped playbook

Do provide the way of deploying service

• You should define the playbook as long as you provide a way of execution for the minimum deployment unit - we call it service in ci pipline
  • Separated playbook file for each service
  • Larger-scoped playbook file with tag-based execution control for each service deployment
  • Example
  • ansible-playbook -i inventories/prd/eu-central-1 your_service.yaml
  • ansible-playbook -i inventories/prd/eu-central-1 larger_playbook.yaml --tags your_service

Do use the directory inventory in general deployment

• ansible-playbook -i inventories/prd/eu-central-1 playbook.yaml
• It would be also possible to give the required inventory files with multiple -i options but you need to make it sure if all required inventory files are given as arguments

Naming conventions

Variables

• All variables should be snake_case
• Use jinja variable syntax over deprecated variable syntax {{ var }} not $var
• Use spaces around jinja variable names {{ var }} not {{var}}
• Prefix all variables defined in a role with the name of the role (example: myrole_foo)
• how-variables-are-merged

Roles and groups names

• Use underscores (e.g. my_role) not dashes (my-role)
• Roles and group names should be defined to avoid ambiguity or any potential conflicts
• We suggest that you consider using the proper prefix if there are any ambiguity or conflicts in role or group names
  • Example for prefix
  • product-based: ec2_, iam_, cloudwatch_, …
  • project-based: vm_, vpc_, nbs_, …
• It would prevent collisions among multiple role and group names and explicitly show which product/project a role/group belongs to

Tags

• We recommend that you use underscore in your tag names in order to avoid confusion