This tutorial will walk you through using fogg to create a new infrastructure repository.

Imagine that you are a company named Acme Corporation and want to deploy staging and production versions of your website where each one consists of a single server (let’s keep it simple).

Note that fogg works by generating Terraform and Make files. It does not run any Terraform commands for you.

1. create a working directory

   Create a new directory for this tutorial and cd into it.

    mkdir terraform_project
    cd terraform_project
   

2. setup git

    git init
   

   Fogg depends on working from the root of a git repository. The git repo doesn’t need to be pushed anywhere, so git init is enough. After this, your directory should look like this–

    $ tree -a -L 1
    .
    └── .git
   
   

3. initialize fogg

   Fogg uses a fogg.yml file in the root of your repository to define its structure. fogg init can help bootstrap this configuration file by asking some questions.

   $ fogg init
   fogg init
   project name?: acme
   aws region?: us-west-2
   infra bucket name?: acme-infra
   infra dynamo table name?: acme-dynamo
   auth profile?: acme-auth
   owner?: infra@acme.example
   

   A bit about those questions–

   • project name – we've got to name things around here. This is a high level name for your site, infrastructure or product.
   • aws region – our setup is super flexible to run things in any and/or multiple regions. To get started we need a single region that we will configure as a default. This is also the region for the s3 bucket that will hold state files, so maybe think about it a little bit.
   • infra bucket name - we are going to store terraform's state files here. Fogg does not create this bucket, so you will need to do that ahead of time. Note that it should be in the same region you said above.
   • infra dynamo table name - TODO
   • auth profile - we use aws authentication profiles, use this to specify the one to be used as a default. If you only have 1 profile set up, its probably called ‘default’.
   • owner – we make it easy to tag all your resources with their owner. If you put this here will will drop variables everywhere with the owner in it.

   And now your directory should look like this–

    $ tree -a -L 1
    .
    ├── .git
    └── fogg.yml
   
    1 directory, 1 file
   

   And if you look at fogg.yml–

    $ cat fogg.yml
    defaults:
        backend:
            bucket: acme-infra
            dynamodb_table: acme-dynamo
            profile: acme-auth
            region: us-west-2
        owner: infra@acme.example
        project: acme
        providers:
            aws:
                profile: acme-auth
                region: us-west-2
                version: 1.27.0
        terraform_version: 0.12.5
    version: 2
   

   Note that the questions you answered have all been filled into this file with some fogg-supplied defaults.

4. build initial repo

   As we said before fogg works by generating code (terraform, make and bash) and the general workflow is–

   1. edit fogg.yml
      • Add an account_id to the aws provider

       providers:
           aws:
               account_id: "000"
               profile: acme-auth
               region: us-west-2
               version: 1.27.0
       terraform_version: 0.12.5
      

   2. run fogg apply

   Apply is the command that actually writes out all the changes we’ve specified in fogg.yml.

   So now that we’ve written an initial fogg.yml let’s do an apply–

    $ fogg apply
    INFO README.md touched
    INFO scripts/dependencies.mk removed
    INFO scripts/module.mk copied
    INFO scripts/ssh_config.tmpl removed
    INFO scripts/update-readme.sh copied
    INFO Makefile templated
    INFO scripts/component.mk copied
    INFO scripts/failed_output_only copied
    INFO .fogg-version templated
    INFO .gitattributes copied
    INFO .gitignore copied
    INFO .terraform.d/plugin-cache/.gitignore copied
    INFO scripts/common.mk copied
    INFO scripts/install_or_update.sh removed
    INFO scripts/bless_ssh_config removed
    INFO terraform.d/plugins/linux_amd64/.gitignore copied
    INFO terraform/global/main.tf touched
    INFO terraform/global/outputs.tf touched
    INFO terraform/global/variables.tf touched
    INFO terraform/global/Makefile templated
    INFO terraform/global/README.md touched
    INFO terraform/global/fogg.tf templated
   

   You’ll see some output about what fogg is doing and now we have some structure to our repository–

    .
    ├── Makefile
    ├── README.md
    ├── fogg.yml
    ├── scripts
    │   ├── common.mk
    │   ├── component.mk
    │   ├── failed_output_only
    │   ├── module.mk
    │   └── update-readme.sh
    ├── terraform
    │   └── global
    │       ├── Makefile
    │       ├── README.md
    │       ├── fogg.tf
    │       ├── main.tf
    │       ├── outputs.tf
    │       ├── terraform.d -> ../../terraform.d
    │       └── variables.tf
    └── terraform.d
        └── plugins
            └── linux_amd64
   
    7 directories, 15 files
   

   Fogg has created 7 directories and put some files in them – scripts which exists to hold a handful of shell scripts useful to our operations and terraform which will include all our terraform code, both reusable modules and live infrastructure.

   Before we go on – a bit about how fogg organizes repos –

   Fogg applies an opinionated repo organization. This serves to make it easy to factor your terraform code into many scopes/state-files and also provide some consistency and make working on a team a bit easier.

   Fogg organizes terraform code into global, accounts, envs and components.

   • global - things are trying global across all your infrastructure. A good example is a Route53 zone, to which you want to add recrords from everywhere in your infra.
   • accounts - things that are relavant at the account level (aws here) - most, but not all aws iam stuff goes here. Note that we make it easy to have multiple accounts with configs for each in terraform/accounts/account-name.
   • envs - think staging vs prod here. fogg makes it easy to keep your tf separate for each one
   • components - in addition to separating environments we do one step further and make it easy to have multiple state files for each environment. In fogg we call those components. Each env can have many components and they all get their own statefile. On top of that each gets a terrafom_remote_state data source for all the other components in the same env.

   With that in mind, let’s create a new env.

5. configure a staging env

   Fogg helps you organize your terraform code and the resources they create into separate environments. Think ‘staging’ vs ‘production’. It is advisable to have them separate so that you can operate on them independently. Let’s create a ‘staging’ environment.

   To create a new env, edit your fogg.yaml to look like this–

    defaults:
        backend:
            bucket: acme-infra
            dynamodb_table: acme-dynamo
            profile: acme-auth
            region: us-west-2
        owner: infra@acme.example
        project: acme
        providers:
            aws:
                account_id: "000"
                profile: acme-auth
                region: us-west-2
                version: 1.27.0
        terraform_version: 0.12.5
    accounts: {}
    envs:
        staging: {}
    modules: {}
    version: 2
   

   Note that we added staging: {} in the envs object.

   fogg apply

    $ fogg apply
    INFO terraform.d/plugins/linux_amd64/.gitignore copied
    INFO .gitattributes copied
    INFO README.md skipped touch
    INFO scripts/module.mk copied
    INFO scripts/update-readme.sh copied
    INFO scripts/bless_ssh_config removed
    INFO scripts/dependencies.mk removed
    INFO scripts/install_or_update.sh removed
    INFO .fogg-version templated
    INFO .gitignore copied
    INFO scripts/common.mk copied
    INFO scripts/failed_output_only copied
    INFO .terraform.d/plugin-cache/.gitignore copied
    INFO Makefile templated
    INFO scripts/component.mk copied
    INFO scripts/ssh_config.tmpl removed
    INFO terraform/envs/staging/Makefile templated
    INFO terraform/envs/staging/README.md touched
    INFO terraform/global/fogg.tf templated
    INFO terraform/global/main.tf skipped touch
    INFO terraform/global/outputs.tf skipped touch
    INFO terraform/global/variables.tf skipped touch
    INFO terraform/global/Makefile templated
    INFO terraform/global/README.md skipped touch
   

   And your directory should look like:

    $ tree .
    .
    ├── Makefile
    ├── README.md
    ├── fogg.yml
    ├── scripts
    │   ├── common.mk
    │   ├── component.mk
    │   ├── failed_output_only
    │   ├── module.mk
    │   └── update-readme.sh
    ├── terraform
    │   ├── envs
    │   │   └── staging
    │   │       ├── Makefile
    │   │       └── README.md
    │   └── global
    │       ├── Makefile
    │       ├── README.md
    │       ├── fogg.tf
    │       ├── main.tf
    │       ├── outputs.tf
    │       ├── terraform.d -> ../../terraform.d
    │       └── variables.tf
    └── terraform.d
        └── plugins
            └── linux_amd64
   
    9 directories, 17 files
   

   A terraform/staging directory has been creaded with a few files in it, but nowhere to put terraform files yet– those go in components which are nested in envs. So let’s create a component–

6. create vpc component

   We need a VPC to run the resources we’re about to build. Creating a VPC is a great use-case for Terraform modules. Terraform modules are very useful, but can become tedious if you have to create the same ones repeatedly. Fogg helps with this by allowing you to specify a module source and then code-generating all the parameters and outputs. All that is left for you is to define some locals for the parameters.

   Edit your fogg.yaml like so–

    defaults:
        backend:
            bucket: acme-infra
            dynamodb_table: acme-dynamo
            profile: acme-auth
            region: us-west-2
        owner: infra@acme.example
        project: acme
        providers:
            aws:
                account_id: "000"
                profile: acme-auth
                region: us-west-2
                version: 1.27.0
        terraform_version: 0.12.5
    accounts: {}
    envs:
        staging:
            components:
                vpc:
                    module_source: "github.com/scholzj/terraform-aws-vpc"
    modules: {}
    version: 2
   

   This is telling fogg to create a new component called ‘vpc’, take the specified module source and code-generate an invocation of that module there.

   Run fogg apply and you’ll see some new files–

    $ tree .
    .
    ├── Makefile
    ├── README.md
    ├── fogg.yml
    ├── scripts
    │   ├── common.mk
    │   ├── component.mk
    │   ├── failed_output_only
    │   ├── module.mk
    │   └── update-readme.sh
    ├── terraform
    │   ├── envs
    │   │   └── staging
    │   │       ├── Makefile
    │   │       ├── README.md
    │   │       └── vpc
    │   │           ├── Makefile
    │   │           ├── README.md
    │   │           ├── fogg.tf
    │   │           ├── main.tf
    │   │           ├── outputs.tf
    │   │           ├── terraform.d -> ../../../../terraform.d
    │   │           └── variables.tf
    │   └── global
    │       ├── Makefile
    │       ├── README.md
    │       ├── fogg.tf
    │       ├── main.tf
    │       ├── outputs.tf
    │       ├── terraform.d -> ../../terraform.d
    │       └── variables.tf
    └── terraform.d
        └── plugins
            └── linux_amd64
   
    11 directories, 23 files
   

   If you look in terraform/envs/staging/vpc you’ll see that we’ve generated some terraform code that invokes the module we specified.

    $ cat terraform/envs/staging/vpc/main.tf
    # Auto-generated by fogg. Do not edit
    # Make improvements in fogg, so that everyone can benefit.
   
    module "terraform-aws-vpc" {
        source          = "github.com/scholzj/terraform-aws-vpc"
        aws_region      = "${local.aws_region}"
        aws_zones       = "${local.aws_zones}"
        private_subnets = "${local.private_subnets}"
        tags            = "${local.tags}"
        vpc_cidr        = "${local.vpc_cidr}"
        vpc_name        = "${local.vpc_name}"
    }
   

   Fogg has taken care of the drudgery of figuring out what variables and outputs this module supports. All you have left to do is define some locals to specify the inputs to the module. Create a locals.tf file in that same directory defining the values we need and we’re good to go.

7. create database component

   As we said at the beginning, our goal here is to set up a database and server in a VPC, so next let’s set up the database. Let’s edit the fogg.yml file to look like so–

    defaults:
        backend:
            bucket: acme-infra
            dynamodb_table: acme-dynamo
            profile: acme-auth
            region: us-west-2
        owner: infra@acme.example
        project: acme
        providers:
            aws:
                account_id: "000"
                profile: acme-auth
                region: us-west-2
                version: 1.27.0
        terraform_version: 0.12.5
    accounts: {}
    envs:
        staging:
            components:
                vpc:
                    module_source: "github.com/scholzj/terraform-aws-vpc"
                database: {}
    modules: {}
    version: 2
   

   Note that we’ve added a ‘database’ entry in the staging components. When we fogg apply we’ll now have a directory structure like so–

    $ tree .
    .
    ├── Makefile
    ├── README.md
    ├── fogg.yml
    ├── scripts
    │   ├── common.mk
    │   ├── component.mk
    │   ├── failed_output_only
    │   ├── module.mk
    │   └── update-readme.sh
    ├── terraform
    │   ├── envs
    │   │   └── staging
    │   │       ├── Makefile
    │   │       ├── README.md
    │   │       ├── database
    │   │       │   ├── Makefile
    │   │       │   ├── README.md
    │   │       │   ├── fogg.tf
    │   │       │   ├── main.tf
    │   │       │   ├── outputs.tf
    │   │       │   ├── terraform.d -> ../../../../terraform.d
    │   │       │   └── variables.tf
    │   │       └── vpc
    │   │           ├── Makefile
    │   │           ├── README.md
    │   │           ├── fogg.tf
    │   │           ├── main.tf
    │   │           ├── outputs.tf
    │   │           ├── terraform.d -> ../../../../terraform.d
    │   │           └── variables.tf
    │   └── global
    │       ├── Makefile
    │       ├── README.md
    │       ├── fogg.tf
    │       ├── main.tf
    │       ├── outputs.tf
    │       ├── terraform.d -> ../../terraform.d
    │       └── variables.tf
    └── terraform.d
        └── plugins
            └── linux_amd64
   
    13 directories, 29 files
   

   Note that since we didn’t specify a module_source here, the main.tf file in the database component is empty, Fogg is just creating the scaffolding, not any infrastructure for the database. You can then edit that main.tf file to create the infrastructure you want in that component.

8. create server component

   Now let’s do the same thing for a server component. Edit fogg.yaml like so–

    defaults:
        backend:
            bucket: acme-infra
            dynamodb_table: acme-dynamo
            profile: acme-auth
            region: us-west-2
        owner: infra@acme.example
        project: acme
        providers:
            aws:
                account_id: "000"
                profile: acme-auth
                region: us-west-2
                version: 1.27.0
        terraform_version: 0.12.5
    accounts: {}
    envs:
        staging:
            components:
                vpc:
                    module_source: "github.com/scholzj/terraform-aws-vpc"
                database: {}
                server: {}
    modules: {}
    version: 2
   

   And with fogg apply you’ll now see this–

    $ tree .
    .
    ├── Makefile
    ├── README.md
    ├── fogg.yml
    ├── scripts
    │   ├── common.mk
    │   ├── component.mk
    │   ├── failed_output_only
    │   ├── module.mk
    │   └── update-readme.sh
    ├── terraform
    │   ├── envs
    │   │   └── staging
    │   │       ├── Makefile
    │   │       ├── README.md
    │   │       ├── database
    │   │       │   ├── Makefile
    │   │       │   ├── README.md
    │   │       │   ├── fogg.tf
    │   │       │   ├── main.tf
    │   │       │   ├── outputs.tf
    │   │       │   ├── terraform.d -> ../../../../terraform.d
    │   │       │   └── variables.tf
    │   │       ├── server
    │   │       │   ├── Makefile
    │   │       │   ├── README.md
    │   │       │   ├── fogg.tf
    │   │       │   ├── main.tf
    │   │       │   ├── outputs.tf
    │   │       │   ├── terraform.d -> ../../../../terraform.d
    │   │       │   └── variables.tf
    │   │       └── vpc
    │   │           ├── Makefile
    │   │           ├── README.md
    │   │           ├── fogg.tf
    │   │           ├── main.tf
    │   │           ├── outputs.tf
    │   │           ├── terraform.d -> ../../../../terraform.d
    │   │           └── variables.tf
    │   └── global
    │       ├── Makefile
    │       ├── README.md
    │       ├── fogg.tf
    │       ├── main.tf
    │       ├── outputs.tf
    │       ├── terraform.d -> ../../terraform.d
    │       └── variables.tf
    └── terraform.d
        └── plugins
            └── linux_amd64
   
    15 directories, 35 files
   

   Now you have separate components for your VPC, database and server, in which you can create infrastructure which is managed by isolated Terraform state files.

9. TODO create prod
10. TODO refactor to module
11. TODO create module