dsillman

TF-GCP II: Uploading static content to GCP storage

2025-03-16T00:00:00+00:00

In the last article, we were able to successfully deploy a Google Cloud Storage (GCS) bucket using Terraform. It was a little bit anticlimactic, since the only real indication that we had done anything at all was the output of gsutil ls showing that the bucket was created. In this article, we’ll get closer to our goal of hosting a static site on GCS by getting our implementation to the point where there’s actually a URL we can visit and see some static site content.

First, we’ll need to decide what that static content which is served from GCS will be.

A basic “hello world” site, with Tailwind

For the purposes of this tutorial, we’re not doing anything particularly fancy. In short, we’re going to write a simple index.html file which, when viewed in the browser, looks half-decent. One of the ways we can do that in a low-effort way is by using Tailwind CSS, a utility-first CSS framework that allows us to style our site entirely within HTML without needing to bother with CSS files.

This article isn’t meant to go into depth about how Tailwind works, but I’ll at least give a basic explanation of the utility classes which we’re using for our “hello world” example to make it look nice.

First, I’ll create a content/ directory in my project folder which I’ll use to store the static content for my site. Inside that directory, I’ll create an index.html file with the following content:

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Hello, world!</title>
    <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.0.2/dist/tailwind.min.css" rel="stylesheet">
</head>
<!-- Body goes here -->

</html>

This is a super basic skeleton of an empty HTML page. The only thing we’ve set up is a link to the Tailwind CSS CDN, which will allow us to use Tailwind utility classes to style our site within the document. Additionally, I’ve added a <title> tag with the text “Hello, world!” which will be displayed in the browser tab.

For the body, I let Copilot take the wheel and generate a really basic centered “Hello World” message:

<body class="bg-gray-100 h-screen flex items-center justify-center">
    <h1 class="text-4xl text-gray-800">Hello World</h1>
</body>

You’ll note the use of a bunch of non-sensical tokens like bg-gray-100, items-center and text-4xl. These are the Tailwind utility classes I mentioned earlier. They’re a way of applying styles to elements in a way that’s more readable and maintainable than writing CSS. For example, bg-gray-100 sets the background color of the element to a light gray, items-center centers the content vertically, and text-4xl sets the text size to “quadruple extra large.”

When I open this HTML document from my local filesystem in a browser, I see a centered “Hello World” message in a nice, large font:

Looks like it will do the trick. For good measure, I’ll make a copy of this file called content/404.html, replacing the “Hello World” message(s) with “404,” to act as a fallback page in case the user tries to access a page that doesn’t exist:

...
<body class="bg-gray-100 h-screen flex items-center justify-center">
    <h1 class="text-4xl text-gray-800">404</h1>
</body>
...

Now, let’s get this content uploaded to our GCS bucket - but via Terraform, of course.

GCS bucket object resources

I’m going to create a new dedicated Terraform file for managing the content of my GCS bucket. I’ll called it site_content.tf. With the google provider, objects in cloud storage themselves can be provisioned as resources in Terraform. So, in this new site_content.tf file, I’ll define two google_storage_bucket_object resources, one for each of the HTML files I created earlier:

resource "google_storage_bucket_object" "index_html" {
  bucket       = google_storage_bucket.static_site_bucket.name
  name         = "index.html"
  source       = "content/index.html"
  content_type = "text/html"
}

resource "google_storage_bucket_object" "error_html" {
  bucket       = google_storage_bucket.static_site_bucket.name
  name         = "404.html"
  source       = "content/404.html"
  content_type = "text/html"
}

Ok, these resources look a little different from the ones we’ve been declaring in the previous article. You’ll note that the bucket configuration is not being set to an explicit value. Rather, it’s being set according to the name attribute of the google_storage_bucket resource we created in the last article, called static_site_bucket. So, the syntax for referencing attributes of other resources in Terraform is:

# <resource_type>.<resource_name>.<attribute_name>
google_storage_bucket.static_site_bucket.name

You may ask, “why not just set the bucket attribute to the name of the bucket directly?” The answer is that this way, we can ensure that the google_storage_bucket_object resources are created in the same bucket as the google_storage_bucket resource, even if the name of the bucket changes. Moreover, Terraform also takes stock of referential dependencies between resources like this one, which helps it determine the correct order in which to create resources, and which resources can be provisioned in parallel. So, implicitly, we are telling Terraform that these new google_storage_bucket_object resources depend on the google_storage_bucket resource we created earlier, so they’ll be created in sequence.

The name and content_type arguments tell GCP what to name the object in the bucket and what type of content it is, respectively. The source argument is the path to the file on the local filesystem which we want to upload to the bucket. In this case, it’s the index.html and 404.html files in the content/ directory.

We’re finally ready to validate the execution plan with Terraform’s plan command.

terraform plan output

$ terraform plan

  Terraform used the selected providers to generate the   following execution plan. Resource actions are indicated   with the following symbols:
    + create
  
  Terraform will perform the following actions:
  
    # google_storage_bucket.static_site_bucket will be created
    + resource "google_storage_bucket" "static_site_bucket" {
        + effective_labels            = {
            + "goog-terraform-provisioned" = "true"
          }
        + force_destroy               = true
        + id                          = (known after apply)
        + location                    = "US"
        + name                        = (known after apply)
        + project                     = (known after apply)
        + project_number              = (known after apply)
        + public_access_prevention    = (known after apply)
        + rpo                         = (known after apply)
        + self_link                   = (known after apply)
        + storage_class               = "STANDARD"
        + terraform_labels            = {
            + "goog-terraform-provisioned" = "true"
          }
        + uniform_bucket_level_access = (known after apply)
        + url                         = (known after apply)
  
        + soft_delete_policy (known after apply)
  
        + versioning (known after apply)
  
        + website {
            + main_page_suffix = "index.html"
            + not_found_page   = "404.html"
          }
      }
  
    # google_storage_bucket_object.error_html will be created
    + resource "google_storage_bucket_object" "error_html" {
        + bucket         = (known after apply)
        + content        = (sensitive value)
        + content_type   = "text/html"
        + crc32c         = (known after apply)
        + detect_md5hash = "different hash"
        + generation     = (known after apply)
        + id             = (known after apply)
        + kms_key_name   = (known after apply)
        + md5hash        = (known after apply)
        + media_link     = (known after apply)
        + name           = "404.html"
        + output_name    = (known after apply)
        + self_link      = (known after apply)
        + source         = "content/404.html"
        + storage_class  = (known after apply)
      }
  
    # google_storage_bucket_object.index_html will be created
    + resource "google_storage_bucket_object" "index_html" {
        + bucket         = (known after apply)
        + content        = (sensitive value)
        + content_type   = "text/html"
        + crc32c         = (known after apply)
        + detect_md5hash = "different hash"
        + generation     = (known after apply)
        + id             = (known after apply)
        + kms_key_name   = (known after apply)
        + md5hash        = (known after apply)
        + media_link     = (known after apply)
        + name           = "index.html"
        + output_name    = (known after apply)
        + self_link      = (known after apply)
        + source         = "content/index.html"
        + storage_class  = (known after apply)
      }
  
    # random_uuid.bucket_uuid will be created
    + resource "random_uuid" "bucket_uuid" {
        + id     = (known after apply)
        + result = (known after apply)
      }
  
  Plan: 4 to add, 0 to change, 0 to destroy.

Recall that, at the end of the last article, I destroyed everything in my GCP project. So when I ran terraform plan, Terraform correctly queried the remote GCP state to determine that all of my resources need to be re-created. This is a good sign that our Terraform configuration is working as expected. We also see our expected google_storage_bucket_object resources being created, along with the website block in the google_storage_bucket resource. Let’s apply the changes.

terraform apply output

$ terraform apply
  random_uuid.bucket_uuid: Creating...
  random_uuid.bucket_uuid: Creation complete after 0s   [id=be028675-b3cd-8fbd-c48a-3ddb6629ca7b]
  google_storage_bucket.static_site_bucket: Creating...
  google_storage_bucket.static_site_bucket: Creation complete   after 2s   [id=be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket]
  google_storage_bucket_object.error_html: Creating...
  google_storage_bucket_object.index_html: Creating...
  google_storage_bucket_object.error_html: Creation complete   after 1s   [id=be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket-404.  html]
  google_storage_bucket_object.index_html: Creation complete   after 1s   [id=be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket-index.  html]
  
  Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

I can once again confirm that the bucket is created with gsutil ls:

$ gsutil ls
gs://be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket/

Specifically, I can also see the objects in the bucket:

$ gsutil ls gs://be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket/*
gs://be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket/404.html
gs://be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket/index.html

It correctly identifies that we’ve uploaded the index.html and 404.html files to the bucket. Now, let’s see if we can access the site content via the bucket URL.

Viewing the site content

To view the site content, I’ll navigate to the URL of the bucket in my browser. The URL is in the format https://<bucket-name>.storage.googleapis.com. In this case, the bucket name is be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket, so the URL is https://be028675-b3cd-8fbd-c48a-3ddb6629ca7b-site-bucket.storage.googleapis.com/index.html.

Note: around this point in the exercise, I decided to destroy and re-apply my terraform project, which regenerated my UUID and bucket name. This may continue to happen insofar as randomness is involved in the generation of the bucket name. If you’re following along, please overlook any discrepancies in the bucket name within and between articles.

When I navigate to this URL, I get an error message:

<Error>
    <Code>AccessDenied</Code>
    <Message>Access denied.</Message>
    <Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).</Details>
</Error>

This indicates that, while we’ve successfully uploaded the content to the bucket, the bucket is not publicly accessible, and so our URL is returning an access denied error. We’ll need to update the bucket’s permissions to allow public access to the objects within it.

Making the bucket public

To make the bucket public, we’ll need to update the bucket’s IAM policy to allow all users to view the objects within it. We can do this by adding a new google_storage_bucket_iam_binding resource to our Terraform configuration. I’ll add this to our existing bucket.tf file:

resource "google_storage_bucket_iam_binding" "public_read" {
  bucket = google_storage_bucket.static_site_bucket.name
  role   = "roles/storage.objectViewer"

  members = [
    "allUsers"
  ]
}

Note again that we are using the resource attribute reference syntax to set the bucket attribute to the name of the bucket we created earlier. This ensures that the IAM binding is created for the same bucket as the objects we uploaded.

The role attribute is set to roles/storage.objectViewer, which is a predefined IAM role that grants permission to view objects in a bucket. The members attribute is set to allUsers, which is a special identifier that represents all users, including anonymous users. This means that any user who visits the bucket URL will be able to view the objects within the bucket.

We can check what the output of the terraform plan command looks like in order to confirm that the IAM binding will be created as expected for our bucket.

terraform plan output

$ terraform plan
  random_uuid.bucket_uuid: Refreshing state...   [id=edb54251-622d-bbec-5005-03ddfd1117fa]
  google_storage_bucket.static_site_bucket: Refreshing state...   [id=edb54251-622d-bbec-5005-03ddfd1117fa-site-bucket]
  google_storage_bucket_object.index_html: Refreshing state...   [id=edb54251-622d-bbec-5005-03ddfd1117fa-site-bucket-index.  html]
  google_storage_bucket_object.error_html: Refreshing state...   [id=edb54251-622d-bbec-5005-03ddfd1117fa-site-bucket-404.html]
  
  Terraform used the selected providers to generate the   following execution plan. Resource actions are indicated with   the following symbols:
    + create
  
  Terraform will perform the following actions:
  
    # google_storage_bucket_iam_binding.public_read will be   created
    + resource "google_storage_bucket_iam_binding" "public_read"   {
        + bucket  =   "edb54251-622d-bbec-5005-03ddfd1117fa-site-bucket"
        + etag    = (known after apply)
        + id      = (known after apply)
        + members = [
            + "allUsers",
          ]
        + role    = "roles/storage.objectViewer"
      }
  
  Plan: 1 to add, 0 to change, 0 to destroy.

Looks good to me. Running terraform apply, we see that the policy binding gets successfully created.

$ terraform apply
  google_storage_bucket_iam_binding.public_read: Creating...
  google_storage_bucket_iam_binding.public_read: Creation complete after 5s [id=b/edb54251-622d-bbec-5005-03ddfd1117fa-site-bucket/roles/storage.objectViewer]

Now, when I navigate to the bucket URL in my browser, I see the contents of the index.html file displayed:

Success! We’ve successfully uploaded static content to a GCS bucket and made it publicly accessible. I didn’t promise that the URL used to access the website would necessarily be a nice one - so our goal was accomplished, as stated! In the next article, we’ll take a moment to look back over the terraform we’ve written so far and see if there are any improvements we can make.

Note: Once again, since I’m taking a break from this project, I’ll clean up after myself to save myself any risk of incurring cloud costs while I’m not using the resources. As with the last article, it’s easy to destroy the bucket and IAM binding with terraform destroy.

TF-GCP I: Provisioning GCP resources with Terraform

2025-03-09T00:00:00+00:00

I was recently challenged at work to come up with a long-term personal development goal for the year. I suck at doing that stuff. But as our team is moving increasingly towards cloud computing, I do know that eventually I will hit a ceiling if I don’t learn something about building complex systems in the Cloud. This article is my first stab at that on my own dime, and with my own time.

My team at work is using Amazon (AWS) as the cloud platform for all of our services. This is probably the most economical choice for most purposes, but why take the path most traveled? I wanted to try something new and learn about some of the differences between AWS and Google’s cloud platform, GCP (Google Cloud Platform), so I decided I’d figure out how to use Terraform in that context.

One of the most basic things you can do in the Cloud is set up a publicly-accessible static website by using cloud storage. Basically, you’re just uploading static files to a cloud server and configuring it to serve those files to the public using a managed URL. This is a pretty straightforward way to host all sorts of things from blogs, resumes or project documentation.

How they tell you to do it, vs. how you should do it

If you follow any tutorials on GCP (or any other cloud platform), they’ll explain how to do this using the Google Cloud Console (or AWS cloud console for AWS), which is just a UI interface for managing cloud resources. This is great for learning how things work, but it’s not a great way to manage your infrastructure in the long term. Ideally, your infrastructure specs would be layed out in code and tracked in version control, so that you can see how your infrastructure has changed over time and roll back changes if necessary.

This is where Terraform comes in. Terraform is a DSL (domain-specific language) for defining the cloud resources managed by a project. The cloud resources could straddle many different cloud providers, but in this case we’re just using GCP. Learning Terraform is generally a good way to learn about how different cloud resources can interact with one another to build a larger, multi-component system.

Thankfully, a static site doesn’t require more than one component - our provisioned cloud storage. So this project shouldn’t be too complicated…

…right?

Prerequisites

There are a few things that took me a little bit to realize I needed installed as I was getting started:

Terraform CLI: AKA the terraform CLI. If you’re using Mac OSX, you can install this with Homebrew. But installation for all other platforms is pretty straightforward. You can find the installation instructions on their website.

If you successfully managed to install Terraform, you should be able to run the following command in your terminal:
```
 terraform --version
```
Google Cloud CLI: AKA the gcloud CLI. On Mac OSX, you’re required to download the corresponding .dmg file from the Google Cloud SDK website and follow the given instructions (basically just run a bash script).

If you successfully managed to install the Google Cloud CLI, you should be able to run the following command in your terminal:
```
 gcloud --version
```

Once these two are installed, you’re more or less ready to get started. There are just a few other things which we’d want to configure about our Google Cloud setup before we start provisioning resources.

Setting up Google Cloud

Once gcloud is installed, you’ll want to authenticate your Google account with the CLI. This is done by running the following command:

gcloud auth login

This will take you through an interactive set of web pages to authenticate your Google account with the CLI. Once you’ve authenticated, you’ll be able to run commands against your Google Cloud account.

In Google Cloud, there’s a hierarchical structure to how resources are organized. At the top level, you (optionally) have an Organization, which may contain multiple Projects. Each Project independently manages its permissions and resources. By default, consumer GCP accounts do not have an Organization configured, so you’ll create projects in the root of your account (no Organization containing them).

To get started, we’ll create a Project for the static site we want to host on GCP. I’ll call my project davids-static-site.

gcloud projects create davids-static-site

After creating the project, wait about 30 - 60 seconds for the project to fully materialize in Google Cloud. You can check the status of your project by running the following command:

gcloud projects describe davids-static-site

Once that project is created, meaning its status is “ACTIVE” in the output of the describe command, you can set it as the default project for the gcloud CLI:

gcloud config set project davids-static-site

The last step in ensuring our account and project are properly set up is to enable billing for the project. This is required to provision any resources in GCP. If you don’t currently have a billing account associated with your Google Cloud account, it’s easiest to do that from the Google Cloud console web interface.

You didn’t think you were actually going to do anything in the Cloud without paying for it, did you?

After you have a billing account and confirmed a form of payment, you can enable billing for the project by running the following command:

# Check your available billing account IDs
gcloud billing accounts list
# Set one to use for the new project
gcloud billing projects link davids-static-site --billing-account=<billing-account-id>

Now that we have a project and billing set up for our account, we can start provisioning resources in GCP. We’re finally ready to start tinkering with Terraform!

Setting up our Terraform project

Alright, first things first. I’m going to create a fresh directory for the project where I like to organize these things, which is in my $HOME/terraform-projects directory. I’ll call this project gcp-static-site.

mkdir -p $HOME/terraform-projects/gcp-static-site
cd $HOME/terraform-projects/gcp-static-site

Note: The way that I’m choosing to split my code across files is more or less arbitrary. I’m following some conventions I’ve learned from co-workers and from reading the Terraform documentation. I’m sure there are many ways to organize your Terraform code, but this is just one way that I think is pretty clean.

The first thing I do to set up my project is to create a versions.tf file. The only thing I’ll put in this file is the terraform block for our project. This block specifies the version of Terraform that we’re using, and it’s a good idea to specify this in every Terraform project you create. I also use it to specify the required version of the Google Cloud provider plugin.

terraform {
  required_version = ">= 1.0.0"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "6.24.0"
    }
  }
}

Now that we have this config block in our directory, we can just run terraform init to instruct Terraform to read it and download the required provider plugin for Google Cloud.

terraform init output

$ terraform init
  Initializing the backend...
  Initializing provider plugins...
  - Finding hashicorp/google versions matching "6.24.0"...
  - Installing hashicorp/google v6.24.0...
  - Installed hashicorp/google v6.24.0 (signed by HashiCorp)
  Terraform has created a lock file .terraform.lock.hcl to record the 
  provider selections it made above. Include this file in your version control 
  repository so that Terraform can guarantee to make the same selections by default 
  when you run "terraform init" in the future.
  
  Terraform has been successfully initialized!
  
  You may now begin working with Terraform. Try running "terraform plan" to 
  see any changes that are required for your infrastructure. All Terraform 
  commands should now work.
  
  If you ever set or change modules or backend configuration for Terraform,
  rerun this command to reinitialize your working directory. If you forget, 
  other commands will detect it and remind you to do so if necessary.

This should have done a few things. First, you should see a .terraform directory in the project, which basically saves the downloaded provider plugins and other state information. Second, you should see a .terraform.lock.hcl file, which is a lock file that specifies the exact versions of the provider plugins that were downloaded. I think of this as analogous to poetry.lock, or event requirements.txt in Python projects.

Let’s provision a GCS bucket

Ok, great. So our Terraform project is initialized. How can we get it to provision a Google Cloud Storage (GCS) bucket for us?

A GCS bucket is essentially just a shared storage location, like a mini file system which can be accessed via API requests, much like Amazon’s equivalent Simple Storage Service (S3).

Since our project is essentially just using this cloud storage to host some static site pages, we’ll obviously want the ownership of this storage bucket to fall within our project. So, let’s create a dedicated Terraform file for provisioning the GCS bucket, called bucket.tf, with a resource block defined in it:

resource "google_storage_bucket" "static_site_bucket" {
  name          = "site-bucket"
  location      = "US"
  force_destroy = true
}

The syntax of a resource block is:

resource "<resource type>" "<resource name>" {
    # resource configuration
}

So, above, we’re declaring a new google_storage_bucket resource in the project, and we’re calling it static_site_bucket. There are a minimum of two required arguments for this resource type: name and location. The name is the name of the bucket, and the location is the region where the bucket will be created. The force_destroy argument is optional, but it’s set to true here to allow Terraform to delete the bucket even if it’s not empty.

Note: I’m getting an idea of what the possible arguments are for this resource type by looking at the Google Cloud provider documentation. The Provider docs are very valuable resources for understanding what you can do with a given resource type. It’s basically the only way to use Terraform effectively.

Now that we actually have a resource in our project, we can meaningfully run terraform plan to get a preview of what Terraform thinks it will need to provision to deploy our project.

terraform plan output (error!)

$ terraform plan

  Planning failed. Terraform encountered an error while generating this   plan.
  
  ╷
  │ Error: Invalid provider configuration
  │ 
  │ Provider "registry.terraform.io/hashicorp/google" requires explicit 
  configuration. Add a provider block to the root module and configure the 
  provider's required arguments as described in the
  │ provider documentation.
  │ 
  ╵
  ╷
  │ Error: Attempted to load application default credentials since neither 
  `credentials` nor `access_token` was set in the provider block.  No 
  credentials loaded. To use your gcloud credentials, run 'gcloud auth 
  application-default login'. Original error: google: could not find 
  default credentials. See https://cloud.google.com/docs/authentication/
  external/set-up-adc for more information
  │ 
  │   with provider["registry.terraform.io/hashicorp/google"],
  │   on <empty> line 0:
  │   (source code not available)
  │ 

Ok, so two issues. The first one (“Invalid provider configuration”) is because we haven’t actually configured the Google Cloud provider in our project. Whoops! That’s an easy fix we’ll get to shortly.

The second one (“Attempted to load application default credentials”) is because Terraform doesn’t know how to authenticate with Google Cloud. We went through a manual process to authenticate with gcloud earlier, but Terraform doesn’t know about that. We need to tell Terraform to use the same credentials that gcloud is using. Thankfully, they give us a really easy command to just run to set up Terraform:

gcloud auth application-default login

This command will cache some authenticated credentials on your local filesystem which Terraform will pick up for use in authenticating with Google Cloud. This is a one-time setup, so you shouldn’t need to run this command again unless you change your Google Cloud credentials.

Now let’s create that provider configuration in our Terraform project. I like putting these in a dedicated providers.tf file, with one or more provider blocks defined in it:

provider "google" {
  project = "davids-static-site"
  region  = "us-west2"
}

This is all that’s strictly needed for Terraform to be able to use the Google Cloud provider. The project argument is the name of the Google Cloud project we created earlier, and the region argument is the default region that resources will be created in. This is a good default to set, but it can be overridden on a per-resource basis if necessary.

Now that we have this provider configuration, we can run terraform plan again to see what Terraform thinks it needs to do to deploy our project.

terraform plan output

$ terraform plan

  Terraform used the selected providers to generate the following execution 
  plan. Resource actions are indicated with the following symbols:
    + create
  
  Terraform will perform the following actions:
  
    # google_storage_bucket.static_site_bucket will be created
    + resource "google_storage_bucket" "static_site_bucket" {
        + effective_labels            = {
            + "goog-terraform-provisioned" = "true"
          }
        + force_destroy               = true
        + id                          = (known after apply)
        + location                    = "US"
        + name                        = "site-bucket"
        + project                     = (known after apply)
        + project_number              = (known after apply)
        + public_access_prevention    = (known after apply)
        + rpo                         = (known after apply)
        + self_link                   = (known after apply)
        + storage_class               = "STANDARD"
        + terraform_labels            = {
            + "goog-terraform-provisioned" = "true"
          }
        + uniform_bucket_level_access = (known after apply)
        + url                         = (known after apply)
  
        + soft_delete_policy (known after apply)
  
        + versioning (known after apply)
  
        + website (known after apply)
      }
  
  Plan: 1 to add, 0 to change, 0 to destroy.

Lovely! It correctly sees that, according to our project specifications, running terraform apply ought to create a GCS storage bucket. It seems to also pick up the configurations we set in the bucket.tf file, like the name and location of the bucket.

Do we dare try running terraform apply? This CLI command actually executes the stated plan, submitting the necessary API requests to Google Cloud to create the resources we’ve specified (in this case, just a GCS bucket).

terraform apply output (error!)

$ terraform apply

  ...
  google_storage_bucket.static_site_bucket: Creating...
  ╷
  │ Error: googleapi: Error 409: The requested bucket name is not 
  available. The bucket namespace is shared by all users of the system. 
  Please select a different name and try again., conflict
  │ 
  │   with google_storage_bucket.static_site_bucket,
  │   on bucket.tf line 1, in resource "google_storage_bucket"   "static_site_bucket":
  │    1: resource "google_storage_bucket" "static_site_bucket" {
  │ 
  ╵

Well that wasn’t expected. Let’s take a moment to discuss the way Google Cloud Storage buckets work.

The issue with GCS buckets

This is frankly the most outrageous design decision I’ve seen on the part of the GCP team, especially as someone familiar with using AWS where you can choose your bucket names pretty freely.

In GCP, the bucket namespace is global - if somebody else has already named a GCS bucket site-bucket (the name we were trying to use), then we can’t use that name. That means that everyone has to come up with a globally unique name for their buckets, which is a bit of a pain point. Some users of GCP suggest using a UUID or other unique identifier in the bucket name to ensure uniqueness, then using prefixing or suffixing to add a human readable component to the name.

Is there a way to incorporate some sort of randomness / UUID generation into our Terraform project to ensure that we can always create a bucket with a unique name? With one quick Google search, we will learn that is possible by…

Utilizing Terraform’s `random` provider

Just like there is a google Provider responsible for managing Google Cloud resources, Providers are a more general concept in Terraform. Some providers may do things locally on your system, or talk to non-cloud services. The random provider is one such provider, which can generate random values for use in your Terraform project. It treats a small program which can be used to repeatably generate random values as a resource which is provisioned and maintained by Terraform, even though it’s all actually happening locally on your machine.

Because it’s just a provider like google, we’ll treat it similarly and add entries to versions.tf and providers.tf to configure it:

# versions.tf
terraform {
  ...
  required_providers {
    ...
    random = {
      source  = "hashicorp/random"
      version = "3.7.1"
    }
  }
}

# providers.tf
provider "random" {
  # No configuration needed for the random provider
}

Now that we have the random provider configured, we can use it to generate a random UUID for the bucket name in our bucket.tf file. I’ll still opt to suffix the random string with “site-bucket” so it’s somewhat identifiable within our project.

# bucket.tf
resource "random_uuid" "bucket_uuid" {}

resource "google_storage_bucket" "static_site_bucket" {
  name          = "${random_uuid.bucket_uuid.result}-site-bucket"
  location      = "US"
  force_destroy = true
}

Syntax note: It’s ubiquitous in Terraform to use string-formatting syntax ${} to interpolate values into strings. This is how we’re able to use the random_uuid.bucket_uuid.result value in the name attribute of the google_storage_bucket resource.

Since we just added a new provider to the project, we’ll need to run terraform init again to download the random provider plugin into our project’s local .terraform cache. After running that, we’ll be able to check the impact on our Terraform plan by running terraform plan again:

terraform plan output

$ terraform plan
  
  Terraform used the selected providers to generate the following execution   plan. Resource actions are indicated with the following symbols:
    + create
  
  Terraform will perform the following actions:
  
    # google_storage_bucket.static_site_bucket will be created
    + resource "google_storage_bucket" "static_site_bucket" {
        + effective_labels            = {
            + "goog-terraform-provisioned" = "true"
          }
        + force_destroy               = true
        + id                          = (known after apply)
        + location                    = "US"
        + name                        = (known after apply)
        + project                     = (known after apply)
        + project_number              = (known after apply)
        + public_access_prevention    = (known after apply)
        + rpo                         = (known after apply)
        + self_link                   = (known after apply)
        + storage_class               = "STANDARD"
        + terraform_labels            = {
            + "goog-terraform-provisioned" = "true"
          }
        + uniform_bucket_level_access = (known after apply)
        + url                         = (known after apply)
  
        + soft_delete_policy (known after apply)
  
        + versioning (known after apply)
  
        + website (known after apply)
      }
  
    # random_uuid.bucket_uuid will be created
    + resource "random_uuid" "bucket_uuid" {
        + id     = (known after apply)
        + result = (known after apply)
      }
  
  Plan: 2 to add, 0 to change, 0 to destroy.

Note that Terraform identifies that we addeed a random_uuid resource to the project, which will be created as a part of the project. This explains why the google_storage_bucket resource now has a name attribute that is (known after apply) - it will be set to the result of the random_uuid resource after it’s created (with our suffix appended).

Fair warning: from here on out, you will officially start being billed for the resources you’re provisioning in GCP. Up until this point we’ve been unsuccessful in creating any resources, so no costs were incurred. Starting with the following step, we will be creating resources in GCP, so you will start to incur costs.

Now we can give terraform apply a shot!

terraform apply output

$ terraform apply

  ...
  random_uuid.bucket_uuid: Creating...
  random_uuid.bucket_uuid: Creation complete after 0s   [id=5c01e5eb-1bed-5c2e-9a89-1570b1b405de]
  google_storage_bucket.static_site_bucket: Creating...
  google_storage_bucket.static_site_bucket: Creation complete after 2s [id=5c01e5eb-1bed-5c2e-9a89-1570b1b405de-site-bucket]

Done! We’ve successfully created a GCS bucket in GCP using Terraform. We can verify this by checking the Google Cloud Console, or by running gsutil commands to interact with the bucket. Specifically, if you run gsutil ls, it will list the buckets in your project:

$ gsutil ls
  gs://5c01e5eb-1bed-5c2e-9a89-1570b1b405de-site-bucket/

And there it is! Our bucket is listed in the output of the gsutil ls command. We haven’t quite made it to the point where there’s a static site we can visit and see content. But let’s return back to this for another article later on to discuss how to actually upload content to the bucket and configure it to serve a static site publicly.

Closing note

In the mean-time, we’ve still provisioned a GCS bucket in GCP. Since I’m not planning on using it for anything between now and the next article, I’ll go ahead clean up after myself with the simple Terraform command, terraform destroy. This is one of the nicest things about Terraform in my opinion - everything you can provision, you can also destroy using a single, unified interface.

terraform destroy output

$ terraform destroy
  random_uuid.bucket_uuid: Refreshing state... [id=5c01e5eb-1bed-5c2e-9a89-1570b1b405de]
  google_storage_bucket.static_site_bucket: Refreshing state... [id=5c01e5eb-1bed-5c2e-9a89-1570b1b405de-site-bucket]
  ...
  google_storage_bucket.static_site_bucket: Destroying... [id=5c01e5eb-1bed-5c2e-9a89-1570b1b405de-site-bucket]
  google_storage_bucket.static_site_bucket: Destruction complete after 1s
  random_uuid.bucket_uuid: Destroying... [id=5c01e5eb-1bed-5c2e-9a89-1570b1b405de]
  random_uuid.bucket_uuid: Destruction complete after 0s

I can confirm that it’s cleaned up by noting the empty result when I try gsutil ls again. Nice - now I can rest easy knowing that I’m not being billed for any resources in GCP. I can always come back to this project and run terraform apply to re-provision the bucket if I need to.

Modular YAML released: `yaml-extras` on PyPI

2024-12-30T00:00:00+00:00

I’m proud to announce that I’ve made my first official contribution to the Python Package Index (PyPI) with the release of a package called yaml-extras. This package is the culmination of what I’ve discussed in the previous three blog posts dedicated to loading the contents of YAML files into one another in a modular way. See the other three articles linked below:

The package is available on PyPI here. The source code is available and open source on Github, plus an extended documentation website:

Because the package is available on PyPI, you can install it with your package manager of choice:

# Install with pip
pip install yaml-extras

# Install with poetry
poetry add yaml-extras

If you read this article and are inspired to make something with yaml-extras, I’d love to hear about it! (Shoot me an email at dsillman2000@gmail.com)

Since the final article in the series, I’ve made significant refactors to some of the content I shared which I will briefly summarize here.

`!import.anchor` and `!import-all.anchor`

In the final article, I alluded to the possibility of a follow-up article that discussed “targeted” imports, where you could specify a particular anchor in an external YAML file which you wanted to import. I’ve since implemented this feature in the package, and it’s available for use, but the implementation was somewhat nuanced.

The challenge of implementing an “anchor-fetching” feature as just another Constructor alongside my other custom constructors was that, by the time a yaml.Node reaches the construction phase of loading, all anchors and aliases have been resolved, so anchors are nowhere to be seen. To identify subtrees of the AST which are marked with a particular anchor, I needed to intercept the AST before it reached construction, meaning I needed to use the PyYAML Parser directly.

As a refresher, the YAML Parser is responsible for taking the raw YAML Tokens which are scanned from a file and producing YAML Events. The events are subsequently parsed into an AST of YAML Nodes which finally get constructed into Python objects. The last medium in this process which has any notion of anchors are the YAML Events. So, in abbreviated pseudo-python code, here’s how I implemented a helper function called load_yaml_anchor(file_stream, anchor, loader_type):

def load_yaml_anchor(
    file_stream: IO,
    anchor: str,
    loader_type: Type[yaml.Loader],
) -> Any:
    # "indent" level tracker as we're traversing events to fully consume the anchored subtree
    level = 0
    # Receptacle for the Events which compose the anchored subtree
    events = []
    # Commence parsing
    for event in yaml.parse(file_stream, loader_type):
        # Case 1: we're on the anchored Scalar
        if "event is a Scalar" and "event has anchor == `anchor`":
            events = [event]
            break
        # Case 2: we're entering the anchored Mapping or Sequence
        elif (
            "event is start of a Mapping" or 
            "event is start of a Sequence"
        ) and "event has anchor == `anchor`":
            events = [event]
            level = 1
        # Case 3: we're in the anchored subtree
        elif level > 0:
            events.append(event)
            # Case 3a: we're entering a nested Mapping or Sequence
            if "event is start of a Mapping" or "event is start of a Sequence":
                level += 1
            # Case 3b: we're exiting a nested Mapping or Sequence
            elif "event is end of a Mapping" or "event is end of a Sequence":
                level -= 1
            # Case 3c: we're exiting the anchored subtree
            if level == 0:
                break

It may seem pretty inductive and straightforward when you consider that there are only a few possible structures that a single anchored node can take in a YAML file. But it took a good amount of trial and error against a variety of unit test cases to settle on this design.

If you’d like more in-depth information about how this helper function is used in the package to support the !import.anchor and !import-all.anchor tags, I encourage you to check out the source code on Github.

`!import-all-parameterized`

Something that I didn’t get around to mentioning was a feature that I consider very important to the usability of the package: parameterized imports (though this may seem like a convoluted and unnecessary feature). The idea is that you can specify a glob pattern with one or more wildcards in it (like with !import-all), but you can attach to some of those glob patterns string names which are then merged into the resulting Python objects parsed from each file.

For instance, imagine we had the following directory structure:

recipes/
    - Gingerbread.yml
    - ChocolateChip.yml
    - Sugar.yml
    - Oatmeal.yml
    - PeanutButter.yml
    ...

We could imagine each of these YAML files have a similar structure,

ingredients:
  - ingredient 1
  - ingredient 2
  - ...
instructions:
  - instruction 1
  - instruction 2
  - ...

For the ease of the developer (slash baker), we can assume that they would like to be able to add new recipes to the directory without needing to touch any other configs. There might be an outer recipes.yml file where we want to import each of the recipes into a sequence of mappings of their contents.

# recipes.yml
recipes:
  cookies:
    - name: Gingerbread
      <<: !import recipes/Gingerbread.yml
    - name: Chocolate Chip
      <<: !import recipes/ChocolateChip.yml
    - name: Sugar
      <<: !import recipes/Sugar.yml
    - name: Oatmeal
      <<: !import recipes/Oatmeal.yml
    - name: Peanut Butter
      <<: !import recipes/PeanutButter.yml

# Each recipe looks like
# - name: ...
#   ingredients:
#     - ...
#   instructions:
#     - ...

This simply won’t work because of how unwieldly it is and that it breaks our core requirement that we should be able to add new recipes without adding more YAML entries to an ever-expanding index. Likewise, if we tried to use !import-all, we’d end up with a sequence of recipes which don’t have names!

# recipes.yml
recipes:
  cookies: !import-all recipes/*.yml

# Each recipe looks like
# - ingredients:
#     - ...
#   instructions:
#     - ...

So we could use this specialized tag, import-all-parameterized, to ensure that we capture the name of each file as a dedicated field in each resulting Python object:

# recipes.yml
recipes:
  cookies: !import-all-parameterized recipes/{name:*}.yml

# Each recipe looks like
# - name: Gingerbread
#   ingredients:
#     - ...
#   instructions:
#     - ...

This is a powerful feature for managing modular YAML files in a project. It allows us to outsource some of the hierarchical categories of our YAML documents to the filesystem, which can also make navigating the project’s definitions a little easier.
In addition to simple * wildcards which can store a single component of the path, entire sub-paths can also be stored as a parameter:

# recipes.yml
recipes: !import-all-parameterized recipes/{category_tree:**}/{name:*}.yml

# Each recipe looks like
# - category_tree: baking/cookies
#   name: Gingerbread
#   ingredients:
#     - ...
#   instructions:
#     - ...

Customizing local import behavior

As a final touch to make the package usable in downstream applications, I needed to finally deprecate my assumption that the imports should always be seeking from the Python process’s working directory.

I ultimately implemented this by maintaining a global variable which was used to retrieve the root of the filesystem from which all imports should be resolved. This variable could be get/set by calling a function in the corresponding module of the package:

import os
from yaml_extras import yaml_import

# Get the current import root
root = yaml_import.get_relative_import_dir()

# Set the import root to my home
yaml_import.set_relative_import_dir(os.environ.get("HOME"))

This was a simple solution to a problem that I had been putting off for a while, and I’m glad to have finally implemented it in a way which wasn’t too nasty.

Conclusion

I’m very proud of the work I’ve done on this package, and I’m excited to see if other people see the same potential in it that I do for simplifying certain repetitive and template-worthy tasks in their projects. I’m also excited to see if I can continue to improve the package and make it more user-friendly and suitable for additional use cases.

Links:

Modular YAML III: importing globs with "import-all"

2024-11-26T00:00:00+00:00

What happens when we want PyYAML to dynamically load an entire directory of YAML files into the contents of a single YAML document? At the moment, this is painfully manual, where importing all the contents of a directory require separate !import nodes for each document:

items:
  - !import items/item1.yml
  - !import items/item2.yml
  - !import items/item3.yml

In this article, I propose a new dedicated tag with its own constructor for handling this use case, called !import-all.

Handling globs with `!import-all`

Let’s begin by looking at the dataclass we’ll use to represent an instance of a tagged node, ImportAllSpec. Typical Python Path objects won’t gel nicely with including Unix-style wildcards in the path, so we’ll start by creating a custom class for handling path globs.

@dataclass
class PathPattern:
    pattern: str

We’ll want a function on this type for yielding the list of Paths captured by the glob pattern, which pathlib conveniently includes. For the sake of efficiency, we’ll memoize this function so that we never need to search the filesystem for a particular glob pattern more than once. To hash self in the memoized lookup dictionary, we’ll need to make PathPattern hashable. We can do this by just inheriting the hash of the glob pattern itself, which we also include with our implementation.

from functools import lru_cache
...
@dataclass
class PathPattern:
    pattern: str

    def __hash__(self) -> int:
        return hash(self.pattern)

    @lru_cache
    def glob_results(self) -> list[Path]:
        return list(Path.cwd().glob(self.pattern))

Great, now we have a utility type for handling the glob pattern in our ImportAllSpec. Let’s implement that now:

@dataclass
class ImportAllSpec:
    path_pattern: PathPattern

    @classmethod
    def from_str(cls, pattern_str: str) -> "ImportAllSpec":
        return cls(PathPattern(pattern_str))

This will be the container class we use when we encounter a node tagged with !import-all. Now, let’s implement the constructor for this tag:

@dataclass
class ImportAllConstructor:

    def __call__(self, loader: yaml.Loader, node: yaml.Node) -> list[dict]:
        # Handle a node tagged with !import-all
        import_all_spec: ImportAllSpec
        if isinstance(node, yaml.ScalarNode):
            val = loader.construct_scalar(node)
            if isinstance(val, str):
                import_all_spec = ImportAllSpec.from_str(val)
            else:
                raise TypeError(f"!import-all Expected a string, got {type(val)}")
        else:
            raise TypeError(f"!import-all Expected a string scalar, got {type(node)}")
        return self.load(type(loader), import_all_spec)

    def load(self, loader_type: Type[yaml.Loader], import_all_spec: ImportAllSpec) -> list[dict]:
        # Load the contents of all the files in the glob pattern
        return [
            yaml.load(path.open("r"), loader_type)
            for path in import_all_spec.path_pattern.glob_results()
        ]

Finally, we need to make sure we include the entry for the constructor in our custom ImportLoader:

...
class ImportLoader(yaml.SafeLoader):
    def __init__(self, stream):
        super().__init__(stream)
        self.add_constructor("!import", ImportConstructor())
        self.add_constructor("!import-all", ImportAllConstructor())
...

That turned out to be pretty easy! Now we can load a directory like that in the example at the start of this article with a single !import-all tag:

items: !import-all items/*.yml

When we load a file like this with our custom loader, the contents of all the files in the items directory will be loaded into the Python object, e.g. it might look like this:

items = {
  "items": [
    {
      "name": "item1",
      "value": 1
    },
    {
      "name": "item2",
      "value": 2
    },
    {
      "name": "item3",
      "value": 3
    }
  ]
}

Adding a new item4.yml file to the items directory can now be loaded into the Python object without any changes to the YAML or the underlying Python code performing the load. This is a powerful feature for managing modular YAML files in a project.

What about merging an `!import-all` node?

But there’s one missing piece, as there was before in article I. We can’t use this feature with a merge key! I.e., we can’t do this:

merged-all:
  <<: !import-all items_to_merge/*.yml

If we had a directory of files like A.yml with A: 1, and B.yml with B: 2, etc, we’d want to be able to load them all into a single mapping, like

merged = {
  "merged-all": {
    "A": 1,
    "B": 2,
    ...
  }
}

If we had a directory full of YAML files which we wanted to merge into a single object (not a sequence of mappings), then we’d want the !import-all to resolve to a sequence of mappings, which are subsequently merged. Luckily, we can just update our existing override of flatten_mapping to account for this case. In the event that the << merge key flattening method encounters a !import-all scalar node, we’ll just have it load each of the contents of the files in the glob pattern into a list of their respective Python objects, then re-represent the list as a YAML SequenceNode. The good news is that all of this actually fits into the existing ScalarNode branch of our if statement in the flatten_mapping method. Here’s the only change we need to make to the method:

# Before
if isinstance(value_node, yaml.ScalarNode) and value_node.tag == "!import":
    imported_value = self.construct_object(value_node)
    data_buffer = StringIO()
    imported_repr = yaml.SafeDumper(data_buffer).represent_data(imported_value)
    node.value[i] = (key_node, imported_repr)

# After
if isinstance(value_node, yaml.ScalarNode) and value_node.tag in (
    "!import",
    "!import-all",
):
    imported_value = self.construct_object(value_node)
    data_buffer = StringIO()
    imported_repr = yaml.SafeDumper(data_buffer).represent_data(imported_value)
    node.value[i] = (key_node, imported_repr)

Perfect - now we can use the !import-all tag with the merge key << to merge the contents of a directory of YAML files into a single mapping, like the example above. That example can now be successfully loaded as:

merged = {
  "merged-all": {
    "D": 4,
    "A": 1,
    "C": 3,
    "B": 2
  }
}

And that’s it! We’ve now got a powerful tool for managing modular YAML files in a project, utilizing the directory structure to maintain hierarchical relationships between YAML modules. In the next article, I’ll talk about a way that we can improve this importing system by allowing targeted imports of specific parts of other YAML files.

Safety note

One other thing to mention is that nothing in this article prevents a user from creating a cyclical import, e.g., a pair of files

# A.yml
contents-of-B: !import B.yml

# B.yml
contents-of-A: !import A.yml

This will cause an infinite loop when loading the files which bubbles up to a “max recursion depth exceeded” error. This is a limitation of the current implementation which I may consider addressing in a future article. For now, just be aware of this limitation when using the !import and !import-all tags in your projects.

Modular YAML II: merging imports

2024-11-25T00:00:00+00:00

Where we last left off, I was just able to import a whole YAML file’s contents into another YAML file using a custom Loader and Constructor in PyYAML for the !import tag. But there’s a missing piece. What happens if I want to merge multiple YAML files during import? First, let’s talk about the YAML merge key, “<<”, and how it works.

The YAML merge key

The YAML merge key is a special key which allows you to merge the contents of one mapping into another. It’s denoted by the << symbol. Here’s an example:

mapping-a: &A
  key-a: value-a
  key-b: value-b

mapping-b: &B
  key-b: value-Bee
  key-c: value-Cee

merged-mapping:
  <<: [*A, *B]

If we were to look into the merged-mapping key after loading with PyYAML, we would see:

{
    "key-a": "value-a",
    "key-b": "value-Bee",
    "key-c": "value-Cee"
}

An alternative syntax for using the merge key is to define << as a repeated key in the mapping, e.g.,

merged-mapping:
  <<: *A
  <<: *B

So let’s try to use our custom ImportLoader to load a YAML file which merges the contents of multiple YAML files. Suppose we have an A.yml and B.yml with the key-* mappings above defined in them. Then we can test our loader with the below example file:

# merged.yml
<<: !import A.yml
<<: !import B.yml

When attempting to load the merged.yml file, we get this runtime error:

...
yaml.constructor.ConstructorError: while constructing a mapping
...
expected a mapping or list of mappings for merging, but found scalar

How does YAML merging work?

Digging deeper into the stack trace, it becomes clear that the error is cropping up during the merge key handling, which is initiated in the Loader class using the Constructor.construct_mapping method. This method is responsible for constructing any and all MappingNodes which are encountered during the composition of the YAML document. The construct_mapping method, in turn, calls the Constructor.flatten_mapping method. As compared to the construct_mapping method, the flatten_mapping method actually handles the merge key logic. Check out its implementation in the SafeConstructor class in the PyYAML source code:

def flatten_mapping(self, node):
    merge = []
    index = 0
    while index < len(node.value):
        key_node, value_node = node.value[index]
        if key_node.tag == 'tag:yaml.org,2002:merge':
            del node.value[index]
            if isinstance(value_node, MappingNode):
                self.flatten_mapping(value_node)
                merge.extend(value_node.value)
            elif isinstance(value_node, SequenceNode):
                submerge = []
                for subnode in value_node.value:
                    if not isinstance(subnode, MappingNode):
                        raise ConstructorError("while constructing a mapping",
                                node.start_mark,
                                "expected a mapping for merging, but found %s"
                                % subnode.id, subnode.start_mark)
                    self.flatten_mapping(subnode)
                    submerge.append(subnode.value)
                submerge.reverse()
                for value in submerge:
                    merge.extend(value)
            else:
                raise ConstructorError("while constructing a mapping", node.start_mark,
                        "expected a mapping or list of mappings for merging, but found %s"
                        % value_node.id, value_node.start_mark)
        elif key_node.tag == 'tag:yaml.org,2002:value':
            key_node.tag = 'tag:yaml.org,2002:str'
            index += 1
        else:
            index += 1
    if merge:
        node.value = merge + node.value

The operative condition in the flatten_mapping method is the if key_node.tag == 'tag:yaml.org,2002:merge': check. When a << key is encountered, its value Node is expected to be either a MappingNode or a SequenceNode of MappingNodes. If this method encounters a scalar value, like !import A.yml, we land in the else block, which raises the error we saw above. The crux of the issue is that we want our !import tag to handle the loading of the file before the result is passed to the flatten_mapping method.

Handling the merge key in our custom Loader

The problem is that we can’t implement this behavior in the ImportConstructor class. The ImportConstructor class is only invoked when a node is encountered with the !import tag during construction. This error is cropping up just before we can get around to constructing the !import node. So a work-around I thought of is to override the flatten_mapping method in our custom ImportLoader class such that we can intercept the !import tag before it gets passed to the flatten_mapping method, load the file, and then pass the loaded data to the flatten_mapping method. Here’s the implementation:

from io import StringIO
...
class ImportLoader(yaml.SafeLoader):
  ...
  def flatten_mapping(self, node: yaml.MappingNode):
      for i in range(len(node.value)):
          key_node, value_node = node.value[i]
          if key_node.tag == "tag:yaml.org,2002:merge":
              if isinstance(value_node, yaml.ScalarNode) and value_node.tag == "!import":
                  imported_value = self.construct_object(value_node)
                  data_buffer = StringIO()
                  imported_repr = yaml.SafeDumper(data_buffer).represent_data(imported_value)
                  node.value[i] = (key_node, imported_repr)
      super().flatten_mapping(node)

Note that I’ve pretty much copied the internal logic of the flatten_mapping method from the PyYAML source code, except only including a case for handling the scalar !import-tagged node. We call construct_object on this node, which will identify the !import tag and call our ImportConstructor to load the file.

The problem is that, at this point, imported_value is a Python object representing the contents of the file, but the flatten_mapping method expects a Node object. So that means we need to move back up the representation chain and convert the Python object back into a Node object. Just like PyYAML has a chain of abstractions (Path -> (FileStream, Scanner) -> (Tokens, Parser) -> (Events, Composer) -> (Nodes, Constructor) -> Python Object) which converts a YAML file into a Python object, there is a reverse chain of abstractions which converts a Python object back into a YAML file. For the purposes of this post, let’s just focus on the first step, which converts a Python object back into an AST of YAML Node objects. This is performed by the Representer, which can be accessed directly on a Dumper object using the Dumper.represent_data method. Because instantiating a Dumper requires some target to which the YAML output will be dumped, I just instantiate an StringIO receptacle for this purpose. Finally, this allows me to get the imported_repr object, which is an AST of the contents of the imported YAML file. Now I can pass the result to the SafeConstructor.flatten_mapping method, which will merge the result into the document as-expected.

When we try our code now, we get the expected output:

merged = {
  "merged-mapping": {
    "key-a": "value-a",
    "key-b": "value-Bee",
    "key-c": "value-Cee"
  }
}

This is a pretty good solution, but it’s not completely done. As we saw in the core SafeConstructor.flatten_mapping method, there’s another way you can use the merge key, which is to have a sequence of mappings to merge, like we saw in the above example of merge key usage:

merged-mapping:
  <<: [*A, *B]

Handling sequences of `!import` nodes

Can we modify our ImportLoader.flatten_mapping method to handle this case as well for !import nodes? E.g.,

merged-mapping:
  <<: [!import A.yml, !import B.yml]

Of course we can! It’s just a matter of adding an additional if block to our flatten_mapping override to handle the case where the value is a SequenceNode of ScalarNodes tagged with !import. Here’s the implementation:

class ImportLoader(yaml.SafeLoader):
    ...
    def flatten_mapping(self, node: yaml.MappingNode):
      ...
        if key_node.tag == "tag:yaml.org,2002:merge":
              if isinstance(value_node, yaml.ScalarNode) and value_node.tag == "!import":
                  ...
              if isinstance(value_node, yaml.SequenceNode):
                  for j in range(len(value_node.value)):
                      subnode = value_node.value[j]
                      if isinstance(subnode, yaml.ScalarNode) and subnode.tag == "!import":
                          imported_value = self.construct_object(subnode)
                          data_buffer = StringIO()
                          imported_repr = yaml.SafeDumper(data_buffer).represent_data(
                              imported_value
                          )
                          value_node.value[j] = imported_repr
                  value_node.value.reverse()
                  node.value[i] = (key_node, value_node)
        return super().flatten_mapping(node)

Note that I’m following the exact same order-of-operations as the original SafeConstructor.flatten_mapping implementation for sequences of MappingNodes. When I recognize that the value of the << key is a sequence, I iterate over the sequence values. When I encounter an !import node, I perform the same steps as before to load the file and convert the Python object back into a Node object. Finally, to match the implementation of the merge key for sequences of MappingNodes, I reverse the sequence of Node objects and replace the original SequenceNode with the new sequence of Node objects. This reversal ensures that the precedence of merged mappings is maintained, i.e. the last mapping in the sequence takes precedence over the first mapping.

And that’s it! We now have a fully-functional !import tag which can be used to import YAML files into other YAML files, plus it can handle merging multiple imports into a single mapping using the << YAML merge key. It can even handle a mix of !import and non-!import mappings in the same merge:

override-config: &override
  host: "localhost"
  port: 8080
  threads: 16
  connection_profile: lab

config:
  <<: !import default-config.yml
  <<: *override

The functionality we’ve implemented here allows us to create modular YAML files which can be imported and merged into other YAML files. But there’s more to be done to make this sufficiently general and robust. In the next post, I’ll talk about a variant we can implement to handle the case where we want to import an entire glob pattern of YAML files as a sequence of Python objects.

Modular YAML I: a custom "import" tag

2024-11-22T00:00:00+00:00

Since I’ve started working at Intuitive full-time, I’ve been interacting with YAML files in some capacity pretty much every day. Some of that is because I do a lot of dbt development as a data engineer, and dbt uses YAML files to define models, tests and other properties about the data model. And there’s pre-commit, Gitlab CI, as well as other custom Intuitive tools that use YAML files to define configurations.

This frequent interaction with YAML has led me to also write some of my own tools and scripts which depend upon source data which is laid out in YAML files. One such project I’ve written with these tools is a Python CLI which reads a directory of YAML files and Jinja templates and outputs a directory of rendered files. The original purpose of the kernel of this project was for mass-producing Jira content and standardizing other documentation. I’ve also used it to mass-generate unit tests which are so similar to one another that they can be effectively templated.

Something that most of my Python code would be preoccupied with was crawling the local directories for YAML files and associating them with the right Jinja templates in a way that scaled well, was flexible and DRY. I’m still working on this broader project of general DRY templated document generation to this day.

What are YAML tags?

YAML tags are a way to extend the YAML specification to include custom data types. The YAML specification includes a number of built-in tags, such as !!str for strings, !!int for integers, !!float for floats, and !!bool for booleans. Syntactically, they only need one leading exclamation mark, but most of the built-in tags have two.

# Example of YAML tags
string: !!str "Hello, world!"

As you can see, they’re sort of like anchors in how they just immediately precede a YAML node. Except that they’re more like type hints, or general-purpose annotations. They’re not mutually exclusive - the syntax to tag and anchor a YAML node is as follows:

# Tagging and anchoring a YAML node
tagged-and-anchored: !my-tag &my-anchor "my value"

Unlike anchors, tags can be used as many times as you want (anchors need to be unique for the YAML resolver to properly resolve them). Tags can also be used to define custom data types, which is why sometimes you’ll see them if you dump a YAML representation of a custom class from Python.

# Example of a custom data type
!MyCustomType
custom_instance:
  - my_key: !!str "my value"

So if I wanted to “import” a YAML file from another YAML file, it would be syntactically sound to define a custom tag for this purpose.

Envisioning “!import”

One part of the puzzle which might get me closer to being able to flexibly managing YAML files in a DRY way that scales well is to use a modular approach. If I’m able to organize documents in a directory tree to represent hierarchical relationships between documents, I can effectively split responsibilities between documents. The core mechanism that would allow this functionality is some way to reference the contents of YAML file $X$ from within a YAML file $Y$.

I’d imagine that this operation looks something like this: \

# X.yml
x: !import Y.yml

If we were to suppose that Y.yml contained the following:

# Y.yml
y: "Hello, world!"

Then we’d want to be able to read X.yml into memory as the following Python dictionary:

X = {
  "x": {
    "y": "Hello, world!"
  }
}

Let’s talk about PyYAML

PyYAML is by far the most popular and most widely-used YAML parser for Python. It’s a wrapper around the C library libyaml, and it’s the parser that’s used by dbt to parse YAML configurations.

When Googling to see if anyone’s solved my “YAML importing” problem yet, I was not disappointed by the excellent design of pyyaml-include. Its design of the !include tag was a great inspiration for me to figure out how to implement my own tagging system which followed different conventions to serve different purposes.

Reading the source code for pyyaml-include introduced me to the pattern of using a custom constructor to parse a specialized tag. This is the pattern I would use to implement my own !import tag.

PyYAML loads data from a YAML file into a Python object using a Loader object. The process of loading a YAML file starts with the Scanner, which reads the file character by character and tokenizes it into a stream of tokens. The Parser then takes these tokens to build a sequence of parsed events. The Composer then takes these events and builds an abstract syntax tree (AST) of nodes. Finally, specialized Constructors take each node of the AST to build a complete Python object. All the way through this process, tags are propagated as fields of the events and nodes. However, when the Constructor is being chosen for a node, the tag on the node will determine which constructor is used.

Extending PyYAML

If we want to be able to parse our custom !import tag, we’ll need to create our own Constructor class which gets called to handle nodes which are encountered with an !import tag. For good measure, we’ll also create a custom Loader class to ensure that our custom constructor is used with this specific tag.

from dataclasses import dataclass
from pathlib import Path
from typing import Type, Any
import yaml

@dataclass
class ImportSpec:
    path: Path

    @classmethod
    def from_str(cls, path_str: str) -> "ImportSpec":
        return cls(Path(path_str))

@dataclass
class ImportConstructor:

    def __call__(self, loader: yaml.Loader, node: yaml.Node) -> dict:
        # Handle a node tagged with !import
        import_spec: ImportSpec
        if isinstance(node, yaml.ScalarNode):
            val = loader.construct_scalar(node)
            if isinstance(val, str):
                import_spec = ImportSpec.from_str(val)
            else:
                raise TypeError(f"!import Expected a string, got {type(val)}")
        else:
            raise TypeError(f"!import Expected a string scalar, got {type(node)}")
        return self.load(type(loader), import_spec)

    def load(self, loader_type: Type[yaml.Loader], import_spec: ImportSpec) -> Any:
        # Just load the contents of the file
        return yaml.load(import_spec.path.open("r"), loader_type)

class ImportLoader(yaml.SafeLoader):
    def __init__(self, stream):
        super().__init__(stream)
        self.add_constructor("!import", ImportConstructor())
     

That may seem like a lot of code, but it’s actually pretty simple. The ImportSpec class is a simple dataclass which wraps a pathlib.Path object and provides a method to construct an ImportSpec from a string. The ImportConstructor class is a callable class which takes a yaml.Loader object and a yaml.Node object and returns the contents of the YAML file specified by the scalar string in the node. The ImportLoader class is a subclass of yaml.SafeLoader which adds the !import tag to the constructor map with an ImportConstructor object.

Using our custom tag

Now we can load a YAML file with the !import tag and have the contents of the file loaded into the Python object. Here’s an example of how we might use this in practice:

import yaml
from yaml_import import ImportLoader

X_yml = Path(__file__).parent / "X.yml"
X_data = yaml.load(X_yml.open("r"), ImportLoader)
X = json.dumps(X_data, indent=2)
print(f"{X = }")

Which yields,

X = {
  "x": {
    "y": "Hello, world!"
  }
}

Nice! Now we can import YAML files from other YAML files. There’s more to be done, though. In the next post, I’ll talk about making this !import tag work with the YAML merge key “<<”. Stay tuned!

Linear Regression in Rust with AutoGrad

2022-05-05T00:00:00+00:00

I’ve spent the past two weeks just starting to get a handle on how to use the popular low-level, memory-safe programming language Rust. In general, it gives me C++ vibes when I’m programming in it. That said, I think that it could have good potential as a machine learning development environment, so long as it is supplied with the appropriate external libraries (or “crates,” to use Rust’s terminology.)

So, to get a handle on how I could use Rust specifically for machine learning purposes, I did some (very brief) research into what tools might be useful. I knew that I wanted to make use of an auto-differentiation tool, as this is essentially the bedrock of a machine learning dev environment. Auto-differentiation allows us to compute partial derivatives of the output of our models such that we can tweak and tune them into being better predictors. I looked over three different auto-differentiation crates:

easy-ml : This is actually a much more comprehensive machine learning crate with lots of out-of-the-box machine learning implementations which are easy to implement. It also contains a module, differentiation which houses its auto-differentiation engine. This module could suffice for my purposes, but I don’t like the overhead of needing to install all of the extra bells and whistles provided by easy-ml just to get access to its differentiation module. I’d rather use a crate which contains only the necessary tools & traits for auto-differentiation in particular.
autodiff : The problem with using this crate in particular is that it is mostly tailored toward calculus on scalar functions, whereas most machine learning applications require higher-dimensional differentiation in the form of vectors, matrices and scalars. I don’t know if autodiff is necessarily incapable of implementing these features, but most of their examples and documentation emphasize their scalar equivalents.
autograd : Of the above two alternatives, autograd certainly takes the cake for popularity and satisfies my complaints about the other two. The crate implements auto-differentiation on the standard Rust matrix/tensor crate, ndarray, which can allow for maximal cross-compatibility with other matrix-related crates. Moreover, it is very well documented and small enough for me to be able to scan the source code to answer questions not addressed by the documentation. Moreover, it’s easily extendible for custom tensor operations and I/O pipelines.

So, I decided to go with autograd for my auto-differentiation engine. I could have gone deeper into my research to find other alternatives, but these three appeared to be the most popular (according to crates.io), and autograd serves all of the basic purposes which I am looking for.

Linear Regression as a Neural Network

The simplest machine learning application I could come up with to test this crate out (which wasn’t already done in their tutorials collection) was a simple linear regression neural network. For those who don’t know, a neural network is essentially a statistical model which recursively applies a linear regression and some nonlinear activation function. So, if I wanted to use a neural network for the purposes of a simple, one-dimensional linear regression, I would use a 1D input layer connected directly to a 1D output layer with no activation function. The weight parameter of the single neuron in this network corresponds to the slope of our linear regression, while its bias parameter is the offset. This is why the output equation of our model would look exactly like the formula for a line:

$$ y = mx + b \qquad \Leftrightarrow \qquad y = \text{weight}\cdot x + \text{bias} $$

In a diagram, this linear regression model looks like the neural network pictured below:

Obviously, if this was all I wanted to use my auto-differentiation engine for, it would be woefully overpowered, but this is just a simple “hello world.” More generally, if I really wanted (only) to implement a linear regression algorithm in Rust, I would just use the closed-form formula for a linear regression and use ndarray-linalg for matrix multiplications and inverses to implement it. However, in future, I’d like to construct more complicated neural models which will require the full power of autograd, so this is a small proof-of-concept which allows me to get comfortable with the crate.

Now that I have a basic idea of what sort of model I’m going to be building, I can start writing some Rust code.

Implementing the Model with AutoGrad

The main example from autograd’s GitHub repo with which my project has the most overlap is the sine.rs example, which regresses a multi-layer perceptron (MLP) to the sine function. Because I’m doing something simpler, I’ll only be following it insofar as it is a useful reference for how regression works generally in autograd.

The first thing I do is import a handful of modules from the crate:

use autograd as ag;
use ag::optimizers::*;
use ag::optimizers::adam::Adam;
use ag::prelude::*;
use ag::tensors_ops::*;

The first of these just assigns a nickname, ag, which I can use in place of the longer namespace, autograd. After that, I’m importing tools for optimizers, the Adam optimizer specifically, some high-level traits from autograd’s prelude, and the tensor_ops module.

Next, I create my main function, and start initializing the core engine of autograd: the so-called VariableEnvironment, along with an instance of a random number generator.

fn main() {
    // Initialize VariableEnvironment and (default seed) RNG
    let mut env = ag::VariableEnvironment::new();
    let rng = ag::ndarray_ext::ArrayRng::<f32>::default();
    // --snip--

In short, the VariableEnvironment is a container which manages all of the variables which we use in learning and keeps track of how they change throughout the learning process. In setting it up, I register our two learned parameters in the VariableEnvironment, which involves associating them with a name:

// --snip--
// Initializing our parameter tensors
env.name("w").set(rng.standard_uniform(&[1, 1]));
env.name("b").set(ag::ndarray_ext::zeros(&[1, 1]));
// --snip--

With env.name(), we are registering a new NamedVariableSlot in the environment, which we then immediately give an initial value with the set() method. The arguments of the functions inside of the set call are each references to arrays of usize, each of which state that we are initializing w with a $1\times 1$ tensor of random values, and that we are initializing b with a $1\times 1$ tensor of zeros.

So, our next step in initializing the model is creating an instance of an optimizer which governs how we use the partial derivatives we compute in order to reduce loss and predict outputs more accurately. We’re using the Adam optimizer for this, which we give some parameters to:

// --snip--
// Initializing our optimizer (Adam)
let adam = Adam::new(
  0.01,  // Learning rate (alpha)
  1e-08, // Error Toler. (epsilon)
  0.9,   // First Moment decay (beta1)
  0.999, // Second Moment decay (beta2)
  env.default_namespace().current_var_ids(), // Variable IDs from 
                                              // VariableEnvironment namespace
  &mut env,                                  // VariableEnvironment instance    
  "linear_reg_adam"                          // Name string
);
// --snip--

All of the parameters I’ve put in above are generally considered to be the “default” parameters for most use cases of Adam, but I’ve beefed up the learning rate ($\alpha$) by a factor of 10 to speed up learning on such a dimensionally small model. It’s also noteworthy that I need to provide the optimizer with the Variable IDs from the VariableEnvironment (so it knows what our learned parameters are), as well as a mutable reference to the environment. We also supply it with a name.

Now I start thinking about the training process. For the purposes of this project, I’d like to train the linear regressor in batches of size $64$, over the course of $1000$ epochs (i.e. batches). So I just quickly define some constants for these quantities:

// Initializing training constants
const N_EPOCHS : usize = 1000;
const BATCHSIZE : usize = 64;

At this point, I’m almost ready to code up the training loop. But it feels like I’m forgetting something; perhaps the actual training data itself? Where am I getting that data from? Because we’re working totally with a synthetic dataset for this project, it suffices to define some “true” values we expect our network to learn, then use those values (and some data noise) to generate data points as we train. So, I define the parameters I’d like to learn:

// Setting up the (synthetic) dataset
let true_w = 2.5f32;
let true_b = -1.0f32;
let noise_scale = 0.5f32;

Now we can jump into the meat of our algorithm: the training loop. This consists of a for-loop to iterate over the epochs, along with the following contents which I will explain:

// --snip--
for epoch in 0..N_EPOCHS {
    env.run(|ctx| {
        // Randomly generate input coords
        let x = standard_uniform(&[BATCHSIZE, 1], ctx) * 20. - 10.;

        // Retrieve variables
        let w = ctx.variable("w");
        let b = ctx.variable("b");

        // Compute expected output (w/ noise)
        let y = true_w * x + true_b +
                standard_normal(&[BATCHSIZE, 1], ctx) * noise_scale;

        // Compute LinReg predictions
        let z = matmul(x, w) + b;

        // Compute Mean Sq Error Loss
        let mean_loss = reduce_mean(square(z - y), &[0], false);
        let ns = ctx.default_namespace();
        // Compute gradients for variables
        let (vars, grads) = grad_helper(&[mean_loss], &ns);
        // Compute weight/bias change
        let update_op = adam.get_update_op(&vars, &grads, ctx);
        // "Push" both loss and changes to evaluation buffer
        let results = ctx.evaluator().push(mean_loss).push(update_op).run();

        // Print training loss every 100 epochs
        match epoch % 100 {
            0 => println!(
              "Mean Loss (epoch {}) = {}", epoch, results[0].as_ref().unwrap()
            ),
            _ => ()
        }
    });
}
// --snip--

Here we note that we are passing a function to our VariableEnvironment via the function run(). This function expects its argument to be a function with an argument of a type called Context, which is often abbreviated as ctx in variable names, which I use in the closure. Anything we run inside of the closure we pass to env.run() is used to create a Graph, which is a component frequently seen in auto-differentiation engines. Without getting into too many nitty-gritty details, you can think of this as a graph where nodes are variables in our model, and we have arrows between them indicating how data flows between them such that we can “trace back” how they interact with one another and use this data to compute partial derivatives.

So what does all of this mean for us? It means that anything we do inside of env.run() will be “watched” by the graph so we can compute derivatives. Let’s take a look at what we do inside of the context closure, one digestible chunk at a time:

// Randomly generate input coords
let x = standard_uniform(&[BATCHSIZE, 1], ctx) * 20. - 10.;

// Retrieve variables
let w = ctx.variable("w");
let b = ctx.variable("b");

// Compute expected output (w/ noise)
let y = true_w * x + true_b +
        standard_normal(&[BATCHSIZE, 1], ctx) * noise_scale;

// Compute LinReg predictions
let z = matmul(x, w) + b;

What we initially do is generate a tensor of uniformly-distributed input coordinates, spread evenly on the interval $(-10, 10)$. Then, I call the variable() function from the Context instance, which allows me to retrieve a variable tensor by the name I assigned to it earlier on when I initialized it. The next thing I do is generate the expected output for these data points according to the line formula, to which I add a (scaled) tensor of Gaussian noise to add some realistic data noise like we would see in a real dataset.

On the last line, I assign to z what’s normally called the “forward propagation” of the network. That is, I take the tensor batch of input points and propagate it through the model and compute the output which gives our model’s prediction. This is done through a matrix multiplication (via matmul() from the tensor_ops module) after which we add our bias tensor variable b.

// Compute Mean Sq Error Loss
let mean_loss = reduce_mean(square(z - y), &[0], false);
let ns = ctx.default_namespace();
// Compute gradients for variables
let (vars, grads) = grad_helper(&[mean_loss], &ns);
// Compute weight/bias change
let update_op = adam.get_update_op(&vars, &grads, ctx);
// "Push" both loss and changes to evaluation buffer
let results = ctx.evaluator().push(mean_loss).push(update_op).run();

In this next chunk, I start by computing the current loss of the model on the current batch, storing the value in mean_loss. Because I’m using Mean Squared Error (MSE) as our loss metric, I can compute it very simply by constructing a tensor of the difference between our prediction and expectation (z - y), squaring its elements (via square()), then, taking the average of the resulting tensor in the form of a scalar (accomplished by reduce_mean). The extra arguments at the end of reduce_mean are an array of the axes along which we’re taking the mean (&[0]), and false to indicate that we want to discard the axis we’re taking the mean along, which leaves us with a scalar.

The next meaningful line is where we assign values to vars and grads - this uses the function grad_helper() from the optimizers module, to which I hand both an array of the loss variable(s) I’m trying to minimize (in our case, &[mean_loss], the MSE variable), as well as the namespace from our Context which contains all of our learnable parameters (in our case, w, and b as variable tensors).

I then use the adam optimizer instance to call adam.get_update_op(), which takes in references to the variables and their respective gradients, as well as the Context instance, and returns the adjustments that need to be made to the variables in order to reduce the loss, which I call update_op.

In the next line, I call a sequence of functions in the form of ctx.evaluator().push().run(). The Context instance contains an Evaluator, which actually enacts the changes suggested by our optimizer. The pattern important here is “pushing” values to the evaluator, then finally calling run() to turn those values into instructions that get evaluated. In computer science, data structures that store a set of instructions and only evaluate their output when it’s necessary is known as a “lazy data structure.” This is why autograd is built upon a “Lazy Tensor” evaluation engine - the results aren’t explicitly computed (which can be costly) until we push them to the evaluator and call “run.”

I initially had to ask; “why are we pushing our mean loss?” After all, that’s not necessary for any of the tensor updates. The main reason is so that we can print out the evaluated value of mean_loss. Because we computed mean_loss purely with tensor variables, the resulting scalar is still a tensor variable and doesn’t actually contain a numeric value. Because I’d like to print out what error we have, I need to push it to the evaluator and get that numeric value lazily. In the next line, I use a match statement (which I like to think of as Rust’s equivalent of a switch statement) to print out the loss of the algorithm every 100 epochs.

// Print training loss every 100 epochs
match epoch % 100 {
    0 => println!(
      "Mean Loss (epoch {}) = {}", epoch, results[0].as_ref().unwrap()
    ),
    _ => ()
}

Note that the index I’m using to select from results to get the numeric value of mean_loss corresponds to the order which we pushed symbolic tensors to the evaluator. Because I pushed mean_loss first, it takes index 0. The actual entry at that index is a Result type (which can be either an Ok value for the numeric tensor value, or an Err error value), so I need to unwrap() it.

Now that we’ve coded up the training loop, the last thing there is to do is handle behavior of this script after the training is complete. I’d like to save the learned parameters to a file, which is luckily very easy with autograd:

// --snip--
env.save("results.json").unwrap();      // Save the learned parameters to a JSON file
// --snip--

The very last thing I do before ending the script is print to the command line what the learned parameters are after our last epoch, so we can visually compare them to their “true” values we set for our synthetic dataset.

// --snip--
let ns = env.default_namespace();
// Borrow the learned weight and bias values
let finalw = &ns.get_array_by_name("w").unwrap().borrow();
let finalb = &ns.get_array_by_name("b").unwrap().borrow();
// Print them out
println!("Final w = {}", finalw);
println!("Final b = {}", finalb);

Once again, I make use of the environment’s namespace, storing it as ns. It is from this Namespace instance which I can borrow the latest evaluated values assigned to each variable by calling the function get_array_by_name(), which also results in a Result, so we must unwrap it. However, the Result itself wraps a RefCell object (used often in multi-threaded applications), from which we can get the stored value via the borrow() function.

And that’s pretty much all there is to it! Using cargo build gives us no compiler errors or warnings, and we can execute the resulting compiled program to see the results we expected:

I’m satisfied with these results, because it’s clear that the error quickly converged within the first 700 or so epochs, and our learned parameters are within 0.05 of their “true values.”

Where to go from here

I’d like to find more practical uses of the auto-differentiation utilities provided by autograd in a more comprehensive machine learning project. I think the next application I’ll try to implement with this will be a custom Variational Autoencoder (VAE). Stay tuned for blog updates to see how I get on with implementing this (admittedly, much more complicated) model in the future.

dsillman

TF-GCP II: Uploading static content to GCP storage

A basic “hello world” site, with Tailwind

GCS bucket object resources

Viewing the site content

Making the bucket public

TF-GCP I: Provisioning GCP resources with Terraform

How they tell you to do it, vs. how you should do it

Prerequisites

Setting up Google Cloud

Setting up our Terraform project

Let’s provision a GCS bucket

The issue with GCS buckets

Utilizing Terraform’s random provider

Closing note

Modular YAML released: yaml-extras on PyPI

!import.anchor and !import-all.anchor

!import-all-parameterized

Customizing local import behavior

Conclusion

Modular YAML III: importing globs with "import-all"

Handling globs with !import-all

What about merging an !import-all node?

Safety note

Modular YAML II: merging imports

The YAML merge key

How does YAML merging work?

Handling the merge key in our custom Loader

Handling sequences of !import nodes

Modular YAML I: a custom "import" tag

What are YAML tags?

Envisioning “!import”

Let’s talk about PyYAML

Extending PyYAML

Using our custom tag

Linear Regression in Rust with AutoGrad

Linear Regression as a Neural Network

Implementing the Model with AutoGrad

Where to go from here

Utilizing Terraform’s `random` provider

Modular YAML released: `yaml-extras` on PyPI

`!import.anchor` and `!import-all.anchor`

`!import-all-parameterized`

Handling globs with `!import-all`

What about merging an `!import-all` node?

Handling sequences of `!import` nodes