Recently HDD storage pool in my Homelab started to work “funny”. Funkiness appeared as pool clogging when heavy traffic happened on it. I’ve lived with that, restarting machine occasionally for a month or so, being busy with the life.

But one day I thought that maybe it’s time to make a backup of the data on that pool. Oh boy, was I wrong and right at the same time. Making backups was definitely right thing to do, but not at that specific time.

After I’ve started the backup process, the pool started to clog again. “No problem, another reboot and we’re alive” I’ve thought. And this time it was different - the pool didn’t import anymore.

All disks in pool had clear SMART status before the malfunction. So I’ve started to look into the zfs status output. And it was not good:

1
2
3
4

# zpool import hdd
              cannot import 'hdd': I/O error
        Destroy and re-create the pool from
        a backup source.

One of the disks in the pool was dead. And it was dead that much, that it interfered with controller’s operation. Moreover, last reboot not only didn’t help, but caused the pool to be exported forcefully (I guess) and it broke the metadata.

Troubleshooting

First, I’ve tried to import the pool with -f flag, then -f -F, but it didn’t help either. The pool was definitely dead.

Normally, I would just replace the broken disk with a new one, replace it in zfs and do a resilver. But the problem was that the metadata was corrupted. And pool with corrupted metadata can’t be imported. And without importing the pool, I can’t replace the disk.

After some Googling and more or less proper troubleshooting, I’ve found a way to import the pool with broken metadata. These commands did the trick:

1
2
3

echo 1 > /sys/module/zfs/parameters/zfs_max_missing_tvds
echo 0 > /sys/module/zfs/parameters/spa_load_verify_metadata
echo 0 > /sys/module/zfs/parameters/spa_load_verify_data

First one makes it possible to import the pool with missing disk. The other two disable metadata and data verification. It’s not recommended to use these commands in production. But in my case, I didn’t have anything to lose.

After that, wiser with the knowledge gathered during Googling, I’ve imported the pool in read-only mode:

1

zpool import -f -o readonly=on hdd

And it worked! The pool was imported in read-only mode. I’ve checked the data and it was there. Mostly. Some files were corrupted, but most of them were intact.

Data recovery

After the pool was imported, I’ve tried to copy the data to another disk. First with simple cp, then with rsync. But they all hanged on broken files.

Another research later I’ve found a tool called cpio. It’s a tool that can copy files to and from archives. But with some flags, and some pipes magic, I’ve managed to use it to copy file-by-file.

1

find /hdd -depth -print0 | cpio -pdmv0 /target

This command copies all files from /hdd to /target.

After a few hours, the data was copied. I’ve managed to rescue about 70% of files intact. The rest was corrupted. But it was better than nothing :)

Conclusion

Backups, people!

Shame one me, cause it was second time when I lost some data because of broken disk. And it was second time when I didn’t have a proper, automated backup. I’ve learned my lesson and I’ve started to make backups of my data. And I recommend you to do the same.

The good thing is, I’ve had some old (manual) backup of the very same pool, so I’ve managed to backfill some of the lost files from it.

RAID (or ZRAID) is not a backup

I knew that, but I didn’t act on that knowledge. RAID is not a backup. It’s a redundancy. It’s good to have it, but it’s not enough. You should have a backup of your data.

SMART monitoring won’t help you sometimes

All disks in the pool had clear SMART status before the malfunction. It just happens that disk can die without any SMART errors. So don’t rely only on SMART monitoring.

But it’s still better to have them monitored than not. I’ve had few disks that started showing SMART errors, and I wast fast enough to replace them before they died.

Recently I’ve started The Big Migration™ from Ansible to Terraform in my homelab. But as soon as I started to write my first Terraform manifests, I’ve started to think about remote state storage too.

First, I thought about storing the state file on selfhosted Minio instance. But the problem is, that Minio instance will be managed by the very same Terraform manifests.

Later on, I started researching what else can I manage with Terraform. I’ve found out that Backblaze B2, where I keep backups anyway, is manageable with Terraform! That was the moment when it clicked in my head - all in all, B2 is S3 compatible, so I can use it as a remote state storage for Terraform!

Prerequisites

1. Backblaze B2 account, with:
   1. Bucket in which you want to store the state file
   2. Application key with permissions to manage this bucket
2. Terraform installed on your machine

Terraform configuration

The basic config for storing the state file in S3 would look like this:

1
2
3
4
5
6
7

terraform { 
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "terraform.tfstate"
    region         = "us-east-1"
  }
}

One might think that it’s enough to change the region to B2 substitute, add Backblaze endpoint and we’re good to go. But it’s not that simple - B2 is almost S3 compatible, so we have to do some extra steps.

What we need to do is to skip some checks and validations that Backblaze B2 doesn’t support. And there’s actually quite a lot of them.

A fully working example of Terraform configuration for storing the state file in Backblaze B2 would look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

terraform {
  backend "s3" {
    bucket    = "my-terraform-state-bucket"
    key       = "terraform.tfstate"
    region    = "us-west-004"
    endpoints = {
      s3 = "https://s3.us-west-004.backblazeb2.com"
    }

    skip_credentials_validation = true
    skip_region_validation      = true
    skip_metadata_api_check     = true
    skip_requesting_account_id  = true
    skip_s3_checksum            = true
  }
}

Of course, you’ll have to replace bucket, region, and endpoints.s3 values with proper ones according to your Backblaze B2 config.

As you can see, there’s no access_key and secret_key provided. That’s because I provide them through environment variables (and you should too!). B2’s application key goes to AWS_SECRET_ACCESS_KEY and key ID goes to AWS_ACCESS_KEY_ID env var.

Some security considerations

State bucket

Keep it private. It’s not a good idea to make your state file publicly available, as it might contain secrets.

You might also want to enable versioning on this bucket. With versioning you can easily revert to the previous state if something goes wrong. I’ve seen Terraform go bananas a few times, so it’s a good idea to have it enabled.

Application key

Don’t use your master key for this. Create a separate application key with permissions to manage only this bucket. It’s a good practice to have separate keys for different tasks.

Don’t put your credentials in the Terraform code (or any code, really). Especially if you’re gonna ever put that code publicly eg. on GitHub. One “oops” too far and your keys leaked. Use environment variables to provide them to Terraform. I personally load them into env vars from 1password with op cli tool.

Summary

It seems that Backblaze B2 is enough S3-compatible to be used as a remote state storage for Terraform. It’s good to keep your state file in some remote storage, as it’s a good practice to have it versioned and not stored on your local machine. And if you already use B2 for backups, why not use it for Terraform state file as well?

Compose Key is a special key on your keyboard that allows you to type special characters like →, °, €, ä, ™, ®, ¿ and many more. It’s very useful if you’re writing in any foreign language that uses special characters and don’t want to mangle your keyboard settings all the time.

Examples

• Compose + - + > will produce a → character
• Compose + o + o will produce a ° character
• Compose + c + = will produce a € character
• Compose + a + " will produce a ä character
• Compose + t + m will produce a ™ character
• Compose + r + o will produce a ® character
• Compose + ? + ? will produce a ¿ character

As you can see in the examples above, the combinations are very thought out and easy to remember. Most of the time you don’t need to know them by heart. You just have to think what two (or more) keys from the standard key set combined will give you the character that you want.

How to enable Compose Key?

As I use Gnome 3, I’ll describe how to set it up in this environment. But don’t worry, it can be enabled in pretty much every WM/DE.

So, at first you have to go to the Settings and then Keyboard:

Then you have to click the Compose Key option in the Special Character Entry section. There you can turn it on and choose which key you want to use as a Compose Key.

As you can see on the screenshot, I use the Caps Lock key, but choose whatever you like.

Lists of possible combinations

These two lists are the best that I found in the internet:

1. GTKComposeTable - rather short, very readable list of most useful combinations.
2. libX11 documentation - very long, hard to read but very complete list of all possible combinations.

I’m Andrzej, I live in Poland, Warsaw and at the time of writing these words, I’m 29 years old. I have 7 years of professional experience as a Python developer and 2 years as DevOps and a short episode as a team leader. In my free time, I’m maintaining a small homelab and I do various more or less strange things with my cars.

I’m starting my personal blog today.

What I’ll write about? Mostly about the things that I mentioned above, as I know something about them…

…and maybe even some other topics, you never know what you’ll learn by accident 🙂

Homelab

I’m working constantly on my homelab setup. The goal is to have everything as code, so I can easily recreate my homelab from scratch if needed. There’s a lot of stuff in there, from Ansible playbooks, through Kubernetes manifests, to Terraform configurations.

• GitHub repository: andrzej3393/homelab-v2
• Documentation: homelab.andrzejgor.ski