I have this client – they’re in the manufacturing vertical and utilize me for projects and maintenance of their SCADA environment. Because this is a secured environment, the network is segmented with no internet access for nearly all machines and very restricted access for those that require it. Running Veeam Backup & Replication segmented from the internet is generally no problem and in many ways is even recommended. And it has worked great for years in their standard configuration. With that said, the customer was utilizing a couple of QNAP NAS’s as SMB backup repositories but they always have an eye on securing their environment loved the idea of data immutability. As such, when one of the old NAS’s failed, options were discussed. While they already leverage tapes for an air-gapped, offsite backup copy, they also aren’t the best at swapping out those tapes when the time comes every month, so they were also looking for a more hands-off approach to storing their backup data with immutability as an added bonus.

Enter the Object First OOTBI (Out of the Box Immutability) appliance.

While leveraging a Veeam Hardened Repository was also a consideration, the customer liked the turn-key approach and the advantages of utilizing object storage. In this particular configuration, they wanted to leverage two repositories as they had in the past — one as the primary repository and the second as a copy job target. Each repository would reside in separate buildings on their sprawling campus, and they could or course still utilize tapes to take their copies off-site.

For those that aren’t aware, installing an Object First OOTBI is simple-stupid. Rack the devices, plug them in, some quick configuring, create buckets, add to the buckets as repo’s in VBR, and start using them. In fact, Object First has often-times noted that you can go from unboxing to backing up in 15 minutes. Now, my experience is that it takes a bit longer, but it’s arguably close to it if you exclude the racking and configuration of your network. Except today that wasn’t the case.

While I installed and configured the appliances with no issue, I ran into issues adding the bucket as a repository. I encountered the below errors when attempting to list the buckets and save the object repository.

Error    Failed to save object storage repository: Failed to establish connection to AmazonS3Compatible. Error: 503 (ServiceUnavailable). See log file Util.InfraItemSaver on server <SERVERNAME> for details.

Error    Infrastructure item save failed Error: Failed to establish connection to AmazonS3Compatible. Error: 503 (ServiceUnavailable). See log file Util.InfraItemSaver on server <SERVERNAME> for details.

Now let me start with just applauding Object First support for being excellent at their jobs. Support was very responsive and knew exactly what to be looking at. I was going to be told to check out two possible factors.

1. When using object storage, the server time must remain in sync with the repository, and thusly, make sure that the OOTBI is set to sync with a valid time server. In this case, the default time servers were still in place. They default to time.cloudflare.com, pool.ntp.org and ntp.ubunti.com. However, in a disconnected environment with no internet access, these will obviously not work.
2. When using object storage, certificates are utilized for data encryption because the data flows via an HTTPS connection. However, as noted, Windows will attempt to validate certificates and when a server is offline, it cannot reach out to the internet to validate the certificate chain. In that case, Windows needs to be configured to not attempt validate that certificate chain, and in reality, this is probably pointless because the certificate is self-signed from the OOTBI appliance itself so it’s not going to validate anyway.

In my case I had already determined there was an issue with NTP because Object First is happy to show a warning when NTP servers are unavailable for too long as seen below. In my case, I had to update the utilize the local NTP servers, in this case, the Active Directory Domain Controllers which is what the VBR server syncs to as well. Clearly, this was not the source of the issue.

That leaves the certificate verification issue. Now, I will note that when attempting to add the repository, it was very slow to load the certificate because it is attempting to verify the certificate chain. And once you click through to the next screen, after hitting Browse to view the list of buckets, it will likely sit for a fair amount of time at “Getting the list of S3 Compatible buckets…” before throwing the aforementioned “503 (ServiceUnavailable)” error. I found that I could however manually specify the bucket name, or in most cases, attempt to list the buckets a second time, but event after working around the error, I still will receive the failure messages on the final screen when saving the repository to the Veeam configuration. So back to the fix…

On your VBR server, you’re going to want to open your security policy editor (gpedit.msc). In the Local Group Policy Editor, browse to Computer Configuration > Windows Settings > Security Settings > Public Key Policies and select Certificate Path Validation Settings. Then select the Network Retrieval tab, check Define these policy settings and uncheck Automatically update certificates in the Microsoft Root Certificate Program (recommended) and click OK to set this policy. In my experience, setting a local security policy does not require a reboot and you can go right back to adding the object repository.

Now when adding the repository, the certificate validate happens much faster (because the validation doesn’t happen – though you will still need to verify and confirm acceptance of the self-signed certificate that is presented) as does listing the available buckets. And of course, success when saving the repository!

Next up, stay tuned for how to properly update the firmware on your OOTBI when you cannot access the updates via the internet directly.