Extra backup protection for Veeam with Synology snapshots
Background
In some of my previous blog posts, I have been exploring the functionality and capabilities of a Synology NAS which so far has been storage configuration and testing as well as Synology native Office 365 backup. I also wrote about data protection strategies to pose the question, what is enough in terms of data protection? Recently I have been reading about the enhanced 3-2-1 backup rule which was 3 copies of your data, 2 different media types, 1 offsite. This has been expanded to the 3-2-1-1-0 rule to further enhance your data protection posture. The extra 1 and 0 are 1 copy that is air-gapped, immutable or offline and the 0 is for zero recovery errors. A backup is only as good as its ability to be recovered from, right?
With that in mind, I thought I would explore the Synology storage snapshot feature to try and satisfy the air-gapped, offline criteria. Now arguably this is not an offline copy of the data, however, it is not accessible from the Veeam management console which is where a compromised account could delete backup sets from. It could just be the get out of jail free card you need to recover backup data quickly in the event backup sets have been deleted.
Setup
For this to work with a Synology device, the storage needs to be carved out as a thin iSCSI volume. Thick provisioned storage does not support storage-based snapshots as far as I could tell. I will run through the process of configuring both so you can see the differences with each.
Step 1 – Create a thick iSCSI LUN
First off we need to log onto the NAS and create an iSCSI LUN and Target. This is the storage volume and destination the Veeam repository server will connect to. The thick LUN will be used for Veeam primary backups.
Open iSCSI manager from within DSM.
Synology offers a tool to manage storage for multiple NAS devices from either Windows or a VMware environment. I chose not to download this as I am only managing one device.
We want to create a new LUN.
Define the LUN name, choose where to take the space from. Define the size and choose the space allocation method. Notice that snapshot is not available to tick with the thick provisioned volume.
As an iSCSI target does not exist, the setup wizard asks us to create one.
Give the target a name and modify the IQN if you would like it to be something more easily identifiable from the device that will connect to it. For extra security, I enabled CHAP user name and password option which stops anonymous devices connecting to the LUN.
Confirm the settings.
And now we have a new 10TB LUN ready to go.
Step 2 – Present storage to Windows and Veeam
A point to note with my setup, I am only able to make use of a single 1Gbps network interface on the NAS, so there is no multipathing enabled to the device.
Within Windows, open the iSCSI initiator, browse to the Discovery tab and click on Discover Portal.
Type in the IP address of the NAS device. The dedicated 1Gbps network interface I mentioned before is connected to a dedicated storage subnet. After typing in the IP address, click the Advanced button,
On the Advanced Settings page, choose which NIC on the server to connect from. You do not need to specify CHAP credentials to make the initial discovery connection to the NAS.
Now click back over to the Targets tab, you should see a new LUN waiting to be connected to with the IQN that was specified earlier. Select it and click connect.
Same deal as before with the advanced settings, except this time the CHAP credentials need to be specified before the connection will be allowed to the LUN.
Now open Disk Manager.
Find the new disk in the list of disks, right-click on it and chose properties and make sure it is the Synology device. You do not want to accidentally format the wrong volume!
As this is going to be used for Veeam, I formatted the volume with ReFS to make use of Veeams Fast Clone functionality.
Now into Veeam to add the new backup repository.
This is for all intents and purposes, direct attached storage.
Mounted to a Windows server.
Give the backup repository a name.
Browse to the server where the storage is mounted and chose the drive letter.
Click on advanced to enabled per-vm backup chains.
Like this.
Then pretty much click through until the wizard is complete.
et voilÃ
A quick bit of testing to evaluate data throughput. Seems about right for a 1Gbps network.
Interestingly though, I was seeing pretty much 100% cache hit on the Synology device when writing the backup set. I am going to put that down to a lack of network bandwidth to saturate the cache.
Update 17th May 2021
One of my fellow UK Veeam user Group leaders, James Kilby, pointed out the following for why the cache hit rate is likely 100%.
Step 3 – Create a thin iSCSI LUN
For the thin-provisioned LUN, I will use this as a backup copy target. In reality, you wouldn’t have both the primary backup and backup copy data on the same device, but I have this set up for demonstration purposes. I will skip some of the setup steps I covered above in this section to avoid duplication.
Back to Synology DSM and open the iSCSI manager again.
This time I will create a thin-provisioned LUN. Notice how the bottom three options are now available, with Snapshot being one of them.
We now have a new thin provisioned LUN.
Notice how there is a note to show there is No Scheduled Protection. Let’s fix that.
I have enabled a snapshot schedule to run every day at midnight.
And I want to keep 7 snapshots. This means I should have an additional 7 days of backup copy job retention above what I configure in Veeam.
There is also an option for Synology to coordinate creating an application-consistent snapshot of the data. This is not something I need to use for this setup but could be useful for other applications that work with quisence.
Again a little bit more testing from Veeam. Throughput is a little better with the backup copy job. I am not sure if this is because Synology is doing something clever in the background with hardware-assisted data transfer or just the numbers are skewed, but the figure is faster than a 1Gbps network can provide.
and in this test, we see a lot of cache misses, which is to be expected. The backup data that is being copied should not still reside in the cache tier once it has been written to the device, so it will be pulling it from the spinning disks behind it.
Recovery from backup deletion
So all the pieces are in place, what to do if the worst happens and somehow the backup sets are deleted from the disk?
Let’s simulate this exact scenario and see what happens.
So you have a nice backup set on disk, when suddenly….
Someone deletes them all.
At this point, disable the associated backup jobs.
Log onto the Synology NAS and navigate to Snapshot in the iSCSI settings.
Select the LUN and then choose Snapshot List from the Snapshot drop-down menu.
Choose the snapshot to roll back to and click restore.
If the Veeam repository server is still connected you will receive the following warning. It is very important to disconnect the server from the storage otherwise this doesn’t work. Ask me how I know that 😀
Disconnect the target from the iSCSI Manager in Windows.
Hit the restore button from DSM and then reconnect the storage. With any luck, everything will be back to the way it was previously.
Enable the job and it should be like nothing happened.
What is really cool is how quickly the LUN was rolled back to the snapshot. 1 second!
Conclusion
We have proven that enabling LUN level snapshots can prove to be another effective step to preventing a total data loss scenario. This is not an air gap solution but just another layer of defence against the bad guys. The more layers we deploy, the harder it will be to defeat them all.
If you would like some further reading, check out this blog from Synology about how to protect yourself from ransomware.
Hi, I use to connect Veeam to Synology repository through NFS instead of iSCSI LUN, I have heard that is less susceptible to be attacked by ransomware due to not to be presented as “native” Windows file system. Is it correct?
Hi Albert,
I guess it depends on what you are trying to achieve here. Yes, it is less susceptible in the way that a ransomware attack would not have access to a native Windows volume to perform some kind of crypto locker attack. In much the same way if you utilised an SMB share on a Synology device, it would be less likely to be attacked.
The concern that none of those storage presentation types addresses though, is if a compromised account has access to the Veeam server. There would be nothing stopping some one opening the console or running some PowerShell code to delete the backup files. At least with something like a snapshot on the Synology device you have an option to restore the data assuming that both the Synology and Veeam account are not compromised. Building up those layers of defence.
Cheers,
Ian
is not totally correct what you said.
nfs , is setup with ACL on IP address of what device can access on it. so noone have access to smb/nfs.
second, veeam need to be hardening , no internet , no out lan , just allow update of win , and allow ip of nas.
And 3 , you need to have an entra hardening linux server with chattr +i on backup so noone can erase it. neither veeam, and hardening of linux.
If you use shared folder u will be crypt for sure on windows/nas.
I assume this is in reference to my other comment at the bottom of the blog post. I agree with what you are saying and it does highlight some additional steps you can take to help secure your backup data. I guess what I did not highlight in my response is that the SMB or NFS shares on a NAS are likely using a different set of ACLs to a native Windows share, so could be less susceptible to a crypto attack that uses a compromised active directory account.
This blog post provides an additional strategy by leveraging NAS level volume snapshots to recover data in the event of a crypto type attack. The ACL and management of the volume on the NAS should be completely isolated from the volume presentation to Windows, thus if the attack came about through a compromised account gaining access to a volume an SMB or NFS share, it doesn’t matter, there is another “hidden” copy of the data on the NAS to recover from.