This post has been republished via RSS; it originally appeared at: ITOps Talk Blog articles.
Recently I was involved in a troubleshooting on a SQL Always-On failover cluster on a Windows Server 2012R2, and let me say… ”A SPECIAL ALWAYS ON CLUSTER!”
PROBLEM DESCRIPTION
The customer has a branch office in a remote site, with a physical SQL Always On Cluster.
He wants to move this physical cluster from the remote site to the central site but on a different hardware.
This can be achieved with third party tools and data replication. 
When the customer try to start the SQL Always On cluster, on the central site, he receive this strange error:
==========================================================================
Log Name: System
Source: Service Control Manager
Date: 29.1.2018 18:24:28
Event ID: 7001
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: SERVERNAME.Contoso.com
Description:
The Cluster Service service depends on the Csv File System Driver service which failed to start because of the following error:
The system cannot find the file specified.
==========================================================================
TROUBLESHOOTING
For sure we know that, the cluster service have a dependency on the “CSV File System driver”:
So probably I need to investigate on the disk part? But it is strange I don’t need CSV disk on the SQL Always On….
Fortunately the cluster log help me to identify the root cause, by using this command:
Get-ClusterLog -Destination C:\temp\Clusterlog -UseLocalTime -TimeSpan 10
and here you can see the relevant part of the cluster log:
============================================================
00002de8.00002d34::2018/01/29-12:28:27.589 INFO -----------------------------+ LOG BEGIN +----------------------------
00002de8.00002d34::2018/01/29-12:28:27.589 INFO [CS] Starting clussvc as a service
00002de8.00002d34::2018/01/29-12:28:27.589 INFO [CS] cluster service logging level is 5
00002de8.00001038::2018/01/29-12:28:27.589 INFO [CS] Creating cluster node <vector len='1'>
00002de8.00001038::2018/01/29-12:28:27.589 INFO <item>ClusSvc</item>
00002de8.00001038::2018/01/29-12:28:27.589 INFO </vector>
00002de8.000016f8::2018/01/29-12:28:27.589 INFO [StartupConfig]: Initializing.
00002de8.000016f8::2018/01/29-12:28:27.589 INFO [StartupConfig]: Local node name: BANZAI-NODE1.
00002de8.000016f8::2018/01/29-12:28:27.589 INFO [StartupConfig]: Initializing NetFT adapter management information.
00002de8.000016f8::2018/01/29-12:28:27.605 ERR [CS] Service CreateNodeThread Failed, (4319)' because of 'Network interface for NetFT adapter not found.'
…………………….
===========================================================
From the cluster log, we can see that the cluster Service was not able to find the NetFT Adapter (alias the “Microsoft Failover cluster Virtual Adapter”), you can see this adapter from the devmgmt.msc by selecting the “Show hidden devices” option:
In my case from the list of network adapters I wasn’t able to find this network card, so probably something went wrong with the procedure to move the cluster from the branch site to the central site or…..someone removed it, honestly I don’t believe in ghosts yet :).
RESOLUTION
To resolve this issue I have tried to install manually the “Microsoft Failover cluster Virtual Adapter” by using an old and legacy procedure that I didn’t used for a long time:
After this simple procedure, the cluster was able to start.
TO BE HONEST
I have tried to reproduce the same exact issue on my lab, to try to identify the correlation between the CSV File System error event and the “Microsoft Failover cluster Virtual Adapter”, but I haven’t been able to reproduce it…..I have always received this useful error message:
============================================================
Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 1/02/2018 13:51:38
Event ID: 1289
Task Category: Cluster Virtual Adapter
Level: Critical
Keywords:
User: SYSTEM
Computer: BANZAI-NODE1.contoso.com
Description:
The Cluster Service was unable to access network adapter 'Microsoft Failover Cluster Virtual Miniport'. Verify that other network adapters are functioning properly and check the device manager for errors associated with adapter 'Microsoft Failover Cluster Virtual Miniport'. If the configuration for adapter 'Microsoft Failover Cluster Virtual Miniport' has been changed, it may become necessary to reinstall the failover clustering feature on this computer.
============================================================
I think that, this is due to some updates present on my lab, but not present on the customer environment.