I was doing a POC of SQL Server 2019 Big Data Cluster (BDC) recently. We did the POC in Azure as they didn’t have a Kubernetes environment on-prem that we could work with. Everything they have in Azure is PaaS we the Kubernetes environment (and the vNet that it requires) were brand new. I set up the BDC using Azure Data Studio and everything was working fine. We went to set up the site to site VPN and found that the IP range that was used by default to set up the Kubernetes environment in Azure was already in use on their internal network (bad Microsoft, bad).
No problem, we just added in the correct IP range to the possible addresses for the vNet, added a new Subnet and moved the VMs over to the new subnet (which caused the VMs to reboot, but that was expected).
It turns on that BDC in SQL Server 2019 doesn’t like having the IPs changed for the aks nodes. The problem stems from the fact that BDC is generating its certificates off of the IP address of the node, so if the IP address of the node changes (even if you are using DHCP for on-prem nodes and DHCP gives you a new IP address) your BDC won’t respond.
There are three possible solutions at this point.
- Change the IPs back to what they were before.
- Rebuilt the certificates that BDC uses based on the new IP Addresses
- Toss the BDC and build a new one
Option 1 wasn’t going to work as the old IPs were a conflict with on-prem
Option 2 isn’t great because this is a totally undocumented process to build the new certificates. So you have to figure out how to do this on your own. Given that our clients pay us by the hour, this is not really good use of their money or my time (mostly there money).
Option 3 was a really good option in this case, as there was no data in the environment. Just to be safe I tossed the entire Kubernetes environment and started from scratch.
The moral of this story is going to be, make sure you never need to change the IP addresses of the Kubernetes (or aks or k8s) that hosts your SQL 2019 Big Data Cluster after the fact. Doing so will make your Big Data Cluster unusable; so when it comes to changing the IP address of your container environment and your running SQL Server 2019 Big Data Cluster, plan ahead.
SQL Server Big Data Cluster requires a good amount of planning in order to have a successful deployment. The team at Denny Cherry & Associates Consulting would love to help you plan and deploy SQL Server Big Data Cluster with you. Contact the team and we’ll get the process started.