One of the projects I've been recently involved in is a EA to CSP subscriptions migration for a client and those who have been a part of such a project will know that there are two methods for the migration. One is a manual subscription to subscription migration, which requires quite a bit of planning before execution, and the other is a billing transfer that is available to Partners who are Azure Expert MSPs. I've been fortunate enough to have access to the Azure Expert MSPs method (https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/mpa-request-ownership#request-billing-ownership), which eliminates majority of the heavy lifting. With that said, I did have to perform a manual migration a year ago so I wanted to share the assessment, planning and migration tips I have in hopes of providing help to anyone have to go through this exercise.
Useful documentation, blog posts and forums for subscription migration information
Let me begin by providing the following list of links that I found very useful when planning for a subscription to subscription migration through the use moving resources between EA to CSP and Pay-as-you-go to CSP.
Microsoft Documentation:
Move resources to a new resource group or subscription - most important to documentation to read to start the migration planning:
https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/move-resource-group-and-subscription
Move operation support for resources - use this to determine whether a resource can be migrated between subscriptions by obtaining its type and reviewing the information provided by the table (e.g. Microsoft.ClassicCompute/domainnames):
https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/move-support-resources
How to move a Recovery Services vault across Azure Subscriptions and Resource Groups:
https://docs.microsoft.com/en-gb/azure/backup/backup-azure-move-recovery-services-vault
Move guidance for virtual machines:
https://docs.microsoft.com/en-gb/azure/azure-resource-manager/management/move-limitations/virtual-machines-move-limitations
Blogs and GitHub:
A must read blog post by Jack Tracey, a Microsoft Cloud Solutions Architect, that provides many important details about manually migrating resources as well as the Azure Expert MSP billing transfer:
https://jacktracey.co.uk/migration/azure-subscription-migrations/
Igor Shastitko wrote a LinkedIn post outlining his experience, items to lookout for, and a high level plan:
https://www.linkedin.com/pulse/azure-paygea-csp-subscriptions-migrations-limitations-igor-shastitko/
Morten Pedholt provides his experience, migration planning tips and how to call the validation API to determine whether resources can be migrated from one subscription to another:
https://pedholtlab.com/migrate-between-azure-subscriptions-like-a-pro/
Pantelis Apostolidis provides a way to use Postman to run the resource migration validation:
https://www.cloudcorner.gr/microsoft/azure/validate-azure-resource-move-with-postman/
Jeroen shares his experience with subscription to subscription migration:
http://www.jeroenwint.com/2017/08/09/lessons-learned-migrate-between-azure-ea-and-azure-csp-subscription/
How to migrate Azure IaaS VMs if they are not encrypted with SSE with PMK with Azure Key Vault:
https://github.com/MicrosoftDocs/azure-docs/issues/8684#issuecomment-400024929
How to migrate application gateways:
https://github.com/MicrosoftDocs/azure-docs/issues/8037
PowerShell script to validate whether a resource can be moved across subscriptions:
https://www.powershellbros.com/check-possibility-of-azure-resource-migration/
Azure Resource Mover for moving resources:
https://www.red-gate.com/simple-talk/cloud/infrastructure-as-a-service/lets-move-azure-resource-mover/
While not related to migrating resources from one subscriptions to another, Wesley Haakman outlines the process of transfer the billing ownership of an Azure subscription from an Enterprise Agreement to a CSP:
https://www.wesleyhaakman.org/transferring-ea-subscriptions-to-csp/
Export subscription resources and review items for migration
Begin by downloading Jack Tracey's Azure Resource Migration Support Tool spreadsheet from his blog: AzureResourceMigrationSupportV10.xlsx
https://jacktracey.co.uk/migration/azure-subscription-migrations/
The Introduction tab of the spreadsheet will provide a detailed explanation of the spreadsheet's features and functionality. Before proceeding to populate the spreadsheet with data, it would be best to update it because the spreadsheet available on Jack Tracey's blog was last updated on October 2020 and Microsoft continues to make new improvements to the subscription migration feature so he has provided links to github where CSVs containing an updated list of whether resources can be migrated. Proceed to download the latest move-support-resources.csv CSV file:
Then copy it into the move-support-resources tab in the AzureResourceMigrationSupportV10.xlsx spreadsheet:
With the spreadsheet updated, proceed to the Introduction tab, which will contain the following PowerShell cmdlet to export the Name, ResourceGroupName, Location and Type of every resource in the subscription into a CSV file:
Connect-AzAccount
Get-AzSubscription -SubscriptionId xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx | Set-AzContext
Get-AzResource | Select Name,ResourceGroupName,Location,Type | Export-CSV SubscriptionXXXXExport.csv -NoTypeInformation
Another method of obtaining the resources export is to use Azure Resource Graph Explorer with the following query:
resources
| project name, resourceGroup, location, type
| order by type asc
With the CSV file of the resources created, paste the resources into the first 4 columns (Name, Resource Group, Region, Resource Type) SubsX tab, which will then automatically fill in the remaining 4 columns (Resource Group Migration Supported? Subscription Migration Supported? Resource Group Migration Limitations, Subscription Migration Limitations) that provide information about the ability to migrate the resources along with the migration limitations of each resource.
This spreadsheet provides a great overview of the resources in the subscription and identify what can be migrated and what cannot. It is important to note that the spreadsheet will place a Yes for classic resources even though it is not supported in a CSP migration so don't make the mistake of only displaying the items with No for the Subscription Migration Supported? column and assume the rest can be migrated:
Not supported in CSP - Must migrate to ARM model first.
Target subscription cannot contain any other classic resources.
All classic resources must be moved in one go.
Organization Subscriptions into Groups
The next step is to use the spreadsheet to separate subscriptions into different groups. The following are the 3 groups I use but it may not work for everyone so create as many or as little:
1. All resources can be migrated
2. All resources can be migrated but there are classic resources
3. Not all resources can be migrated
Group #1 - All resources can be migrated
Resources that can be migrated means that they can be moved without any service disruption to the resource during the move. However, subscriptions where all of the resources can be migrated does not mean you can navigate into portal.azure.com, click on All resources, select all the resources and move them. Even if a resource can be migrated cross subscriptions, it could have dependencies, which would require it to be moved with other resources or be placed in the resource group that it was originally created. It is also likely that some of the dependencies would mean downtime will be required. The following are the steps I perform for this group:
1. Organize all of the resource types in a spreadsheet and sort them by resource type
2. Log into portal.azure.com, launch Resource Explorer, navigate to Subscriptions > "Subscription Name" > Providers:
3. Have the Resource Explorer and spreadsheet side by side, analyze each type of resources to determine what the dependencies are. The reason why I like to use the Resource Explorer is because you can easily skip through the resource types that have no dependencies and click through each resource that do and use the Open blade button to navigate directly to the properties of the resource for further investigation
4. Create an action and notes column to the resources and fill in what actions need to be taken before moving them.
Migration Restrictions
The following are some of the restrictions that are likely to be encountered even if all the resources can be moved:
1. You can only move multiple resources if they are stored in the same resource group. It is not possible to select resources from multiple resource groups and move them.
2. When you move a virtual machine, dependent components such as the VNet the NIC is attached to also needs to be moved, which means if the VNet has a peering then it needs to be removed thus potentially causing downtime.
3. You cannot move a virtual machine and the associated VNet without moving any other virtual machines that are attached to the same VNet.
4. Azure App Services must be placed to original Resource Groups where they were created. If you are unsure where the App Service was originally created, you can use the following cmdlet to retrieve the serverFarmID that contains path of the original Resource Group:
$checkapp = Get-AzResource -ResourceGroupName <current_apps_RG_name> -Name <app_name> -ResourceType "Microsoft.Web/sites"
$checkapp.Properties.serverFarmId
5. Any IaaS VMs with backups configured need to have them disabled and with restore collection points deleted. Note that the deleting the restore collection points does not mean the data will be deleted. Use the PowerShell cmdlet in the following document to bulk delete the restore points: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/move-limitations/virtual-machines-move-limitations#powershell
6. You cannot move a virtual machine without the public IP address that is attached to it. Note that it is possible to move Basic SKU public IP addresses but not Standard SKU public IP addresses.
7. You cannot move a VNet that is configured with an App Service with VNet integration.
8. When you move an App Service, you need to move its App Service Plan, which means you need to move all Apps in the same Apps Service Plan.
9. The destination subscription's resource group for an App Service to be moved cannot contain existing App Services.
10. When moving App Services or Functions, you cannot move them without moving other App Services or Functions in the same Resource Group.
11. You cannot move an App Service that has an uploaded but unbinded SSL certificate to another subscription. In order to move the App Service, we will need to move the certificate as well but the certificate object is not displayed in the resource group because it is a hidden resource so use the enable the Show hidden types option to select the certificate.
12. The increased quotas limits on the source subscriptions will need to be requested in the target
There are too many dependencies for me to list but careful analyzation of the resources should reveal all of them.
Group #2 - All resources can be moved but there are classic resources
The second group would contain resources that can be moved but with classic resources that need to be upgraded to ARM before they can be moved. I won't go into the details of upgrading classic resources to ARM as there are plenty of documentation on how this can be done but one of the scenarios I've come across is where a client had Cloud Apps in their subscription but it hosted an application that had been upgraded for other divisions. Rather than attempting to upgrade the Cloud Apps to ARM, we simply planned for deploying the newer version of the application in parallel then migrated over.
Group #3 - Not all resources can be moved
The last group, which likely would apply to most production environments, would contain resources that cannot be moved. The components I've typically come across are ones with the following resources:
1. Load Balancers with a Standard SKU
2. Public IP address with a Standard SKU
3. Azure Front Door
4. Private Endpoints
5. Azure Key Vaults with CMK and ADE
6. Azure SQL Managed Instances
7. ExpressRoute
8. Azure Monitor and Alerts
9. Subscriptions with different tenants
10. MarketPlace items with a plan (e.g. F5, CheckPoint, Fortinet, Citrix)
Every non-migratable components will need to be planned around and the following are a few options to provide ideas:
Load Balancers - Basic SKU Load Balancers can be migrated but no production environment would use basic SKU LBs. What I've typically done is deploy these resources in parallel, migrate one of the two, three or more backend servers to the destination subscription, add it to the new LB backend pool, make the new LB live and then migrate the remaining servers over. This can significantly reduce downtime.
Public IP Address - IPs with Standard SKUs cannot be migrated can have a new Public IP address configured in the destination subscription that is ready to be applied to migrated resources. Other alternatives could be to create a basic SKU IP address, assign it to the resource and move them together if the resource can be moved. More options are available for clusters so be creative.
Azure Front Door - These can be deployed in parallel and in the new subscription, and point them to, say, the App Services in the old subscriptions via public URL and migrate App Service over live.
Private Endpoints - These require more labour as you'll need to change all of them to use Service EndPoints, migrate the resources and then recreate them.
Azure Key Vaults - These can be a pain to migrate if the environment uses CMK and ADE as you'll need to decrypt the resources prior to moving them.
Azure SQL Managed Instances - These managed instances can be a headache as they're expensive but building new instances in parallel is likely the way to go for most cases.
ExpressRoute - These require a lot of planning as it is best to deploy in parallel but aside from the cost, these links can serve many services' ingress and egress traffic to on-premises datacenters or offices. Network address planning will also be required.
Azure Monitor and Alerts - Some of these cannot be migrated and therefore will need to be recreated.
Subscriptions with different tenants - the source and destination subscriptions need to be associated to the same tenant so if the destination a transfer may be required if they are not the same. Note that when you transfer a subscription to a different Azure AD directory, some resources are not transferred to the target directory. For example, all role assignments and custom roles in Azure role-based access control (Azure RBAC) are permanently deleted from the source directory and are not be transferred to the target directory.
Marketplace Items - These items can be laborious as you'll need to identify them in the current subscription and ensure they are also offered in the CSP subscription. The native Azure tools I've had luck with migrating these items are:
- Disk snapshots: create snapshots of VMs, migrate snapshots to the new subscription, then a new VM with the same Marketplace image, and finally swap the disks with the migrated disks.
- Azure Site Recovery: I generally prefer this method if possible as it requires much less manual labour than snapshots.
- Note that there are many instances where a migrated Marketplace resources such as firewalls will require the license to be reapplied. Some WAFs such as Citrix NetScalers have their licenses bounded to the NIC's MAC so if that changes then a new license will need to get reallocated.
How well you plan around all the resources whether they can be migrated or not will dictate how successful the migration can be.
Clean up and delete unused resources
I’ve found that most of the environments I’ve worked in almost always have resources that are either orphaned or no longer needed and therefore can be deleted. Cleaning up resources and RGs will save a lot of time for the next step so delete any resources that are identified to not be used. Azure Resource Graph Explorer can help with identifying resources such as unattached disks and the following is a query that can be used across all subscriptions:
resources
| where type =~ 'Microsoft.Compute/disks'
| where properties.diskState =~ 'Unattached'
| project name, resourceGroup, subscriptionId, location, tenantId
Grouping resources into resource groups
Once all resources have been reviewed, the next step is to start organizing them into resource groups that will be migrated. Resources that need to be migrated together should be grouped into the same RG so all of them can be migrated in one move. Also don't forget that there are limitations on what the destination RG can have depending on what the source resources are being moved into it (e.g. you can't move App Service resources into a RG that already have App Services).
Test Migration of Resources
At the time of this writing (June 19, 2021), the migration feature within the Azure Portal randomly presents two methods where one will instantly move the resources selected if validation succeeds, while the other will halt and wait for the administrator to interactively click on the button to proceed. This will likely change in the future but if it hasn't, use the following two resources I mentioned earlier to test the migration:
Morten Pedholt provides his experience, migration planning tips and how to call the validation API to determine whether resources can be migrated from one subscription to another:
https://pedholtlab.com/migrate-between-azure-subscriptions-like-a-pro/
Pantelis Apostolidis provides a way to use Postman to run the resource migration validation:
https://www.cloudcorner.gr/microsoft/azure/validate-azure-resource-move-with-postman/
It is best to also create a test environment with two subscriptions so you can test the migration of resources in the environment. I’ve performed some tests for moving resources in the following two blog posts:
Moving an Azure virtual machine and the Recovery Services Vault with its backups from one subscription to another
http://terenceluk.blogspot.com/2021/04/moving-azure-virtual-machine-and.html
Moving Azure App Service resources from one subscription to another subscription
http://terenceluk.blogspot.com/2021/04/moving-app-service-resources-from-one.html
Identify any dependencies on the resource ID
One of the common items that can get missed with subscription migrations are any applications that integrate with Azure and reference resource IDs or subscription IDs. Given how the source and destination subscriptions are different, the resource ID of all resources will change.
Another common consideration I come across are any integration with the EA portal (ea.azure.com) for billing information as going to CSP means the EA portal will no longer provide subscription information.
Using Azure Resource Graph Explorer to help with migration planning
I've found that one of the best tools available for gathering information is Azure Resource Graph Explorer as you can query and filter results as you please.
List subscriptions with Classic Resources
You can quickly export a list of Classic Resources in one or all subscriptions that need to be upgraded/migrated to ARM with the following query:
Resources
| where type contains 'Microsoft.Classic'
| order by type asc
Look for Marketplace solutions
You can use the following query to query for marketplace solutions with plans:
resources
| where isnotnull(plan)
| sort by (type) asc
List non-Microsoft native resources
The following query can be used to look up non-Microsoft Azure native resources such as SendGrid, which would need to be recreated in the target subscription:
resources
| where type notcontains ('microsoft.')
| sort by (type) asc
List Load Balancers across subscriptions
The following query will assist with obtaining a list of load balancers across subscriptions:
resources
| where type contains 'Microsoft.Network/Loadbalancers'
| order by type asc
Retrieve list of VM disks configured with SSE with CMK
Obtaining a list of virtual machine disks configured with SSE with CMK Encryption:
resources
| where type =~ 'Microsoft.Compute/disks'
| where properties.encryption.type =~ 'EncryptionAtRestWithCustomerKey'
Retrieve list of VMs with disks configured with ADE
Obtain a list of virtual machines disks with Azure Disk Encryption (ADE) enabled, which would need to be disabled before migration:
resources
| where type =~ 'Microsoft.Compute/disks'
| where properties.encryptionSettingsCollection.enabled =~ 'true'
I've been very excited ever since Azure Resource Graph Explorer became available as it made viewing, filtering, searching for resources so convenient and it has proven to be a very useful tool for analyzing subscriptions. Refer to the following documentation for more information:
Starter Resource Graph query samples
https://docs.microsoft.com/en-us/azure/governance/resource-graph/samples/starter?tabs=azure-cli
Post Migration Tasks
The post migration tasks to consider are as follows:
1. Allow customer to view charges as pay-as-you-go pricing (configured in the portal.azure.com portal under the Policies settings)
2. Allow customer to request Reserved Instances (configured in the Partner Central portal)
3. Grant the Foreign Principal representing the CSP provider (IBM, TechData, etc) to the migrated subscriptions so tickets can be opened by the CSP (this is specific to subscriptions that were migrated with the MSP Expert tool)
Granting a CSP Foreign Principal the Reader or Owner role onto an Azure Subscription with PowerShell
http://terenceluk.blogspot.com/2021/09/granting-csp-foreign-principal-reader.html
----------------------------------------------------------------------------------------------------------------
I'm sure there are more considerations that I may not have covered but I hope I can contribute to the limited information around planning available for this type of project.
No comments:
Post a Comment