Much VM, Very Secure: Virtualization-Based Security (VBS) is a controversial security feature enabled by default during Windows 11 installation. By turning the OS into a virtual machine running on top of the Hyper-V hypervisor, data protection and integrity are greatly enhanced, though performance is negatively impacted.
Gamers and traditional users are usually advised to disable VBS and Hyper-V-based virtualization to achieve a noticeable performance boost in both gaming and regular software. However, Microsoft is adamant that VBS can greatly improve security in Windows 10/11. The company is now introducing another VBS-based feature called VBS enclaves, which can provide an entirely new way to build applications when data protection is the top priority.
A VBS enclave is a "software-based trusted execution environment (TEE) inside a host application," Microsoft explains. Thanks to Hyper-V, VBS can create an environment with a higher privilege level than the operating system running in a VM on top of the hypervisor. VBS enclaves allow developers to protect specific portions of their applications using Dynamic Link Library (DLL) files that can be loaded by any standard Windows program.
The isolated, privileged virtual environment created by VBS through the Hyper-V hypervisor is known as Virtual Trust Level 1 (VTL1), which Microsoft describes as the "root of trust of the OS." The traditional Windows environment operates at a lower privilege level (VTL0), while VTL1 is further divided into isolated user mode and the secure kernel.
A virtualized Windows installation hosts many of its security features in VTL1, and VBS enclaves can be used to isolate portions of an application in VTL1 as well. Nothing running in VTL0 should be able to access the secure enclave in VTL1, allowing developers to protect "secrets" like passwords, seal data, and perform decryption operations in an isolated, hacker-free environment – at least, in theory.
Creating and using software designed to employ VBS enclaves relies on specific device requirements, including a virtualized Windows installation with the VBS/HVCI feature enabled. Windows 11 or Windows Server 2019 is also required. Developers must use Visual Studio 2022 version 17.9 or later to code their project, and they will need to sign their VBS code with an "enclave certificate" provided by Microsoft.
While offering greatly enhanced security, VBS enclaves are designed to have limited access to Windows APIs. Microsoft opted to provide a limited range of functionality to expose a smaller attack surface to cybercriminals, which should, in turn, make maintaining the integrity of VTL1 easier. Coders should not trust the host while designing their VBS enclaves, as a DLL file can potentially be loaded by any program and not just the "host intended application."