Repo:   https://github.com/lersek/edk2.git
Branch: sev_smram_save_state_map

This patch series replaces the following patch from Brijesh:

  [PATCH v2 1/2] OvmfPkg/AmdSevDxe: Clear the C-bit from SMM Saved State
  http://mid.mail-archive.com/20180228161415.28723-2-brijesh.singh@amd.com
  https://lists.01.org/pipermail/edk2-devel/2018-February/022016.html

The series modifies four modules under OvmfPkg:
- MemEncryptSevLib
- PlatformPei
- SmmCpuFeaturesLib
- AmdSevDxe

Patches 01 through 16 are cleanups for these modules, without change in
functionality. The series is formatted with 20 lines of context, for
simplifying the review of these patches.

Patches 17 through 20 (one patch per module listed above) implement
$SUBJECT:

- MemEncryptSevLib gets a new helper function used by the other three
  modules.

- PlatformPei makes sure that DXE stays out of the page(s) where the
  initial SMRAM save state map lives.

- SmmCpuFeaturesLib and AmdSevDxe basically do what they did in
  Brijesh's patch, just better separated, and with minor tweaks.

The series is bisectable.

I regression-tested my usual non-SEV guests (with Brijesh's v2 2/2 patch
applied on top, from the above-referenced series), which covers i440fx
(no SMM, X64), q35 (SMM, IA32 and IA32X64), Fedora and Windows, normal
boot and S3.

I also tried to test the series with SEV guests (again with Brijesh's v2
2/2 patch applied on top). Unfortunately, I didn't get good results with
or without SMM. Without SMM, the guest OS boots to a point, but then it
gets stuck with the CPU spinning. With SMM, OVMF gets stuck in SMBASE
relocation.

I should mention however that my SEV host setup dates back to November
2017, including host kernel, QEMU and guest OS. I suppose all those
components have seen many changes since, on the respective upstream
lists. I'll have to work with Brijesh to update my SEV host to the
latest bits.

Until then, Brijesh, can you please test this series? Thank you!

Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Brijesh Singh <brijesh.singh@amd.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>

Cheers
Laszlo

Laszlo Ersek (20):
  OvmfPkg/MemEncryptSevLib: rewrap to 79 characters width
  OvmfPkg/MemEncryptSevLib: clean up MemEncryptSevIsEnabled() decl
  OvmfPkg/MemEncryptSevLib: clean up MemEncryptSevClearPageEncMask()
    decl
  OvmfPkg/MemEncryptSevLib: clean up MemEncryptSevSetPageEncMask() decl
  OvmfPkg/MemEncryptSevLib: clean up SetMemoryEncDec() comment block
  OvmfPkg/MemEncryptSevLib: clean up
    InternalMemEncryptSevSetMemoryDecrypted() decl
  OvmfPkg/MemEncryptSevLib: clean up
    InternalMemEncryptSevSetMemoryEncrypted() decl
  OvmfPkg/MemEncryptSevLib: sort #includes, and entries in INF file
    sections
  OvmfPkg/PlatformPei: sort #includes in "AmdSev.c"
  OvmfPkg/SmmCpuFeaturesLib: rewrap to 79 columns
  OvmfPkg/SmmCpuFeaturesLib: upper-case the "static" keyword
  OvmfPkg/SmmCpuFeaturesLib: sort #includes, and entries in INF file
    sections
  OvmfPkg/SmmCpuFeaturesLib: remove unneeded #includes and
    LibraryClasses
  OvmfPkg/AmdSevDxe: rewrap to 79 characters width
  OvmfPkg/AmdSevDxe: sort #includes, and entries in INF file sections
  OvmfPkg/AmdSevDxe: refresh #includes and LibraryClasses
  OvmfPkg/MemEncryptSevLib: find pages of initial SMRAM save state map
  OvmfPkg/PlatformPei: SEV: allocate pages of initial SMRAM save state
    map
  OvmfPkg/SmmCpuFeaturesLib: SEV: encrypt+free pages of init. save state
    map
  OvmfPkg/AmdSevDxe: decrypt the pages of the initial SMRAM save state
    map

 OvmfPkg/AmdSevDxe/AmdSevDxe.c                                   |  88 ++-
 OvmfPkg/AmdSevDxe/AmdSevDxe.inf                                 |  25 +-
 OvmfPkg/Include/Library/MemEncryptSevLib.h                      |  89 ++-
 OvmfPkg/Library/BaseMemEncryptSevLib/BaseMemEncryptSevLib.inf   |  17 +-
 OvmfPkg/Library/BaseMemEncryptSevLib/Ia32/MemEncryptSevLib.c    |  72 ++-
 OvmfPkg/Library/BaseMemEncryptSevLib/MemEncryptSevLibInternal.c |  66 +-
 OvmfPkg/Library/BaseMemEncryptSevLib/X64/MemEncryptSevLib.c     |  74 ++-
 OvmfPkg/Library/BaseMemEncryptSevLib/X64/VirtualMemory.c        | 172 ++++--
 OvmfPkg/Library/BaseMemEncryptSevLib/X64/VirtualMemory.h        | 137 +++--
 OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c           | 650 ++++++++++++++++----
 OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf         |  13 +-
 OvmfPkg/PlatformPei/AmdSev.c                                    |  36 +-
 12 files changed, 1067 insertions(+), 372 deletions(-)

-- 
2.14.1.3.gb7cf6e02401b

_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

On 3/1/18 6:03 PM, Laszlo Ersek wrote:
> Repo:   https://github.com/lersek/edk2.git
> Branch: sev_smram_save_state_map
>
> This patch series replaces the following patch from Brijesh:
>
>   [PATCH v2 1/2] OvmfPkg/AmdSevDxe: Clear the C-bit from SMM Saved State
>   http://mid.mail-archive.com/20180228161415.28723-2-brijesh.singh@amd.com
>   https://lists.01.org/pipermail/edk2-devel/2018-February/022016.html
>
> The series modifies four modules under OvmfPkg:
> - MemEncryptSevLib
> - PlatformPei
> - SmmCpuFeaturesLib
> - AmdSevDxe
>
> Patches 01 through 16 are cleanups for these modules, without change in
> functionality. The series is formatted with 20 lines of context, for
> simplifying the review of these patches.
>
> Patches 17 through 20 (one patch per module listed above) implement
> $SUBJECT:
>
> - MemEncryptSevLib gets a new helper function used by the other three
>   modules.
>
> - PlatformPei makes sure that DXE stays out of the page(s) where the
>   initial SMRAM save state map lives.
>
> - SmmCpuFeaturesLib and AmdSevDxe basically do what they did in
>   Brijesh's patch, just better separated, and with minor tweaks.
>
> The series is bisectable.
>
> I regression-tested my usual non-SEV guests (with Brijesh's v2 2/2 patch
> applied on top, from the above-referenced series), which covers i440fx
> (no SMM, X64), q35 (SMM, IA32 and IA32X64), Fedora and Windows, normal
> boot and S3.
>
> I also tried to test the series with SEV guests (again with Brijesh's v2
> 2/2 patch applied on top). Unfortunately, I didn't get good results with
> or without SMM. Without SMM, the guest OS boots to a point, but then it
> gets stuck with the CPU spinning. With SMM, OVMF gets stuck in SMBASE
> relocation.

To boot the SEV guest with SMM support we need this KVM patch, without
this we will get either #UD or some undefined behavior.

https://git.kernel.org/pub/scm/virt/kvm/kvm.git/commit/?id=7607b7174405aec7441ff6c970833c463114040a

It's strange that you are having trouble booting SEV guest without SMM
support. It's possible that we might have some mismatch kernel kvm +
qemu + ovmf patches.
>
> I should mention however that my SEV host setup dates back to November
> 2017, including host kernel, QEMU and guest OS. I suppose all those
> components have seen many changes since, on the respective upstream
> lists. I'll have to work with Brijesh to update my SEV host to the
> latest bits.
>
> Until then, Brijesh, can you please test this series? Thank you!


Sure, I will try the series tomorrow morning. thank you so much for the
cleanup and remaining SMM work.


>
> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Cc: Brijesh Singh <brijesh.singh@amd.com>
> Cc: Jordan Justen <jordan.l.justen@intel.com>
>
> Cheers
> Laszlo
>
> Laszlo Ersek (20):
>   OvmfPkg/MemEncryptSevLib: rewrap to 79 characters width
>   OvmfPkg/MemEncryptSevLib: clean up MemEncryptSevIsEnabled() decl
>   OvmfPkg/MemEncryptSevLib: clean up MemEncryptSevClearPageEncMask()
>     decl
>   OvmfPkg/MemEncryptSevLib: clean up MemEncryptSevSetPageEncMask() decl
>   OvmfPkg/MemEncryptSevLib: clean up SetMemoryEncDec() comment block
>   OvmfPkg/MemEncryptSevLib: clean up
>     InternalMemEncryptSevSetMemoryDecrypted() decl
>   OvmfPkg/MemEncryptSevLib: clean up
>     InternalMemEncryptSevSetMemoryEncrypted() decl
>   OvmfPkg/MemEncryptSevLib: sort #includes, and entries in INF file
>     sections
>   OvmfPkg/PlatformPei: sort #includes in "AmdSev.c"
>   OvmfPkg/SmmCpuFeaturesLib: rewrap to 79 columns
>   OvmfPkg/SmmCpuFeaturesLib: upper-case the "static" keyword
>   OvmfPkg/SmmCpuFeaturesLib: sort #includes, and entries in INF file
>     sections
>   OvmfPkg/SmmCpuFeaturesLib: remove unneeded #includes and
>     LibraryClasses
>   OvmfPkg/AmdSevDxe: rewrap to 79 characters width
>   OvmfPkg/AmdSevDxe: sort #includes, and entries in INF file sections
>   OvmfPkg/AmdSevDxe: refresh #includes and LibraryClasses
>   OvmfPkg/MemEncryptSevLib: find pages of initial SMRAM save state map
>   OvmfPkg/PlatformPei: SEV: allocate pages of initial SMRAM save state
>     map
>   OvmfPkg/SmmCpuFeaturesLib: SEV: encrypt+free pages of init. save state
>     map
>   OvmfPkg/AmdSevDxe: decrypt the pages of the initial SMRAM save state
>     map
>
>  OvmfPkg/AmdSevDxe/AmdSevDxe.c                                   |  88 ++-
>  OvmfPkg/AmdSevDxe/AmdSevDxe.inf                                 |  25 +-
>  OvmfPkg/Include/Library/MemEncryptSevLib.h                      |  89 ++-
>  OvmfPkg/Library/BaseMemEncryptSevLib/BaseMemEncryptSevLib.inf   |  17 +-
>  OvmfPkg/Library/BaseMemEncryptSevLib/Ia32/MemEncryptSevLib.c    |  72 ++-
>  OvmfPkg/Library/BaseMemEncryptSevLib/MemEncryptSevLibInternal.c |  66 +-
>  OvmfPkg/Library/BaseMemEncryptSevLib/X64/MemEncryptSevLib.c     |  74 ++-
>  OvmfPkg/Library/BaseMemEncryptSevLib/X64/VirtualMemory.c        | 172 ++++--
>  OvmfPkg/Library/BaseMemEncryptSevLib/X64/VirtualMemory.h        | 137 +++--
>  OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c           | 650 ++++++++++++++++----
>  OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf         |  13 +-
>  OvmfPkg/PlatformPei/AmdSev.c                                    |  36 +-
>  12 files changed, 1067 insertions(+), 372 deletions(-)
>

_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

On 03/02/18 02:16, Brijesh Singh wrote:
> 
> 
> On 3/1/18 6:03 PM, Laszlo Ersek wrote:

>> I also tried to test the series with SEV guests (again with Brijesh's v2
>> 2/2 patch applied on top). Unfortunately, I didn't get good results with
>> or without SMM. Without SMM, the guest OS boots to a point, but then it
>> gets stuck with the CPU spinning. With SMM, OVMF gets stuck in SMBASE
>> relocation.
> 
> To boot the SEV guest with SMM support we need this KVM patch, without
> this we will get either #UD or some undefined behavior.
> 
> https://git.kernel.org/pub/scm/virt/kvm/kvm.git/commit/?id=7607b7174405aec7441ff6c970833c463114040a

Looks like a very recent commit. What tree (and at what commit) do you
recommend that I build a new host kernel?

> It's strange that you are having trouble booting SEV guest without SMM
> support. It's possible that we might have some mismatch kernel kvm +
> qemu + ovmf patches.

Wait, the details matter: I wrote "the guest OS boots to a point". There
are no problems with the firmware, or the initial OS boot progress. The
issue happens fairly later (but certainly before I reach a login prompt
or similar). Maybe this is nothing new relative to last November; I
don't remember.

>> Until then, Brijesh, can you please test this series? Thank you!
> 
> 
> Sure, I will try the series tomorrow morning. thank you so much for the
> cleanup and remaining SMM work.

Thanks!

Do you have (maybe updated) instructions for setting up the SEV host?
What are the latest bits that are expected to work together?

Thanks!
Laszlo
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

On 3/2/18 5:53 AM, Laszlo Ersek wrote:
> On 03/02/18 02:16, Brijesh Singh wrote:
>>
>> On 3/1/18 6:03 PM, Laszlo Ersek wrote:
>>> I also tried to test the series with SEV guests (again with Brijesh's v2
>>> 2/2 patch applied on top). Unfortunately, I didn't get good results with
>>> or without SMM. Without SMM, the guest OS boots to a point, but then it
>>> gets stuck with the CPU spinning. With SMM, OVMF gets stuck in SMBASE
>>> relocation.
>> To boot the SEV guest with SMM support we need this KVM patch, without
>> this we will get either #UD or some undefined behavior.
>>
>> https://git.kernel.org/pub/scm/virt/kvm/kvm.git/commit/?id=7607b7174405aec7441ff6c970833c463114040a
> Looks like a very recent commit. What tree (and at what commit) do you
> recommend that I build a new host kernel?

Yes this is very recent commit and it was developed during SMM work. For
host kernel we need at least 4.16.0-rc1 but since you are going to boot
the SMM enabled BIOS hence I recommend using latest kvm/master

https://git.kernel.org/pub/scm/virt/kvm/kvm.git/

>
>> It's strange that you are having trouble booting SEV guest without SMM
>> support. It's possible that we might have some mismatch kernel kvm +
>> qemu + ovmf patches.
> Wait, the details matter: I wrote "the guest OS boots to a point". There
> are no problems with the firmware, or the initial OS boot progress. The
> issue happens fairly later (but certainly before I reach a login prompt
> or similar). Maybe this is nothing new relative to last November; I
> don't remember.

Ah, my best guess is that userspace program is getting wrong time using
clock_gettime() and hence the bootscripts are waiting on some events
forever .. IIRC, I was getting boot hang sometime back in Oct or Nov and
debugging took me to the kvmclock support for SEV guest. I was doing
everything right in my patches for kvmclock except the first hunk of the
below patch. When kvmclock is available the clock_getttime() uses vdso
and since kvmclock page is shared between HV and Guest hence we needed
to ensure that userspace pgtable have proper C-bit when accessing this
memory range.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v4.16-rc3&id=819aeee065e5d1b417ecd633897427c89f3253ec

All the SEV guest got accepted in 4.15, hence for guest kernel you can
use Linux kernel >=4.15

>>> Until then, Brijesh, can you please test this series? Thank you!
>>
>> Sure, I will try the series tomorrow morning. thank you so much for the
>> cleanup and remaining SMM work.
> Thanks!
>
> Do you have (maybe updated) instructions for setting up the SEV host?
> What are the latest bits that are expected to work together?

AMDSEV page https://github.com/AMDESE/AMDSEV contains some instruction
and scripts to boot the SEV guest but its still using the older version
of kernel and qemu. Here is what you need to do:

For host kernel:
- use recent kvm/master
- make sure following kernel config is enabled
  CONFIG_KVM_AMD_SEV
  CONFIG_CRYPTO_DEV_SP_PSP
  CONFIG_AMD_MEM_ENCRYPT
  CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT

For guest kernel:
 - you can use host kernel or anything >=4.15
    make sure you have following config enabled in kernel:
      CONFIG_AMD_MEM_ENCRYPT

For qemu:
- v10 patches from this branch
https://github.com/codomania/qemu/tree/v10


> Thanks!
> Laszlo

_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

On 03/02/18 14:17, Brijesh Singh wrote:
> 
> 
> On 3/2/18 5:53 AM, Laszlo Ersek wrote:
>> On 03/02/18 02:16, Brijesh Singh wrote:
>>>
>>> On 3/1/18 6:03 PM, Laszlo Ersek wrote:
>>>> I also tried to test the series with SEV guests (again with Brijesh's v2
>>>> 2/2 patch applied on top). Unfortunately, I didn't get good results with
>>>> or without SMM. Without SMM, the guest OS boots to a point, but then it
>>>> gets stuck with the CPU spinning. With SMM, OVMF gets stuck in SMBASE
>>>> relocation.
>>> To boot the SEV guest with SMM support we need this KVM patch, without
>>> this we will get either #UD or some undefined behavior.
>>>
>>> https://git.kernel.org/pub/scm/virt/kvm/kvm.git/commit/?id=7607b7174405aec7441ff6c970833c463114040a
>> Looks like a very recent commit. What tree (and at what commit) do you
>> recommend that I build a new host kernel?
> 
> Yes this is very recent commit and it was developed during SMM work. For
> host kernel we need at least 4.16.0-rc1 but since you are going to boot
> the SMM enabled BIOS hence I recommend using latest kvm/master
> 
> https://git.kernel.org/pub/scm/virt/kvm/kvm.git/
> 
>>
>>> It's strange that you are having trouble booting SEV guest without SMM
>>> support. It's possible that we might have some mismatch kernel kvm +
>>> qemu + ovmf patches.
>> Wait, the details matter: I wrote "the guest OS boots to a point". There
>> are no problems with the firmware, or the initial OS boot progress. The
>> issue happens fairly later (but certainly before I reach a login prompt
>> or similar). Maybe this is nothing new relative to last November; I
>> don't remember.
> 
> Ah, my best guess is that userspace program is getting wrong time using
> clock_gettime() and hence the bootscripts are waiting on some events
> forever .. IIRC, I was getting boot hang sometime back in Oct or Nov and
> debugging took me to the kvmclock support for SEV guest. I was doing
> everything right in my patches for kvmclock except the first hunk of the
> below patch. When kvmclock is available the clock_getttime() uses vdso
> and since kvmclock page is shared between HV and Guest hence we needed
> to ensure that userspace pgtable have proper C-bit when accessing this
> memory range.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v4.16-rc3&id=819aeee065e5d1b417ecd633897427c89f3253ec
> 
> All the SEV guest got accepted in 4.15, hence for guest kernel you can
> use Linux kernel >=4.15
> 
>>>> Until then, Brijesh, can you please test this series? Thank you!
>>>
>>> Sure, I will try the series tomorrow morning. thank you so much for the
>>> cleanup and remaining SMM work.
>> Thanks!
>>
>> Do you have (maybe updated) instructions for setting up the SEV host?
>> What are the latest bits that are expected to work together?
> 
> AMDSEV page https://github.com/AMDESE/AMDSEV contains some instruction
> and scripts to boot the SEV guest but its still using the older version
> of kernel and qemu. Here is what you need to do:
> 
> For host kernel:
> - use recent kvm/master
> - make sure following kernel config is enabled
>   CONFIG_KVM_AMD_SEV
>   CONFIG_CRYPTO_DEV_SP_PSP
>   CONFIG_AMD_MEM_ENCRYPT
>   CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT
> 
> For guest kernel:
>  - you can use host kernel or anything >=4.15
>     make sure you have following config enabled in kernel:
>       CONFIG_AMD_MEM_ENCRYPT
> 
> For qemu:
> - v10 patches from this branch
> https://github.com/codomania/qemu/tree/v10

Great, I'll work on updating all the bits and I'll report back.

Thanks!
Laszlo
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

On 03/02/18 14:17, Brijesh Singh wrote:
> On 3/2/18 5:53 AM, Laszlo Ersek wrote:

>> Do you have (maybe updated) instructions for setting up the SEV host?
>> What are the latest bits that are expected to work together?

> For host kernel:
> - use recent kvm/master
> - make sure following kernel config is enabled
>   CONFIG_KVM_AMD_SEV
>   CONFIG_CRYPTO_DEV_SP_PSP
>   CONFIG_AMD_MEM_ENCRYPT
>   CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT
> 
> For guest kernel:
>  - you can use host kernel or anything >=4.15
>     make sure you have following config enabled in kernel:
>       CONFIG_AMD_MEM_ENCRYPT
> 
> For qemu:
> - v10 patches from this branch
> https://github.com/codomania/qemu/tree/v10

QEMU exits with the following error for me:

2018-03-05T13:40:12.478835Z qemu-system-x86_64: sev_ram_block_added: failed to register region (0x7f3df3e00000+0x200000000)
2018-03-05T13:40:12.489183Z qemu-system-x86_64: sev_ram_block_added: failed to register region (0x7f3ffaa00000+0x37c000)
2018-03-05T13:40:12.497580Z qemu-system-x86_64: sev_ram_block_added: failed to register region (0x7f3ffa800000+0x20000)
2018-03-05T13:40:12.504485Z qemu-system-x86_64: sev_launch_update_data: LAUNCH_UPDATE ret=-12 fw_error=0 ''
2018-03-05T13:40:12.504493Z qemu-system-x86_64: failed to encrypt pflash rom

Here's my full QEMU command line (started by libvirt) -- this command line does not restrict pflash access to guest code that runs in SMM, and correspondingly, the OVMF build lacks SMM_REQUIRE:

/opt/qemu-installed/bin/qemu-system-x86_64 \
  -name guest=from-brijesh,debug-threads=on \
  -S \
  -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-from-brijesh/master-key.aes \
  -machine pc-q35-2.10,accel=kvm,usb=off,smm=on,dump-guest-core=off \
  -cpu host \
  -drive file=/home/virt-images/OVMF_CODE.4m.fd,if=pflash,format=raw,unit=0,readonly=on \
  -drive file=/var/lib/libvirt/qemu/nvram/from-brijesh_VARS.fd,if=pflash,format=raw,unit=1 \
  -m 8192 \
  -realtime mlock=off \
  -smp 1,sockets=1,cores=1,threads=1 \
  -uuid e2373f13-f481-4008-88d0-d61fa9da16fe \
  -no-user-config \
  -nodefaults \
  -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-from-brijesh/monitor.sock,server,nowait \
  -mon chardev=charmonitor,id=monitor,mode=control \
  -rtc base=utc \
  -no-shutdown \
  -boot strict=on \
  -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
  -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
  -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
  -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
  -device nec-usb-xhci,id=usb,bus=pci.1,addr=0x0 \
  -device virtio-scsi-pci,iommu_platform=on,ats=on,id=scsi0,bus=pci.3,addr=0x0 \
  -drive file=/var/lib/libvirt/images/rhel-7-server.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=writeback,discard=unmap,werror=enospc \
  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
  -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 \
  -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:65:f7:fb,bus=pci.4,addr=0x0,rombar=0 \
  -chardev pty,id=charserial0 \
  -device isa-serial,chardev=charserial0,id=serial0 \
  -device usb-tablet,id=input2,bus=usb.0,port=1 \
  -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on \
  -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \
  -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x0 \
  -global isa-debugcon.iobase=0x402 \
  -debugcon file:/tmp/from-brijesh.log \
  -fw_cfg name=opt/ovmf/PcdResizeXterm,string=y \
  -s \
  -object sev-guest,id=sev0,policy=0x0,cbitpos=47,reduced-phys-bits=5 \
  -machine memory-encryption=sev0 \
  -msg timestamp=on

Thanks,
Laszlo
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Hi Laszlo,



On 03/05/2018 08:00 AM, Laszlo Ersek wrote:
> On 03/02/18 14:17, Brijesh Singh wrote:
>> On 3/2/18 5:53 AM, Laszlo Ersek wrote:
> 
>>> Do you have (maybe updated) instructions for setting up the SEV host?
>>> What are the latest bits that are expected to work together?
> 
>> For host kernel:
>> - use recent kvm/master
>> - make sure following kernel config is enabled
>>    CONFIG_KVM_AMD_SEV
>>    CONFIG_CRYPTO_DEV_SP_PSP
>>    CONFIG_AMD_MEM_ENCRYPT
>>    CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT
>>
>> For guest kernel:
>>   - you can use host kernel or anything >=4.15
>>      make sure you have following config enabled in kernel:
>>        CONFIG_AMD_MEM_ENCRYPT
>>
>> For qemu:
>> - v10 patches from this branch
>> https://github.com/codomania/qemu/tree/v10
> 
> QEMU exits with the following error for me:
> 
> 2018-03-05T13:40:12.478835Z qemu-system-x86_64: sev_ram_block_added: failed to register region (0x7f3df3e00000+0x200000000)
> 2018-03-05T13:40:12.489183Z qemu-system-x86_64: sev_ram_block_added: failed to register region (0x7f3ffaa00000+0x37c000)
> 2018-03-05T13:40:12.497580Z qemu-system-x86_64: sev_ram_block_added: failed to register region (0x7f3ffa800000+0x20000)
> 2018-03-05T13:40:12.504485Z qemu-system-x86_64: sev_launch_update_data: LAUNCH_UPDATE ret=-12 fw_error=0 ''
> 2018-03-05T13:40:12.504493Z qemu-system-x86_64: failed to encrypt pflash rom
> 
> Here's my full QEMU command line (started by libvirt) -- this command line does not restrict pflash access to guest code that runs in SMM, and correspondingly, the OVMF build lacks SMM_REQUIRE:
> 

Are you launching guest as a normal users or root ? If you are launching 
guest as normal user then please make sure you have increased the 'max 
locked memory' limit. The register region function will try to pin the 
memory, while doing so we check the limit and if requested size is 
greater than ulimit then we fail.


# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 966418
max locked memory       (kbytes, -l) 10240000
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 966418
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

If QEMU command is still failing for you then can you please share your 
kernel dmesg. thanks


> /opt/qemu-installed/bin/qemu-system-x86_64 \
>    -name guest=from-brijesh,debug-threads=on \
>    -S \
>    -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-from-brijesh/master-key.aes \
>    -machine pc-q35-2.10,accel=kvm,usb=off,smm=on,dump-guest-core=off \
>    -cpu host \
>    -drive file=/home/virt-images/OVMF_CODE.4m.fd,if=pflash,format=raw,unit=0,readonly=on \
>    -drive file=/var/lib/libvirt/qemu/nvram/from-brijesh_VARS.fd,if=pflash,format=raw,unit=1 \
>    -m 8192 \
>    -realtime mlock=off \
>    -smp 1,sockets=1,cores=1,threads=1 \
>    -uuid e2373f13-f481-4008-88d0-d61fa9da16fe \
>    -no-user-config \
>    -nodefaults \
>    -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-from-brijesh/monitor.sock,server,nowait \
>    -mon chardev=charmonitor,id=monitor,mode=control \
>    -rtc base=utc \
>    -no-shutdown \
>    -boot strict=on \
>    -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
>    -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
>    -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
>    -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
>    -device nec-usb-xhci,id=usb,bus=pci.1,addr=0x0 \
>    -device virtio-scsi-pci,iommu_platform=on,ats=on,id=scsi0,bus=pci.3,addr=0x0 \
>    -drive file=/var/lib/libvirt/images/rhel-7-server.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=writeback,discard=unmap,werror=enospc \
>    -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
>    -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 \
>    -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:65:f7:fb,bus=pci.4,addr=0x0,rombar=0 \
>    -chardev pty,id=charserial0 \
>    -device isa-serial,chardev=charserial0,id=serial0 \
>    -device usb-tablet,id=input2,bus=usb.0,port=1 \
>    -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on \
>    -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \
>    -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x0 \
>    -global isa-debugcon.iobase=0x402 \
>    -debugcon file:/tmp/from-brijesh.log \
>    -fw_cfg name=opt/ovmf/PcdResizeXterm,string=y \
>    -s \
>    -object sev-guest,id=sev0,policy=0x0,cbitpos=47,reduced-phys-bits=5 \
>    -machine memory-encryption=sev0 \
>    -msg timestamp=on
> 
> Thanks,
> Laszlo
> 
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

One more comment.


On 03/05/2018 08:44 AM, Brijesh Singh wrote: >> \
>>    -device 
>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 
>> \
>>    -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 \
>>    -device 
>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:65:f7:fb,bus=pci.4,addr=0x0,rombar=0 
>> \

Please add iommu_platform=on,ats=on argument in virtio network. In order 
the SEV guest to work with virtio we need to use iommu_platform=on.


>>    -chardev pty,id=charserial0 \
>>    -device isa-serial,chardev=charserial0,id=serial0 \
>>    -device usb-tablet,id=input2,bus=usb.0,port=1 \
>>    -spice 
>> port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on \
>>    -device cirrus-vga,id=video0,bus=pcie.0,addr=0x1 \
>>    -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x0 \
>>    -global isa-debugcon.iobase=0x402 \
>>    -debugcon file:/tmp/from-brijesh.log \
>>    -fw_cfg name=opt/ovmf/PcdResizeXterm,string=y \
>>    -s \
>>    -object sev-guest,id=sev0,policy=0x0,cbitpos=47,reduced-phys-bits=5 \
>>    -machine memory-encryption=sev0 \
>>    -msg timestamp=on
>>
>> Thanks,
>> Laszlo
>>
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

On 03/05/18 15:44, Brijesh Singh wrote:
> On 03/05/2018 08:00 AM, Laszlo Ersek wrote:

>> QEMU exits with the following error for me:
>>
>> 2018-03-05T13:40:12.478835Z qemu-system-x86_64: sev_ram_block_added:
>> failed to register region (0x7f3df3e00000+0x200000000)
>> 2018-03-05T13:40:12.489183Z qemu-system-x86_64: sev_ram_block_added:
>> failed to register region (0x7f3ffaa00000+0x37c000)
>> 2018-03-05T13:40:12.497580Z qemu-system-x86_64: sev_ram_block_added:
>> failed to register region (0x7f3ffa800000+0x20000)
>> 2018-03-05T13:40:12.504485Z qemu-system-x86_64:
>> sev_launch_update_data: LAUNCH_UPDATE ret=-12 fw_error=0 ''
>> 2018-03-05T13:40:12.504493Z qemu-system-x86_64: failed to encrypt
>> pflash rom
>>
>> Here's my full QEMU command line (started by libvirt) -- this command
>> line does not restrict pflash access to guest code that runs in SMM,
>> and correspondingly, the OVMF build lacks SMM_REQUIRE:
>>
> 
> Are you launching guest as a normal users or root ? If you are launching
> guest as normal user then please make sure you have increased the 'max
> locked memory' limit. The register region function will try to pin the
> memory, while doing so we check the limit and if requested size is
> greater than ulimit then we fail.
> 
> 
> # ulimit -a
> core file size          (blocks, -c) unlimited
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 966418
> max locked memory       (kbytes, -l) 10240000
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 966418
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited

Good catch! Libvirtd starts the QEMU process with UID=qemu, but the
restriction doesn't come from there.

Instead, it seems like the systemd default for "max locked memory" is
64KB on RHEL-7 anyway. I raised it by setting

  DefaultLimitMEMLOCK=infinity

in "/etc/systemd/system.conf".

(The documentation is at:
- <https://www.freedesktop.org/software/systemd/man/systemd.exec.html>,
-
<https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html>.)

Following your other email, I've now also added
"iommu_platform=on,ats=on" to virtio-net-pci, not just virtio-scsi-pci.

This got a lot farther: the TianoCore splash screen was dispalyed, but
then it got stuck.

Looking more at the libvirt-generated command line, I figured maybe
"vhost" should be disabled for virtio-net (so that the device
implementation would run from QEMU userspace, not in the host kernel).
Thus, ultimately I added

    <interface type='network'>
      <driver name='qemu' iommu='on' ats='on'/>
              ^^^^^^^^^^^
    </interface>

documented at
<https://libvirt.org/formatdomain.html#elementsDriverBackendOptions>.

With these settings, the guest boots & works fine for me! I tested the
SEV guest with 4 VCPUs, both with and without SMM. (I used the same
kernel in the guest as on the host -- you wrote CONFIG_AMD_MEM_ENCRYPT
for the guest, and the host requirements imply that.)

I'm attaching the full domain XML for reference.

Thanks!
Laszlo
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

On 03/01/2018 06:03 PM, Laszlo Ersek wrote:
...

> 
> I regression-tested my usual non-SEV guests (with Brijesh's v2 2/2 patch
> applied on top, from the above-referenced series), which covers i440fx
> (no SMM, X64), q35 (SMM, IA32 and IA32X64), Fedora and Windows, normal
> boot and S3.
> 
> I also tried to test the series with SEV guests (again with Brijesh's v2
> 2/2 patch applied on top). Unfortunately, I didn't get good results with
> or without SMM. Without SMM, the guest OS boots to a point, but then it
> gets stuck with the CPU spinning. With SMM, OVMF gets stuck in SMBASE
> relocation.
> 
> I should mention however that my SEV host setup dates back to November
> 2017, including host kernel, QEMU and guest OS. I suppose all those
> components have seen many changes since, on the respective upstream
> lists. I'll have to work with Brijesh to update my SEV host to the
> latest bits.
> 
> Until then, Brijesh, can you please test this series? Thank you!
> 


I tested the series on SEV and non SEV guest (with my v2 2/2 patch 
applied on top of this series). The result is positive,  I am able boot 
both SMM and non SMM enabled BIOS.

Once again, thank you so much for cleanup.

I will submit by v2 2/2 patch after your series is pushed to the tree.

Tested-by: Brijesh Singh <brijesh.singh@amd.com>
Reviewed-by: Brijesh Singh <brijesh.singh@amd.com>


-Brijesh
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

On 03/02/18 16:21, Brijesh Singh wrote:
> 
> 
> On 03/01/2018 06:03 PM, Laszlo Ersek wrote:
> ...
> 
>>
>> I regression-tested my usual non-SEV guests (with Brijesh's v2 2/2 patch
>> applied on top, from the above-referenced series), which covers i440fx
>> (no SMM, X64), q35 (SMM, IA32 and IA32X64), Fedora and Windows, normal
>> boot and S3.
>>
>> I also tried to test the series with SEV guests (again with Brijesh's v2
>> 2/2 patch applied on top). Unfortunately, I didn't get good results with
>> or without SMM. Without SMM, the guest OS boots to a point, but then it
>> gets stuck with the CPU spinning. With SMM, OVMF gets stuck in SMBASE
>> relocation.
>>
>> I should mention however that my SEV host setup dates back to November
>> 2017, including host kernel, QEMU and guest OS. I suppose all those
>> components have seen many changes since, on the respective upstream
>> lists. I'll have to work with Brijesh to update my SEV host to the
>> latest bits.
>>
>> Until then, Brijesh, can you please test this series? Thank you!
>>
> 
> 
> I tested the series on SEV and non SEV guest (with my v2 2/2 patch
> applied on top of this series). The result is positive,  I am able boot
> both SMM and non SMM enabled BIOS.
> 
> Once again, thank you so much for cleanup.
> 
> I will submit by v2 2/2 patch after your series is pushed to the tree.
> 
> Tested-by: Brijesh Singh <brijesh.singh@amd.com>
> Reviewed-by: Brijesh Singh <brijesh.singh@amd.com>

Series pushed: 66f2329446db..5e2e5647b9fb.

Thanks!
Laszlo
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel