Re: CURRENT: FreeBSD not reporting AES-NI on Intel(R) Xeon(R) CPU E5-1650 v3

On Fri, Mar 17, 2017 at 1:31 PM, O. Hartmann <ohartmann_at_walstatt.org> wrote:

> Am Fri, 17 Mar 2017 20:07:35 +0300
> Slawa Olhovchenkov <slw_at_zxy.spb.ru> schrieb:
>
> > On Fri, Mar 17, 2017 at 05:53:24PM +0100, O. Hartmann wrote:
> >
> > > Am Fri, 17 Mar 2017 15:04:29 +0300
> > > Slawa Olhovchenkov <slw_at_zxy.spb.ru> schrieb:
> > >
> > > > On Fri, Mar 17, 2017 at 12:36:25PM +0100, O. Hartmann wrote:
> > > >
> > > > > Running recent CURRENT on a Fujitsu Celsius M740 equipted with an
> Intel(R)
> > > > > Xeon(R) CPU E5-1650 v3 _at_ 3.50GHz CPU makes me some trouble.
> > > > >
> > > > > FreeBSD does not report the existence or availability of AES-NI
> feature, which
> > > > > is supposed to be a feature of this type of CPU:
> > > >
> > > > What reassons to detect AES-NI by FreeBSD?
> > >
> > > What do you mean? I do not understand! FreeBSD is supposed to read the
> CPUID and
> > > therefore the capabilities as every other OS, too. But there may some
> circumstances
> > > why FBSD won't. I do not know, that is the reason why I'm asking here.
> >
> > This sample can have disabled AES-NI by vendor, in BIOS, for example.
> > As I show by links this is posible.
> >
> > CPUID in you example don't show AES-NI capabilities, for example
> > 1650v4 w/ AES-NI
> >
> > CPU: Intel(R) Xeon(R) CPU E5-1650 v4 _at_ 3.60GHz (3600.07-MHz K8-class CPU)
> >   Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
> >   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,
> APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,
> MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> >   Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,
> VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,
> SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,
> OSXSAVE,AVX,F16C,RDRAND>
> >
>
>   ^^^^^^
> >   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
> >   AMD Features2=0x121<LAHF,ABM,Prefetch>
> >   Structured Extended
> > Features=0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,
> BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE>
> > XSAVE Features=0x1<XSAVEOPT> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,
> VID,PostIntr
> >   TSC: P-state invariant, performance statistics
> >
> > In you sample: "TSCDLT,XSAVE"
> >
> > May be AES-NI disabled by vendor and FreeBSD correct show this. Or some
> bug in FreeBSD,
> > AES-NI work and other OS show AES-NI capabilities.
> >
> > _______________________________________________
> > freebsd-current_at_freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_
> freebsd.org"
>
> We have some LGA1151 XEON based 19 inch rack server, also equipted with
> Haswell
> E3-12XX-v3 XEONs and FreeBSD, also CURRENT, does show AES-NI.
>
> You're right, the vendor could have disabled AES-NI by intention - but
> they offered this
> box especially with AES-NI capabilities.
>
> See here:
>
>   http://freebsd.1045724.x6.nabble.com/r285947-broken-
> AESNI-support-No-aesni0-on-Intel-XEON-E5-1650-v3-on-Fujitsu-Celsius-M740-
> td6028895.html
>
> I feel a bit pissed off right now due to Fujitsu, because we started
> testing some
> encrypting features and I'd like to use AES-NI and I run into this issue
> again.
>
> I need to know that FreeBSD is not the issue with this specific CPU type.
> I'm still
> frustrated by that stupid comment "UNIX is not supoorted" I got that time
> then when I
> reported 2015 the issue to Fujitsu.
>
>
>
It's pretty straightforward to gain confidence that FreeBSD is not the
issue here.  The 'Features2=' line is printed by printcpuinfo() in
sys/x86/x86/identcpu.c based on the bits set in a variable called
cpu_feature2 (the printf is currently at line 802).  The value of
cpu_feature2 is set in identify_cpu() identcpu.c (for amd64, currently at
line 1401) based on the result of the cpuid instruction that is executed by
a call to do_cpuid(), which itself resides in sys/amd64/include/cpufunc.h.
In other words, a single asm instruction is executed and the set bits from
the result are printed.

Based on some poking around in open source bits (tianocore, coreboot), it
appears that AES-NI is something the BIOS can irreversibly
disable-until-next-reset by twiddling bits in the appropriate MSR
register.  There is no code that does this in FreeBSD on purpose, so there
would have to be a bug introduced in -CURRENT that somehow clobbers those
MSR bits early on - a bug that was also not merged to 11-STABLE (since
Slawa shows AESNI enabled on the same processor under 11-STABLE).

I will also say that I have dealt with a manufacturer of Xeon hardware in
Europe who will not provide a stock BIOS that allows you to enable AES-NI,
out of concerns over violating export/import rules governing encryption
technology.  With that vendor, you have to pass an end-user verification
and then they will make you a custom BIOS that gives you the option to
enable AES-NI.  It took quite some time working through the outer layers of
their support organization to even determine that this was the underlying
(bureaucratic) issue.

Here is another data point: 11.0-RELEASE-p1 running on E5-1650 v3 with
AESNI recognized as enabled.  You could install that stock FreeBSD release
and if it does not show AESNI as enabled on your system, you will clearly
know it is not a FreeBSD problem without any question as to whether there
is a bug in CURRENT or having to build and install the exact rev of
11-STABLE that Slawa is using.

uname:
FreeBSD ttest2 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep
29 01:43:23 UTC 2016
root_at_releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC
 amd64

dmesg:
CPU: Intel(R) Xeon(R) CPU E5-1650 v3 _at_ 3.50GHz (3500.08-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>


-Patrick