Re: OpenSSL breaks factor(6)

On Sat, Dec 28, 2019 at 09:34:52PM -0800, Rodney W. Grimes wrote:
> > On Sat, Dec 28, 2019 at 10:46:52PM -0500, Garance A Drosehn wrote:
> > > On 27 Dec 2019, at 17:42, Steve Kargl wrote:
> > > >
> > > > This patch now includes a fix for hexadecimal conversion.  It
> > > > simple scans the string for a hex digit in [a,...,f] and assumes
> > > > that a hexadecimal string has been entered.  A string that includes
> > > > character from the decimal digits is assumed to by a decimal
> > > > representation.
> > > 
> > > What if the user wants to factor a hexadecimal value which does not
> > > happen to include [a...f]?
> > > 
> > > How about recognizing a prefix of '0x' as a way to indicate the value
> > > is hexadecimal?
> > > 
> > 
> > An interested user will need to add that support.  AFAIK, factor(6)
> > has never recognized the 0x prefix, and I'm not trying to add new 
> > features.  I'm simply fixing factor(6) to match its documentation, 
> > and trying to ensure WITH_OPENSSL and WITHOUT_OPENSSL give the 
> > same results where applicable.
> > 
> > The logic is to first try to convert the string to a decimal if
> > the leading digits are members of the set [0,...,9].  If this
> > conversion fails, then try to convert the string as a hexadecimal
> > number.  A problem occurs because OpenSSL's BN_dec2bn does not fail
> > for a number like '1abc' (converts it to 1) whereas the local
> > implementation of BN_dec2bn fails during the conversion, and so
> > the BN_hex2bn function is executed and '1abc' is converted.
> 
> Wasnt the hex conversion undocumented? 

Yes, and I fixed the manpage to document the behavior.
And, I fixed factor(6) to match the documentation in 
other aspects (e.g., leading '+' character).

> Since it seems to have issues, and is of dubious value
> might it might be best to just remove it?

It has been a part of FreeBSD's factor since
r104722 | fanf | 2002-10-09
That is 17 years.  Are you sure no one is using this feature
in some script?  What about backwards compatibility?

AFAICT, with my limited testing, my patch should fix
the issues that I discovered.

Do what you want with the patch (including ignoring it).
Hopefully, someone in the FreeBSD project will now 
recognize that factor(6) with and without OpenSSL gives
inconsistent results, and neither matches factor(6)'s
manpage.

-- 
Steve