Re: OpenSSL breaks factor(6)

On Sat, Dec 28, 2019 at 10:46:52PM -0500, Garance A Drosehn wrote:
> On 27 Dec 2019, at 17:42, Steve Kargl wrote:
> >
> > This patch now includes a fix for hexadecimal conversion.  It
> > simple scans the string for a hex digit in [a,...,f] and assumes
> > that a hexadecimal string has been entered.  A string that includes
> > character from the decimal digits is assumed to by a decimal
> > representation.
> 
> What if the user wants to factor a hexadecimal value which does not
> happen to include [a...f]?
> 
> How about recognizing a prefix of '0x' as a way to indicate the value
> is hexadecimal?
> 

An interested user will need to add that support.  AFAIK, factor(6)
has never recognized the 0x prefix, and I'm not trying to add new 
features.  I'm simply fixing factor(6) to match its documentation, 
and trying to ensure WITH_OPENSSL and WITHOUT_OPENSSL give the 
same results where applicable.

The logic is to first try to convert the string to a decimal if
the leading digits are members of the set [0,...,9].  If this
conversion fails, then try to convert the string as a hexadecimal
number.  A problem occurs because OpenSSL's BN_dec2bn does not fail
for a number like '1abc' (converts it to 1) whereas the local
implementation of BN_dec2bn fails during the conversion, and so
the BN_hex2bn function is executed and '1abc' is converted.

-- 
Steve