util: avoid signed integer overflows in git__strntol64

While `git__strntol64` tries to detect integer overflows when doing the
necessary arithmetics to come up with the final result, it does the
detection only after the fact. This check thus relies on undefined
behavior of signed integer overflows. Fix this by instead checking
up-front whether the multiplications or additions will overflow.

Note that a detected overflow will not cause us to abort parsing the
current sequence of digits. In the case of an overflow, previous
behavior was to still set up the end pointer correctly to point to the
first character immediately after the currently parsed number. We do not
want to change this now as code may rely on the end pointer being set up
correctly even if the parsed number is too big to be represented as
64 bit integer.

Author: pks-t

src/util.c

@@ -126,10 +126,20 @@ int git__strntol64(int64_t *result, const char *nptr, size_t nptr_len, const cha
 			v = c - 'A' + 10;
 		if (v >= base)
 			break;
-		nn = n * base + (neg ? -v : v);
-		if ((!neg && nn < n) || (neg && nn > n))
+		v = neg ? -v : v;
+		if (n > INT64_MAX / base || n < INT64_MIN / base) {
 			ovfl = 1;
-		n = nn;
+			/* Keep on iterating until the end of this number */
+			continue;
+		}
+		nn = n * base;
+		if ((v > 0 && nn > INT64_MAX - v) ||
+		    (v < 0 && nn < INT64_MIN - v)) {
+			ovfl = 1;
+			/* Keep on iterating until the end of this number */
+			continue;
+		}
+		n = nn + v;
 	}
 
 Return: