Skip to content
Reverse Engineering

Concepts

general

Two's Complement

Two's complement is the universal encoding of signed integers on x86-64 — understanding it explains why negation is ~x+1, and how overflow and sign extension work.

Two's complement encodes both positive and negative integers in the same bit pattern such that addition, subtraction, and multiplication work identically for signed and unsigned values — the CPU does not need separate arithmetic units.

Encoding

For an N-bit value:

  • Positive values: standard binary (0 to 2^(N-1) - 1).
  • Negative values: 2^N - |value|.
  • The MSB (most significant bit) is the sign bit: 0 = positive, 1 = negative.
8-bit examples:
  0b00000001 =  1
  0b01111111 =  127
  0b10000000 = -128  (most negative)
  0b11111111 = -1
  0b11111110 = -2

Negation: ~x + 1

To negate a value: invert all bits (NOT), then add 1.

asm
not rax
add rax, 1   ; equivalent to: neg rax
c
// -x == (~x) + 1  (for two's complement)
assert(-42 == (~42) + 1);  // true

Overflow and the OF flag

Signed overflow occurs when the result cannot be represented in the destination width. The CPU sets OF when the sign of the result is mathematically wrong.

asm
mov al,  127   ; 0x7F
add al,  1     ; result = 0x80 = -128  ← signed overflow!  OF=1

Unsigned overflow (carry) sets CF instead.

Sign extension

Extending a value to a wider type preserves the numeric value by filling high bits with copies of the sign bit:

c
int8_t  s = -1;   // 0xFF
int16_t w = s;    // 0xFFFF = -1  (sign-extended)
int32_t d = s;    // 0xFFFFFFFF = -1

On x86-64 this is done with MOVSX / MOVSXD / CBW / CWDE / CDQE.

Reverse-engineering notes

  • neg rax is one instruction for two's-complement negation; encountering not rax; add rax, 1 is equivalent and sometimes emitted by compilers or hand-written code.
  • The range asymmetry (-128 to 127 for 8-bit) means INT_MIN negated overflows back to INT_MIN — a source of security bugs.
  • SAR (arithmetic right shift) sign-extends during shifting, preserving the sign of the operand — the compiler uses it for signed division by powers of two.
  • When a decompiler assigns a negative constant to an unsigned variable (e.g. 0xFFFFFFFF as a uint32_t) the value is the two's complement representation of -1; interpret it in the appropriate signed/unsigned context.