Understanding Numeration in Computer Programming
Numeration systems shape every low-level decision a programmer makes, from the bit patterns that travel across buses to the colors that appear on screen. Understanding how numbers are represented, manipulated, and interpreted by silicon is the fastest way to eliminate whole classes of bugs before they compile.
Yet most courses stop at “binary is ones and zeros.” Real software demands fluency in signed encodings, arbitrary-length integers, fixed-point fractions, and the subtle rounding rules hidden inside every floating-point unit. The gaps between mathematical ideals and electronic realities are where crashes, leaks, and security flaws breed.
Binary and Bits: The Atomic Layer
Electric circuits distinguish only high and low voltage. A single bit is that difference frozen in time.
Group eight bits and you get a byte—256 distinct voltage histories that can be interpreted as unsigned integers 0–255, as ASCII letters, as x86 opcodes, or as the alpha channel of a pixel. The interpretation is not baked into the silicon; it is a contract you write in code.
Shift left by one position and you multiply by two for free; shift right and you divide, but the hardware may inject a 1 into the most-significant bit if the value was negative in two’s-complement form. Knowing whether the shift is arithmetic or logical prevents subtle off-by-one errors in graphics kernels.
Bitwise Operations as Control Tools
AND, OR, and XOR are not mere Boolean toys. A single AND with 0xFF extracts the low byte of a 32-bit register without a slow modulo operation.
Combine OR with shifted masks to pack four 4-bit nibbles into one 16-bit short for a retro game sprite. The same operators let you toggle, test, and clear feature flags in a configuration word atomically, avoiding mutex contention.
Endianness in Network Protocols
Send 0x1234 over TCP from an x86 laptop to an ARM router and the bytes may arrive as 0x3412. The C struct you cast over the buffer will then misread the opcode field.
htons and ntohs solve this by swapping bytes when the host byte order differs from network big-endian. Forgetting the swap once in a 10 000-line codebase can corrupt every packet without crashing the program, making the bug excruciating to trace.
Two’s Complement: The Signed Number Standard
Two’s complement is not “sign bit plus magnitude.” It is an offset that makes addition circuitry identical for signed and unsigned values.
Flip every bit of 5 (00000101) to get 11111010, add one, and you have 11111011, the encoding for −5. The same adder silicon that computes 5 + 3 = 8 also computes −5 + 3 = −2 without extra logic.
Overflow into the carry flag is therefore predictable: if two large positives sum to a negative, the carry flag remains 0 but the overflow flag sets, telling software to widen the register or signal an exception.
Detecting Overflow Without Assembly
In Java, adding two positive ints that yield a negative is still allowed, so wrap detection must be explicit. Compare the result against Integer.MIN_VALUE or use Math.addExact which throws ArithmeticException on overflow.
C# offers checked{ } blocks that inject trap instructions; the jitter rewrites them to efficient branch-on-overflow code. Turning checked on only for financial modules costs nothing in graphics loops where overflow is harmless.
Bit Width Trade-offs in Embedded Sensors
A 10-bit ADC reading ranges 0–1023. Storing it in uint16_t wastes 6 bits of RAM, but packing two samples into a 3-byte array complicates alignment.
ARM Cortex-M0 lacks hardware barrel shifters; unpacking nibbles costs four cycles per sample. Profiling showed that keeping 16-bit alignment halved interrupt latency, outweighing the 33 % RAM savings from bit packing.
Hexadecimal as a Compression Layer
Binary is unreadable at scale; decimal hides bit boundaries. Base 16 maps four bits to one glyph, turning 1111111100001010 into 0xFF0A.
That single line lets graphics engineers see the red and alpha channels of a pixel at a glance. Memory dumps line up on 16-byte boundaries, making pointer arithmetic errors obvious.
Color Codes and Bit Fields
CSS #3A7DFF encodes 0x3A red, 0x7D green, 0xFF blue. Swapping the bytes on a little-endian machine yields 0xFF7D3A, the integer 16734618.
Bitwise extraction becomes (color >> 16) & 0xFF for red, no division required. The same pattern applies to 5-6-5 16-bit textures on GPUs, where green gets the extra bit because human eyes resolve more shades of green.
Checksum Algorithms
Ethernet CRC32 treats the entire frame as a polynomial in GF(2). Each byte is a coefficient; shifting and XORing with the generator polynomial divides the message.
The remainder is appended as four hex bytes, letting receivers detect burst errors up to 32 bits long. Implementing the algorithm with lookup tables turns the polynomial math into 256 iterations of pre-computed bit shifts, cutting CPU time by 90 %.
Octal: The Forgotten Unix Footprint
Early PDP machines had 18-bit words divisible by three, so octal grouped bits cleanly. Unix file permissions still echo this heritage: 0755 means owner can read-write-execute, group and world can read-execute.
Each digit packs three bits, mapping exactly to rwx slots. Misremembering 644 versus 664 leaks write permission to strangers, a common server breach vector.
Escape Sequences in Strings
C compilers parse ‘ 33’ as octal 27, the ASCII escape character that starts ANSI color codes. Writing ‘x1B’ is clearer today, but legacy Makefiles still embed octal for terminal control.
A single off-by-one typo produces ‘ 333’ which is octal 333, emitting an illegal byte sequence that can lock older terminals until power cycled.
Arbitrary-Precision Integers: When 64 Bits Aren’t Enough
Public-key cryptography multiplies 2048-bit primes. Hardware registers top out at 64 bits, so software represents big integers as arrays of machine words.
GMP, Java BigInteger, and .NET BigInteger store bits in little-endian order regardless of CPU, simplifying carry propagation across architectures. Multiplication uses Karatsuba divide-and-conquer until operands shrink below a threshold, then drops to hardware MUL instructions for speed.
Constant-Time Arithmetic for Crypto
Timing attacks measure operand-dependent branches. Implementing BigInteger.modPow with simple if-carry statements leaks key bits through cache timing.
Cryptographic libraries replace branches with bitwise select operations: mask = (carry − 1) sets all bits to 0 or 1, then result = (mask & value) | (~mask & alternative). The execution path becomes flat, denying attackers a signal.
Memory Layout of Big Digits
OpenSSL’s BIGNUM uses bn_mul_words that operates on arrays of uint32_t on 32-bit platforms and uint64_t on 64-bit platforms. Aligning each digit to the natural word size avoids partial carries and lets the compiler vectorize with SIMD.
Profiling RSA-4096 on Apple M1 showed a 35 % speedup when digits were padded to 64-bit boundaries, even though the theoretical bit density dropped.
Floating-Point: The Approximate Realm
IEEE 754 trades exactness for range. A 32-bit float dedicates 1 bit to sign, 8 to exponent, 23 to mantissa, giving 7 decimal digits of precision and values up to 10^38.
0.1 becomes 0.100000001490116119384765625. Adding ten million of these accumulates an error of nearly one unit in the last place, enough to push a physics simulation through a wall.
Unit in the Last Place (ULP)
Two floats are equal within 1 ULP if their integer encodings differ by at most one. Comparing fabsf(a − b) <= FLT_EPSILON * fmaxf(fabsf(a), fabsf(b)) handles scaling but still fails near zero.
Game engines instead use integer casting: if (floatAsInt(a) − floatAsInt(b) <= maxUlps) accounts for exponent skew automatically. maxUlps of 4 gives sub-pixel collision stability without expensive branches.
Denormals and Performance Traps
When the exponent field is zero, the float enters denormal mode, disabling hardware barrel shifters. A sudden cascade of near-zero values can drop frame rates from 120 FPS to 6.
Setting the MXCSR register’s flush-to-zero bit on game startup forces tiny numbers to zero, trading minute accuracy for consistent timing. Audio DSP code does the same to keep convolution reverbs glitch-free.
Fixed-Point: Deterministic Fractions
Embedded DSPs lack FPUs. A 16-bit DAC that must output 0–5 V with 1 mV resolution stores values as 1.15 fixed-point: 1 integer bit, 15 fractional bits, scaling 1.0 to 32768.
Multiplication of two 1.15 numbers yields a 2.30 product; keeping the high 16 bits renormalizes back to 1.15. The operation is a single integer MAC instruction, taking one cycle and zero pipeline stalls.
Overflow Saturation Strategies
Without clamping, 200 % brightness wraps to dark in a pixel shader. ARM Cortex-M4 offers a QADD instruction that saturates on overflow, turning 32767 + 2 into 32767 instead of −32768.
Manual saturation in C is (x > MAX) ? MAX : (x < MIN) ? MIN : x; but the ternary compiles to branches. Replacing with SIMD intrinsics _qadd16 processes two pixels at once, doubling throughput.
Base64 and Radix Economy
Transmitting binary over JSON requires printable characters. Base64 maps 6 bits to one of 64 safe glyphs, expanding data by 33 % but surviving SMTP gateways that rewrite byte 0x0A.
The choice of 64 is not accidental: it is the largest power-of-two below the ASCII printable range, maximizing information density while avoiding quotes and backslashes. Custom bases like Base85 squeeze another 7 % but risk delimiter collisions in URLs.
URL-Safe Variants
RFC 4648 swaps ‘+’ and ‘/’ for ‘-’ and ‘_’ to avoid percent-encoding in query strings. Forgetting the swap breaks OAuth signatures when the callback URL contains padding.
Libraries expose separate encodeURL functions; calling the wrong one produces tokens that validate on the client but fail server-side, a discrepancy that only surfaces in production under specific locale configurations.
Custom Radices for Human Memory
Git short-SHA prefixes use base 36 (0-9a-z) to keep hashes pronounceable. Seven characters encode 36^7 ≈ 78 billion combinations, enough to uniquely identify any commit in repositories under that size.
Choosing base 58 removes visually confusing 0, O, l, I for handwritten backup codes. Bitcoin addresses employ this alphabet, reducing transcription errors when users read keys aloud over the phone.
Error Detection Base Conversion
Base32 checksums embed a modulo-37 residue. Any single character typo changes the residue, revealing the mistake before the code leaves the user’s notebook.
The algorithm multiplies each digit by a position-dependent weight derived from 2^i mod 37, a cheap operation on 8-bit microcontrollers that lack hardware division.
Big-Endian vs Little-Endian in File Formats
PNG stores integers in network byte order to remain architecture-agnostic. A BMP header uses little-endian because the format originated on x86.
Writing cross-platform parsers requires explicit byte swapping macros. Template meta-programming in C++ can generate swap code at compile time, eliminating branches when the host and file endianness match.
Memory-Mapped Structures
Casting a char* directly onto a struct works only if every member is naturally aligned and the file endianness equals the CPU. Misaligned access on ARM triggers a bus error even in user space.
Portable libraries deserialize field by field with explicit shifts, trading a few cycles for immunity to future compiler padding changes. Google Protobuf encodes integers as varint, a little-endian byte stream that removes endian concerns entirely.
Character Encoding: Numbers That Look Like Letters
UTF-8 encodes Unicode code points as variable-length byte sequences. The bit pattern 110xxxxx 10xxxxxx stores 11 useful bits across two bytes, enough for Greek, Cyrillic, and symbols.
Misinterpreting a UTF-8 stream as Latin-1 turns 0xC3 0xA9 into “é”, breaking user names in databases. The fix is not more escaping; it is declaring column charset utf8mb4 and connecting with the same encoding.
Normalization Forms
É can be a single code point U+00C9 or the sequence U+0045 U+0301. String comparison without normalization treats them as distinct, causing duplicate account creation.
Apple’s HFS+ enforces NFD, while NTFS uses NFC. A zip file created on macOS containing “café” may contain decomposed sequences that appear corrupted on Windows unless the application normalizes on extraction.
Time Stamps: Epochs and Leap Seconds
Unix time counts non-leap seconds since 1970-01-01 00:00:00 UTC. Leap seconds insert 23:59:60, a timestamp that does not exist in POSIX.
Google smears the extra second over 24 hours, doubling the length of one second in their NTP servers. Databases that use monotonic microsecond counters must never rewind; they instead pause until real time catches up.
64-Bit Millisecond Roll-Over
System.currentTimeMillis() returns a 64-bit signed long; it will not overflow for 292 million years. But custom embedded protocols that truncate to 32-bit milliseconds roll over every 49.7 days.
Telemetry radios in drones rebooted mid-flight when their uptime counter wrapped. The fix subtracts a base timestamp on the ground station, keeping the airborne packet size at 4 bytes while avoiding wrap ambiguity.
Random Numbers and Entropy Budget
/dev/urandom pools environmental noise into 4096 bits. Once initialized, it stretches entropy with a CSPRNG, providing unlimited random bytes without blocking.
Virtual machines cloned from the same golden image boot with identical seed files, producing duplicate SSH keys. Cloud providers now inject fresh entropy from the hypervisor at first boot via virtio-rng.
Modulo Bias Elimination
Taking rand() % 10 skews toward 0 because 32768 is not divisible by ten. Rejection sampling discards values above 32759 and retries, giving uniform digits at the cost of unpredictable loop iterations.
C++20 std::uniform_int_distribution implements this behind the scenes, but older codebases must hand-roll the algorithm to prevent predictable game loot drops that ruin monetization fairness.
Check digits in the Real World
ISBN-13 appends a modulo-10 checksum weighted 1,3,1,3… Any single digit error or adjacent transposition changes the residue, catching manual entry mistakes at bookstore tills.
Credit card PANs use Luhn algorithm, a recursive sum that treats digits as base 10 but processes them from right to left. Online payment gateways reject invalid PANs before sending them to the network, saving interchange fees on obviously fake numbers.
Base Conversion for Humans
UK bank sort codes are printed as 99-99-99 but stored as 6-digit integers in COBOL databases. The hyphen is presentation only; mobile apps strip it with a regex before calling the sort-code validation API.
Failure to normalize causes duplicate payment batches when the same sort code is keyed twice with and without hyphens, a bug that only surfaces during end-of-day reconciliation when totals refuse to balance.