The Anatomy of Memory Vulnerabilities: How Buffer Overflow, ROP Chains, and OS Defenses Work

Imagine you have a form with a field for entering a name. Instead of three words, you type in a thousand characters of continuous text. Instead of displaying an error message, the application suddenly hands over full control of the computer to an attacker. Welcome to the world of memory vulnerabilities—a place where a lack of boundary checks in the code can bring down even the most secure systems in the world.

Even in 2026, memory corruption vulnerabilities continue to make up a critical portion of reported security vulnerabilities (CVEs). From the legendary Morris Worm of 1988 to modern attacks on smartphones and IoT devices, the buffer overflow principle remains one of hackers’ most feared weapons.

In this article, we’ll look at how the memory stack (Stack) works, how a buffer overflow occurs, how attackers have adapted to security mechanisms using the ROP technique, and why tech giants are trying to replace C and C++ with the more secure Rust.

1. The Stack: How a Program Keeps Track of What It’s Doing

To understand how memory can be “manipulated,” we first need to know how C/C++ programs manage memory when calling functions.

When a program calls a function (for example spocitaj_dane()), the operating system allocates a portion of memory for it on what’s called the stack. The stack operates on the LIFO (Last-In, First-Out) principle—think of it as a stack of plates.

The following are stored on the stack in memory:

1. Local variables: Arrays of characters, numbers, structures (e.g., char meno[64]).
2. Return Address (EIP/RIP): An instruction for the processor indicating where in the code it should return after the function completes.

And this is where a fundamental architectural problem arises: Local variables and the return address are stored right next to each other in memory.

2. Classic Buffer Overflow: When the Cup Overflows

Imagine the following C code, which seems innocent at first glance:

void over_heslo() {
    char buffer[64]; // Vyhradený priestor pre 64 znakov
    gets(buffer);    // NEBEZPEČNÉ: načítava text bez kontroly dĺžky!
}

The function gets() carelessly reads user input and stores it in a variable buffer.

• If the user enters 20 characters, everything is fine.
• If the user enters 100 characters, the first 64 fill buffer and the remaining 36 characters begin to overwrite everything in memory behind them!

How can an attacker exploit this?

1. Overwriting the Return Address: The attacker precisely calculates the distance (offset) between the buffer and the return address.
2. Shellcode Injection: The attacker typically embeds a small piece of executable code written in assembly language (known as “shellcode”) into the input, which opens a command line (shell).
3. Flow Redirection: The attacker overwrites the return address so that it does not point back to the original program, but to the address of their own shellcode.

When the function ends, the processor reads the injected address from memory and unsuspectingly executes the attacker’s code with the privileges of the running program.

3. The First Line of Defense and the Hackers’ Response

The IT world did not sit idly by. Developers of operating systems and compilers gradually introduced security barriers. But for every barrier, the hacker community devised a countermeasure.

Protection 1: DEP / NX Bit (Data Execution Prevention)

• Principle: Memory regions are marked with flags: Read (R), Write (W), or Execute (X). The stack allows writing but prohibits code execution.
• Consequence: Even if an attacker stores their shellcode on the stack, the processor will refuse to execute it, and the program will crash (Segmentation Fault).

Attackers’ Countermeasure: ROP (Return-Oriented Programming)

When attackers could no longer execute their own code from memory, they decided to use code that already existed in the program (or in system libraries such as libc)!

• Gadgets: The attacker searches legitimate executable code for small snippets of instructions ending with the command ret (e.g., pop eax; ret;).
• ROP Chain: By stacking hundreds of such return addresses one after another on the stack, the attacker can force the processor to sequentially execute individual “gadgets” and assemble them into arbitrary logic without executing a single byte of their own unpublished code.

4. Modern Security Shield in the OS

Today’s operating systems (Windows 11, Linux, macOS, iOS, Android) use a multi-layered protection architecture that makes buffer overflow exploits extremely difficult:

Protection Mechanism	How does it work?
ASLR (Address Space Layout Randomization)	Every time a program is launched, it randomly shuffles the addresses in RAM. An attacker cannot know in advance where functions or gadgets are located in memory.
Stack Canaries	The compiler inserts a secret random value before the return address. If this value is overwritten during a buffer overflow, the program detects the attack and terminates immediately.
CFI (Control Flow Integrity)	Hardware and software checks that ensure code jumps only to valid and expected locations.

5. Security Researchers’ Tools (with GitHub links)

Ethical hackers and security analysts use specialized tools to detect and test memory vulnerabilities:

• Pwntools: The most popular Python framework for creating exploits and CTF (Capture The Flag) competitions.
  • GitHub: Gallopsled/pwntools
• ROPgadget: A tool that automatically scans a binary file and finds exploitable ROP gadgets within it to bypass DEP.
  • GitHub: JonathanSalwan/ROPgadget
• GDB with the GEF/PEDA extension: The legendary debugger enhanced with stack visualization, canary checks, and memory address lookup.
  • GitHub: hugsy/gef

6. The Future: The End of C/C++ in Critical Systems?

Despite decades of educating developers, memory errors continue to recur. The reason is simple: C and C++ give programmers direct control over memory but provide no safety nets.

That is why the U.S. National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA) have officially recommended a transition to “memory-safe” programming languages.

Companies such as Microsoft, Google (in the Android system), and the Linux kernel are now adopting Rust on a massive scale. Thanks to the concepts of ownership and the borrow checker, Rust can guarantee during compilation that the code will not suffer from buffer overflows, "Use-After-Free," or pointer invalidation—all without any loss of performance and without the need for a garbage collector.

Conclusion

Memory hacking and the fight against it resemble an endless game of cat and mouse. Every new OS protection mechanism forces researchers to discover a new, even more sophisticated attack technique.

Understanding how buffer overflows work and how to manipulate the stack is not only the foundation of ethical hacking but, above all, essential knowledge for any software engineer who wants to write secure and resilient code.