The Intel Pentium F00F Bug

Description and Workarounds

By Robert R. Collins

Such an illegal encoding should generate the requisite #UD. As one would expect, a CMPXCHG8B EAX instruction does generate a #UD. However, when this illegal encoding is prepended with a LOCK prefix, the processor fails to work correctly.

Using the LOCK prefix on this form of CMPXCHG8B is illegal in and of itself. LOCK prefixes are only allowed on memory-based read-modify-write instructions. Hence a LOCK prefix on the register-based CMPXCHG8B EAX instruction should also generate an invalid opcode exception.

Instead, the Pentium processor locks up and freezes the entire computer when it encounters this instruction. This bug is especially nasty, because any user can construct a program with this instruction, and upload it to a network computer, or incorporate it within an Active-X applet. Once the program is run on the network, the network server crashes. The only possible recovery comes by hitting the big red switch. Suppose you download an Active-X applet which contains this code. As soon as the code executes, YOUR computer will freeze up.

The serious nature of this bug has prompted Intel to give it their highest attention. Within one week, they announced a software workaround, which can be incorporated into virtually any operating system (except real-mode operating systems, like DOS).

Here's how the bug works

When the processor encounters this instruction (F0 0F C7 C8, or anything from F0 0F C7 C8..CF), the F00F bug occurs. The processor recognizes that an invalid opcode has occurred and tries to dispatch the #UD handler. Because of the LOCK prefix, the processor is confused. When the processor issues the bus reads to get the #UD handler vector address, the processor erroneously asserts the LOCK# signal. The LOCK# signal can only be asserted for read-modify-write instructions which modify memory. When the bus is locked, a locked memory read must be followed by a locked memory write, lest unpredictable results may occur. But in this case, the LOCK# signal remains asserted for the two consecutive memory reads required to retrieve the #UD vector address. The processor never issues any intervening locked write, and then hangs itself. This behavior is shown in the logic analyzer trace in Listing 1. As you can see, the Pentium tries to retrieve the #UD vector with two locked reads. After that point, all processor activity stops.

Sequence Address  Data          Mnemonic                               Timestamp
--------------------------------------------------------------------------------
T 524285 000000B2 ----7E--------(-IO-WRITE-)------------------------------300-ns
  524286 00000018   ----E14C    ( LOCKED MEM READ )                       440 ns
  524287 0000001A   F000----    ( LOCKED MEM READ )                       100 ns

Listing 1 -- F00F Bug Example

The Various Workarounds

There are various possible workarounds to this bug. Not all of them are good. In fact a few of them are outright kludgey. Intel has proposed two workarounds. One of the workarounds actually takes advantage of the bug behavior to do the right thing. Their other workaround is ingenious, though it's a horrible kludge. The first two alternate workarounds presented below, are given for academic purposes only. Even though the workarounds have demonstrated their ability to obscure the bug behavior, they are not entirely reliable.

Intel's first workaround

1. Align the Interrupt Descriptor Table (IDT) such that it spans a 4KB page boundary by placing the first entry starting 56 bytes from the end of the first 4KB page. This places the first seven entries (0-6) on the first 4KB page, and the remaining entries on the second page.
2. The page containing the first seven entries of the IDT must not have a mapping in the OS page tables. This will cause any of exceptions 0-6 to generate a page not present fault. A page fault prevents the bus lock condition and gives the OS complete control to process these exceptions as appropriate. Note that exception 6 is the invalid opcode exception, so with this scheme an OS has complete control of any program executing an invalid CMPXCHG8B instruction.

1. Recognize accesses to the first page of the IDT by testing the fault address in CR2. Page not present faults on other addresses can be processed normally.
2. For page not present faults on the first page of the IDT, the OS must recognize and dispatch the exception which caused the page not present fault. Before proceeding, test the fault address in CR2 to determine if it is in the address range corresponding to exceptions 0-6.
3. Calculate which exception caused the page not present fault from the fault address in CR2.
4. Depending on the operating system, certain privilege level checks may be required, along with adjustments to the interrupt stack.
5. Jump to the normal handler for the appropriate exception.

This is an ingenious solution to a horrible problem. Unfortunately, the solution is just as bad as the problem. When the processor receives any of the first seven exceptions (Divide by Zero through Invalid Opcode), the processor generates a page fault instead of the appropriate exception. The page fault handler gets mucked up with all kinds of code to check privilege levels and whether or not the fault was caused by another exception. If I had my druthers, I'd stay as far away from this solution as possible.

Intel's Second Workaround

1. Mark the page containing the first seven entries (0-6) of the IDT as read only by setting bit 1 of the page table entry to zero. Also set CR0.WP (bit 16) to 1. Now when the invalid opcode exception occurs on the locked CMPXCHG8B instruction, the processor will trigger a page fault since it does not have write access to the page containing entry 6 of the IDT. This page fault prevents the bus lock condition and gives the OS complete control to process the invalid operand exception as appropriate. Note that exception 6 is the invalid opcode exception, so with this scheme an OS has complete control of any program executing an invalid CMPXCHG8B instruction.
2. Optional: If updates to entries 7-255 of the IDT occur during the course of normal operation, page faults should be avoided on writes to these IDT entries. These page faults can be avoided by aligning the IDT across a 4KB page boundary such that the first seven entries (0-6) of the IDT are on the first read only page and the remaining entries are on a read/writeable page.

1. Modify the page fault handler to calculate which exception caused the page fault using the fault address in CR2. If the error code on the stack indicates the exception occurred from ring 0 and if the address corresponds to the invalid opcode exception, then pop the error code off the stack and jump to the invalid opcode exception handler. Otherwise continue with the normal page fault handler.

This workaround is really quite clever. This workaround takes advantage of the bug as a means to provide a fix to the problem. When any of the first six exceptions occur, they are handled as they normally would. Divide by Zero through BOUND exceptions vector to their normal exception handlers without any intervening code in the page fault handler. However, when the F00F bug occurs, the page fault handler is invoked instead of the #UD handler. Why? CR0.WP=1 instructs the microprocessor to generate a page fault when an attempt is made by the supervisor to modify a memory page. The processor doesn't actually attempt to modify the Interrupt Descriptor Page (IDT page holding the #UD vector address) when the F00F opcode is encountered. But the bug actually makes the processor think it's modifying the IDT page with the #UD vector. The locked memory cycle somehow convinces the internal state of the Pentium to think that a write cycle is going to occur. Since the transition to the #UD handler is considered a supervisor task, the processor thinks it's going to write to this page. Thus when CR0.WP=1, a page fault occurs.

Even though this is a very clever fix, there are two things I don't like about it:

1. The fix requires a kludge to the page fault handler code. The page fault handler must contain code to detect this condition, and vector to the #UD handler.
2. Setting CR0.WP=1 isn't a viable solution for everybody. Some operating systems may not be able to use this workaround.

If I were forced to choose between two of Intel's "blessed" solutions, I'd choose this one. However, because Intel set a precedent in documenting a solution that actually takes advantage of the bug behavior, this could give rise to much more elegant solutions that also take advantage of the bug behavior.

Why Intel failed to find the F00F Bug

On November 6, 1997 a message was posted anonymously on comp.os.linux.advocacy -- one of the thousands of Internet newsgroups. The message warned users of a bug in the Pentium and Pentium MMX processors. The bug could completely lock up the computer from any operating mode in any operating system. At first glance, the ordinary news reader might say "so what!" After all, many users are accustomed to their Windows 3.1 and Windows 95 systems locking up with a regular frequency. But the placement of the bug on comp.os.linux.advocacy was calculated and intentional. The readers on that newsgroup knew exactly what the bug meant. To them, the bug meant that an ordinary user or saboteur could unleash a program to bring down their network servers.

These servers form the backbone of our modern Internet community. Internet Service Providers, Web hosts, Government agencies, and Computer departments at universities were petrified of the potential damage that could be caused by this bug. Internet Service Providers (ISPs) could be attacked with a program that would bring down their web servers. Saboteurs could incorporate the code for this bug into an Active-X control that could be downloaded and used to crash YOUR computer. Students that didn't want to do their homework could attack universities. Think of it, the bug could be the latest variation of "my dog ate my homework." Therefore the news of the bug had a huge impact in this small community. But the history of this bug didn't start on November 6^th. The real story starts months, maybe even years before this date.

A couple of months before this bug anonymously appeared on the Internet, I received a phone call from an industry colleague. This person was elated about this bug and strongly urged me to publish it at my web site (http://www.x86.org). The whole story didn't sit well with me, especially their desire for me to make the public disclosure, instead of them. I felt like I was being used. But that wasn't my only reason for refusing to disclose the bug. Quite simply, I didn't want to be responsible for telling a potential saboteur the computing equivalent of how to build an atom bomb, or the legal liability associated with the lawsuits that might accompany such a disclosure.

Within a day or so, I called another colleague who used to work at one of Intel's competitors. I told him about the bug. To which he responded: "yeah, I found that bug about a year and a half ago." If this bug was so easy for one or more of Intel's competitors to find, why hadn't they ever discovered it, disclosed it, or quietly fixed it? The answers lie in the nature of Intel's competition, and Intel's design verification methodology.

It is quite natural for Intel's competition to find bugs like this. They are in the microprocessor cloning business. Therefore, they must write programs that scan the entire opcode space in search of hidden and undocumented instructions. Invalid opcodes come in many forms. The most obvious form is an undefined instruction -- an opcode that doesn't map to any instruction. These opcodes are easy to scan programmatically. Other invalid opcodes are actually invalid encodings of valid instructions. These invalid encodings are often times overlooked when scanning for undefined instructions.

Intel's competition must find these instructions. Therefore they write programs that scan the entire invalid opcode space. A well-designed program will test all of the invalid opcodes and invalid encodings of valid opcodes. This is how my colleague found this bug 18 months ago. Even though he knew about the bug, his knowledge was the intellectual property of his company. Therefore he would have been prohibited from warning the Internet community, even if he wanted to warn them.

Intel doesn't need to write programs like this…or do they? Intel defined the opcode space. If anybody knows what's in it, they do. Therefore they don't really have a need to hire low-level assembly language programmers to write such programs…or do they? In fact, Intel's design verification department has turned down some of the best x86 experts in the industry, citing Intel's lack of need for their low-level assembly language skills. Instead, Intel has relied on test generation programs, primarily written in C. These random test generation programs (RTPGs) randomly create programs, which runs on the target microprocessor. RTPGs are great for finding certain types of microprocessor bugs. But the RTPG is only as good as its design rules allow. Most likely, Intel's RTPG would not test for these invalid opcode encodings. Without augmenting their RTPG with low-level assembly programmers to write such programs, Intel's design verification methodology ultimately failed to find the F00F bug.

Maybe Intel Design Verification Department will think twice next time when presented with an x86 assembly language expert.