blob: 06f368bb623e8eccbcdcb7b197090c110a442bcf [file] [log] [blame]
Martin Roth9b1b3352016-02-24 12:27:06 -08001 The Anatomy & Physiology of Memtest86-SMP
2 -----------------------------------------
41. Binary layout
Martin Roth4dcd13d2016-02-24 13:53:07 -08005
Martin Roth9b1b3352016-02-24 12:27:06 -08006 ---------------------------------------------------------------
7 | bootsect.o | setup.o | head.o memtest_shared |
8 ---------------------------------------------------------------
9Labels _start<-------memtest---------->_end
10 -----------------------------------------------------------
11addr 0 512 512+4*512 |
12 -----------------------------------------------------------
142. The following steps occur after we power on.
Martin Roth4dcd13d2016-02-24 13:53:07 -080015 a. The bootsect.o code gets loaded at 0x7c00
16 and copies
Martin Roth9b1b3352016-02-24 12:27:06 -080017 i. itself to 0x90000
18 ii. setup.o to 0x90200
Martin Roth4dcd13d2016-02-24 13:53:07 -080019 iii. everything between _start and _end i.e memtest
Martin Roth9b1b3352016-02-24 12:27:06 -080020 to 0x10000
21 b. jumps somewhere into the copied bootsect.o code at 0x90000
22 ,does some trivial stuff and jumps to setup.o
23 c. setup.o puts the processor in protected mode, with a basic
Martin Roth4dcd13d2016-02-24 13:53:07 -080024 gdt and idt and does a long jump to the start of the
25 memtest code (startup_32, see 4 below). The code and data
26 segment base address are all set to 0x0. So a linear
Martin Roth9b1b3352016-02-24 12:27:06 -080027 address range and no paging is enabled.
Martin Roth4dcd13d2016-02-24 13:53:07 -080028 d. From now on we no longer required the bootsect.o and setup.o
Martin Roth9b1b3352016-02-24 12:27:06 -080029 code.
303. The code in memtest is compiled as position independent
Martin Roth4dcd13d2016-02-24 13:53:07 -080031 code. Which implies that the code can be moved dynamically in
Martin Roth9b1b3352016-02-24 12:27:06 -080032 the address space and can still work. Since we are now in head.o,
Martin Roth4dcd13d2016-02-24 13:53:07 -080033 which is compiled with PIC , we no longer should use absolute
34 addresses references while accessing functions or globals.
Martin Roth9b1b3352016-02-24 12:27:06 -080035 All symbols are stored in a table called Global Offset Table(GOT)
Martin Roth4dcd13d2016-02-24 13:53:07 -080036 and %ebx is set to point to the base of that table. So to get/set
37 the value of a symbol we need to read (%ebx + symbolOffsetIntoGOT) to
Martin Roth9b1b3352016-02-24 12:27:06 -080038 get the symbol value. For eg. if foo is global varible the assembly
39 code to store %eax value into foo will be changed from
40 mov %eax, foo
Martin Roth4dcd13d2016-02-24 13:53:07 -080041 to
Martin Roth9b1b3352016-02-24 12:27:06 -080042 mov %eax, foo@GOTOFF(%ebx)
Martin Roth4dcd13d2016-02-24 13:53:07 -0800434. (startup_32) The first step done in head.o is to change
44 the gdtr and idtr register values to point to the final(!)
45 gdt and ldt tables in head.o, since we can no longer use the
46 gdt and ldt tables in setup.o, and call the dynamic linker
47 stub in memtest_shared (see call _dl_start in head.S). This
48 dynamic linker stub relocates all the code in memtest w.r.t
49 the new base location i.e 0x1000. Finally we call the test_start()
Martin Roth9b1b3352016-02-24 12:27:06 -080050 'C' routine.
Martin Roth4dcd13d2016-02-24 13:53:07 -0800515. The test_start() C routine is the main routine which lets the BSP
52 bring up the APs from their halt state, relocate the code
53 (if necessary) to new address, move the APs to the newly
Martin Roth9b1b3352016-02-24 12:27:06 -080054 relocated address and execute the tests. The BSP is the
55 master which controls the execution of the APs, and mostly
Martin Roth4dcd13d2016-02-24 13:53:07 -080056 it is the one which manupulates the global variables.
Martin Roth9b1b3352016-02-24 12:27:06 -080057 i. we change the stack to a private per cpu stack.
58 (this step happens every time we move to a new location)
59 ii. We kick start the APs in the system by
Martin Roth4dcd13d2016-02-24 13:53:07 -080060 a. Putting a temporary real mode code
61 (_ap_trampoline_start - _ap_trampoline_protmode)
62 at 0x9000, which puts the AP in protected mode and jumps
63 to _ap_trampoline_protmode in head.o. The code in
64 _ap_trampoline_protmode calls start_32 in head.o which
Martin Roth9b1b3352016-02-24 12:27:06 -080065 reinitialises the AP's gdt and idt to point to the
66 final(!) gdt and idt. (see step 4 above)
67 b. Since the APs also traverse through the same initialisation
68 code(startup_32 in head.o), the APs also call test_start().
Martin Roth4dcd13d2016-02-24 13:53:07 -080069 The APs just spin wait (see AP_SpinWaitStart) till the
70 are instructed by the BSP to jump to a new location,
71 which can either be a test execution or spin wait at a
Martin Roth9b1b3352016-02-24 12:27:06 -080072 new location.
73 iii. The base address at which memtest tries to execute as far
74 as possible is 0x2000. This is the lowest possible address
Martin Roth4dcd13d2016-02-24 13:53:07 -080075 memtest can put itself at. So the next step is to
Martin Roth9b1b3352016-02-24 12:27:06 -080076 move to 0x2000, which it cannot directly, since copying
77 code to 0x2000 will override the existing code at 0x1000.
78 0x2000 +sizeof(memtest) will usually be greater than 0x1000.
79 so we temporarily relocated to 0x200000 and then relocate
80 back to 0x2000. Every time the BSP relocates the code to the
Martin Roth4dcd13d2016-02-24 13:53:07 -080081 new location, it pulls up the APs spin waiting at the old
82 location to spin wait at the corresponding relocated
Martin Roth9b1b3352016-02-24 12:27:06 -080083 spin wait location, by making them jump to the new
84 statup_32 relocated location(see 4 above).
Martin Roth4dcd13d2016-02-24 13:53:07 -080085 Hence forth during the tests 0x200000 is the only place
86 we relocate to if we need to test a memory window
Martin Roth9b1b3352016-02-24 12:27:06 -080087 (see v. below to get a description of what a window is)
88 which includes address range 0x2000.
90 Address map during normal execution.
91 --------------------------------------------------------------------
92 | head.o memtest_shared | |RAM_END
93 --------------------------------------------------------------------
94Labels _start<-------memtest---------->_end
95 --------------------------------------------------------------------
96addr 0x0 0x2000 | Memory that is being tested.. |RAM_END
97 --------------------------------------------------------------------
99 Address map during relocated state.
100 --------------------------------------------------------------------
101 | head.o memtest_shared | |RAM_END
102 --------------------------------------------------------------------
103Labels _start<-------memtest---------->_end
104 --------------------------------------------------------------------
105addr memory that is being tested... |0x200000 | |RAM_END
106 --------------------------------------------------------------------
Martin Roth4dcd13d2016-02-24 13:53:07 -0800108 iv. Once we are at 0x2000 we initialise the system, and
109 determine the memory map ,usually via the bios e820 map.
110 The sorted, and non-overlapping RAM page ranges are
111 placed in v->pmap[] array. This array is the reference
112 of the RAM memory map on the system.
113 v. The memory range(in page numbers) which the
Martin Roth9b1b3352016-02-24 12:27:06 -0800114 memtest86 can test is partitioned into windows.
115 the current version of memtest86-smp has the capability
116 to test the memory from 0x0 - 0xFFFFFFFFF (max address
Martin Roth4dcd13d2016-02-24 13:53:07 -0800117 when pae mode is enabled).
Martin Roth9b1b3352016-02-24 12:27:06 -0800118 We then compute the linear memory address ranges(called
Martin Roth4dcd13d2016-02-24 13:53:07 -0800119 segments) for the window we are currently about to
Martin Roth9b1b3352016-02-24 12:27:06 -0800120 test. The windows are
Martin Roth4dcd13d2016-02-24 13:53:07 -0800121 a. 0 - 640K
Martin Roth9b1b3352016-02-24 12:27:06 -0800122 b. (0x2000 + (_end - _start)) - 4G (since the code is at 0x2000).
Martin Roth4dcd13d2016-02-24 13:53:07 -0800123 c. >4G to test pae address range, each window with size
Martin Roth9b1b3352016-02-24 12:27:06 -0800124 of 0x80000(2G), large enough to be mapped in one page directory
Martin Roth4dcd13d2016-02-24 13:53:07 -0800125 entry. So a window size of 0x80000 means we can map 1024 page
Martin Roth9b1b3352016-02-24 12:27:06 -0800126 table entries, with page size of 2M(pae mode), with one
127 page directory entry. Something similar to kseg entry
128 in linux. The upper bound page number is 0x1000000 which
129 corresponds to linear address 0xFFFFFFFFF + 1 which uses
Martin Roth4dcd13d2016-02-24 13:53:07 -0800130 all the 36 address bits.
131 Each window is compared against the sorted & non-overlapping
132 e820 map which we have stored in v->pmap[] array, since all
Martin Roth9b1b3352016-02-24 12:27:06 -0800133 memory in the selected window address range may correspond to
134 RAM or can be usable. A list of segments within the window is
Martin Roth4dcd13d2016-02-24 13:53:07 -0800135 created , which contain the usable portions of the window.
Martin Roth9b1b3352016-02-24 12:27:06 -0800136 This is stored in v->mmap[] array.
Martin Roth4dcd13d2016-02-24 13:53:07 -0800137 vi. Once the v->mmap[] array populated, we have the list of
Martin Roth9b1b3352016-02-24 12:27:06 -0800138 non-overlapping segments in the current window which are the
139 final address ranges that can be tested. The BSP executes the
140 test first and lets each AP execute the test one by one. Once
141 all the APs finish execting the same test, the BSP moves to the
Martin Roth4dcd13d2016-02-24 13:53:07 -0800142 next window follows the same procedure till all the windows
Martin Roth9b1b3352016-02-24 12:27:06 -0800143 are done. Once all the windows are done, the BSP moves to the
144 next test. Before executing in any window the BSP checks if
145 the window overlaps with the code/data of memtest86, if so
Martin Roth4dcd13d2016-02-24 13:53:07 -0800146 tries to relocate to 0x200000. If the window includes both
Martin Roth9b1b3352016-02-24 12:27:06 -0800147 0x2000 as well as 0x200000 the BSP skips that window.
148 Looking at the window values the only time the memtest
149 relocates is when testing the 0 - 640K window.
151Known Issues:
152* Memtest86-smp does not work on IBM-NUMA machines, x440 and friends.
154email comments to:
155Kalyan Rajasekharuni<>
156Sub: Memtest86-SMP