diff options
| -rw-r--r-- | docs/Boot-explained.txt (renamed from Boot-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/Building-and-running-explained.txt (renamed from Building-and-running-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/Exception-vector-explained.txt (renamed from Exception-vector-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/IRQ-explained.txt (renamed from IRQ-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/Linker-scripts-explained.txt (renamed from Linker-scripts-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/MMU-explained.txt (renamed from MMU-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/Makefile-explained.txt (renamed from Makefile-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/PSRs-explained.txt (renamed from PSRs-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/Project-structure-explained.txt (renamed from Project-structure-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/Ramfs-explained.txt (renamed from Ramfs-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/Scheduler-explained.txt (renamed from Scheduler-explained.txt) | 0 | ||||
| -rw-r--r-- | docs/Sources.txt (renamed from Sources.txt) | 0 | ||||
| -rw-r--r-- | docs/Various-random-things-explained-I-dont-feel-like-creating-a-separate-txt-for-each-or-ordering-them-in-any-way.txt (renamed from Various-random-things-explained-I-dont-feel-like-creating-a-separate-txt-for-each-or-ordering-them-in-any-way.txt) | 0 | ||||
| -rw-r--r-- | docs/processor-modes-explained.txt (renamed from processor-modes-explained.txt) | 0 | ||||
| -rw-r--r-- | document.md | 606 | ||||
| -rwxr-xr-x | makeDoc.sh | 2 |
16 files changed, 1 insertions, 607 deletions
diff --git a/Boot-explained.txt b/docs/Boot-explained.txt index 20528cc..20528cc 100644 --- a/Boot-explained.txt +++ b/docs/Boot-explained.txt diff --git a/Building-and-running-explained.txt b/docs/Building-and-running-explained.txt index 257c75f..257c75f 100644 --- a/Building-and-running-explained.txt +++ b/docs/Building-and-running-explained.txt diff --git a/Exception-vector-explained.txt b/docs/Exception-vector-explained.txt index b82eadb..b82eadb 100644 --- a/Exception-vector-explained.txt +++ b/docs/Exception-vector-explained.txt diff --git a/IRQ-explained.txt b/docs/IRQ-explained.txt index d67d52d..d67d52d 100644 --- a/IRQ-explained.txt +++ b/docs/IRQ-explained.txt diff --git a/Linker-scripts-explained.txt b/docs/Linker-scripts-explained.txt index 130eb0c..130eb0c 100644 --- a/Linker-scripts-explained.txt +++ b/docs/Linker-scripts-explained.txt diff --git a/MMU-explained.txt b/docs/MMU-explained.txt index a26c670..a26c670 100644 --- a/MMU-explained.txt +++ b/docs/MMU-explained.txt diff --git a/Makefile-explained.txt b/docs/Makefile-explained.txt index 98d73d0..98d73d0 100644 --- a/Makefile-explained.txt +++ b/docs/Makefile-explained.txt diff --git a/PSRs-explained.txt b/docs/PSRs-explained.txt index a125add..a125add 100644 --- a/PSRs-explained.txt +++ b/docs/PSRs-explained.txt diff --git a/Project-structure-explained.txt b/docs/Project-structure-explained.txt index 355910d..355910d 100644 --- a/Project-structure-explained.txt +++ b/docs/Project-structure-explained.txt diff --git a/Ramfs-explained.txt b/docs/Ramfs-explained.txt index 85d5ae5..85d5ae5 100644 --- a/Ramfs-explained.txt +++ b/docs/Ramfs-explained.txt diff --git a/Scheduler-explained.txt b/docs/Scheduler-explained.txt index f67fca6..f67fca6 100644 --- a/Scheduler-explained.txt +++ b/docs/Scheduler-explained.txt diff --git a/Sources.txt b/docs/Sources.txt index 7a48250..7a48250 100644 --- a/Sources.txt +++ b/docs/Sources.txt diff --git a/Various-random-things-explained-I-dont-feel-like-creating-a-separate-txt-for-each-or-ordering-them-in-any-way.txt b/docs/Various-random-things-explained-I-dont-feel-like-creating-a-separate-txt-for-each-or-ordering-them-in-any-way.txt index 0a3ae06..0a3ae06 100644 --- a/Various-random-things-explained-I-dont-feel-like-creating-a-separate-txt-for-each-or-ordering-them-in-any-way.txt +++ b/docs/Various-random-things-explained-I-dont-feel-like-creating-a-separate-txt-for-each-or-ordering-them-in-any-way.txt diff --git a/processor-modes-explained.txt b/docs/processor-modes-explained.txt index 6631655..6631655 100644 --- a/processor-modes-explained.txt +++ b/docs/processor-modes-explained.txt diff --git a/document.md b/document.md deleted file mode 100644 index 4802ac2..0000000 --- a/document.md +++ /dev/null @@ -1,606 +0,0 @@ -# Raspberry PI MMU project -//TODO insert [TOC] here -# Building and running - -Dependencies: - -1. Native GCC (+ binutils) -2. ARM cross-compiler GCC (+ binutils) (arm-none-eabi works - others might or might not) -3. GNU Make -4. rpi-open-firmware (for running on the Pi) -5. GNU screen (for communicating with the kernel when running on the Pi) -6. socat (for communicating with the bootloader when running on the Pi) -7. Qemu ARM (for emulating the Pi). - -For building rpi-open-firmware You will need more tools (not listed here). - -The project has been tested only in Qemu emulating Pi 2 and on real Pi 3. Running on Pis other than Pi 2 and Pi 3 is sure to require changing the definition in global.h (because peripheral base addresses differ between Pi versions) and might also require other modifications, not known at this time. - -## Building the project - -Assuming make, gcc, arm-none-eabi-gcc and its binutils are in the PATH, the kernel can be built with: - - $ make kernel.img -which is the same as: - - $ make -The bootloader can be built with: - - $ make loader.img - -Both loader and kernel can then be found in build/ - -## Running in Qemu - -To run the kernel (passed as elf file) in qemu, run: - - $ make qemu-elf - -If You want to pass a binary image to qemu, run: - - $ make qemu-bin - -If You want to pass loader image to qemu and pipe kernel to it through emulated uart, run: - - $ make qemu-loader -Note, that with qemu-loader the kernel will run, but will be unable to receive any keyboard input. - -The timer used by this project is the ARM timer ("based on an ARM AP804", with registers mapped at 0x7E00B000 in the GPU address space). It's absent in emulated environment, so no timer interrupts can be witnessed in qemu, as it would require to us to use simulated qemu clock. - -## Running on real hardware. - -First, build and test rpi-open-firmware. Now, copy either kernel.img or loader.img to the SD card (next to bootcode.bin) and rename it to zImage. Also copy .dtb file corresponding to your Pi (actually, any .dtb will do, it is not used right now) from stock firmware files to the SD card and name it rpi.dtb. Finally, create a cmdline.txt on the SD card (content doesn't matter). - -Now, connect RaspberryPi via UART to Your machine. GPIO on the Pi works with 3.3V, so You should make sure, that UART device on the other end is also working wih 3.3V. This is the pinout of the RaspberyPi 3 model B that has been used for testing so far: - -Top left of the board is here - -+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ -| 2| 4| 6| 8|10|12|14|16|18|20|22|24|26|28|30|32|34|36|38|40| -+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ -| 1| 3| 5| 7| 9|11|13|15|17|19|21|23|25|27|29|31|33|35|37|39| -+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+ - -Under rpi-open-firmware (stock firmware might map UARTs differently): - - pin 6 is Ground - pin 8 is TX - pin 10 is RX - -Once UART is connected, You can power on the board. - -Assuming a USB to UART adapter is used and it is seen by Your system as /dev/ttyUSB0: - -If You copied the kernel to the SD card, You can start communicating with it by running: - - $ screen /dev/ttyUSB0 115200,cs8,-parenb,-cstopb,-hupcl - -If You copied the loader, You can send it the kernel image and start communicating with the system by running: - - $ make run-on-rpi - -To run again, replug USB to UART adapter and Pi's power supply (order matters!) and re-enter the command. Replugging power supply and running the command again should also work unless screen dies. - -Running under stock firmware has not been performed. In particular, the default configuration on RaspberryPi 3 seems to map other UART than used by the kernel (so-called miniUART) to pins 6, 8 and 10. This is supposed to be configurable through the use of overlays. -## Makefile -To maintain order, all files created with the use of make, that is binaries, object files, natively executed helper programs, etc. get placed in build/. - -Our project contains 2 Makefiles: one in it's root directory and one in build/. The reason is that it is easier to use Makefile to simply, elegantly and efficiently produce files in the same directory where it is. To produce files in directory other than Makefile's own, it requires this directory to be specified in many rules across the Makefile and in general it complicates things. Also, a problem arises when trying to link objects not from within the current directory. If an object is referenced by name in linker script (which is a frequent practice in our scripts) and is passed to gcc with a path, then it'd need to also appear with that path in the linker script. -Because of that a Makefile in build/ is present, that produces files into it's own directory and the Makefile in project's root is used as a proxy to that first one - it calls make recursively in build/ with the same target it was called with. These changes makes it easier to read. - -From now on only Makefile in build/ will be discussed. - -In the Makefile, variables with the names of certain tools and their command line flags are defined (using =? assignment, which allows one to specify their own value of that variable on the command line). In case a cross-compiler with a different triple should be used, ARM_BASE, normally set to arm-none-eabi, can be set to something like arm-linux-gnueabi or even /usr/local/bin/arm-none-eabi. - -All variables discussed below are defined using := assignment, which causes them to only be evaluated once instead of on every reference to them. - -Objects that should be linked together to create each of the .elf files are listed in their respective variables. I.e. objects to be used for creating kernel_stage2.elf are all listed in KERNEL_STAGE2_OBJECTS. When adding a new source file to the kernel, it is enough to add it's respective .o file to that list to make it compile and link properly. No other Makefile modifications are needed. -In a similar fashion, RAMFS_FILES variable specifies files, that should be put in the ramfs image, that will be embedded in the kernel. Adding another file only requires listing it there. However, if the file is to be found somewhere else that build/, it might be useful to use the vpath directive to tell make where to look for it. - -Variables dirs and dirs_colon are defined to store list of all directories within src/, separated with spaces and colons, respectively. dirs_colons are used for vpath directive. 'dirs' variable is used in ARM_FLAGS to pass all the directories as include search paths to gcc. empty and space are helper variables - defining dirs_colon could be achieved without them (but it's clearer this way). - -The vpath directive tells make to look for assembler sources, C sources and linker scripts in all direct and indirect subdirectories of src/ (including itself). All other files shall be found/created in build/. - -### Targets - -The default target is the binary image of the kernel. - -The generic rule for compiling C sources uses cross-compiler or native compiler with appropriate flags depending on whether the source file is located somewhere under arm/ directory (which lies in src/) or enywhere else. - -The generic rules for making a stripped binary image out of elf file, for assembling an assembly file, for making an arbitrary file a linkable object and for linking objects are ARM-only. - -In C world it is possible to embed a file in an executable by using objcopy to create an object file from it and then linking that object file into the executable. In this project, at the current time, this is used only for embedding ramfs in the kernel (incbin is used for embedding kernel and loader second stages in their first stages). Generic rule for making a binary image into object file is present, in case it is needed somewhere else again. - -To link elf files, the generic rule is combined with a rule that specifies the elf's objects. Objects are listed in variables whenever more than one of them is needed. - -At this point in the Makefile, the dependence of objects created from assembly on files referenced in the assembly source via incbin is marked. - -Simple ram filesystem is created from files it should contain with the use of our own simple tool - makefs. - -Another 2 rules specify how native programs (for the machine we're working on) are to be linked. - -### Aliased Rules - -Rule qemu-elf runs the kernel in qemu emulating RaspberryPi 2 with 256MiB of memory by passing the elf file of the kernel to the emulator. - -Rule qemu-bin does the same, but passes the binary image of the kernel to qemu. - -Rule qemu-loader does the same, but first passes the binary image of the bootloader to qemu and the actual kernel is piped to qemu's standard input, received by bootloader as uart data and run. This method currently makes it impossible to pass any keyboard input to kernel once it's running. - -Rule run-on-rpi pipes the kernel through uart, assuming it is available under /dev/ttyUSB0, and then opens a screen session on that interface. This allows for executing the kernel on the Pi connected through UART, provided that our bootloader is running on the board. - -Rule clean removes all the files generated in build/. - -Rules that don't generate files are marked as PHONY. -## Project structure -Directory structure of the project: - - doc/ - build/ - Makefile - Makefile - src/ - lib/ - rs232/ - rs232.c - rs232.h - host/ - pipe_image.c - makefs.c - arm/ - common/ - svc_interface.h - strings.c - io.h - io.c - strings.h - PL0/ - PL0_utils.h - svc.S - PL0_utils.c - PL0_test.c - PL0_test.ld - PL1/ - loader/ - loader_stage2.ld - loader_stage2.c - loader_stage1.S - loader.ld - kernel/ - demo_functionality.c - paging.h - setup.c - interrupts.h - interrupt_vector.S - kernel.ld - scheduler.h - atags.c - translation_table_descriptors.h - bcmclock.h - ramfs.c - kernel_stage1.S - paging.c - ramfs.h - interrupts.c - armclock.h - atags.h - kernel_stage2.ld - cp_regs.h - psr.h - scheduler.c - memory.h - demo_functionality.h - PL1_common/ - global.h - uart.h - uart.c - - -### Most Significant Directories - - doc/ - Contains documentation of the project. - - build/ - Contains main Makefile of the project. All objects created during the build process are placed there. - - Makefile - Proxies all calls to Makefile in build/. - - src/ - Contains all sources of the project. - - src/host/ - Contains sources of helper programs to be compiled using native GCC and run on the machine where development takes place. - - src/arm/ - Contains sources to be compiled using ARM cross-compiler GCC and run on the RaspberryPi. - - src/arm/common - Contains sources used in both: privileged mode and unprivileged mode. - - src/arm/PL0 - Contains sources used exclusively in unprivileged, user-mode (PL0) program, as well as the program's linker script. - - src/arm/PL1 - Contains sources used exclusively in privileged (PL1) mode. - - src/arm/PL1/loader - Contains sources used exclusively in the bootloader, as well as linker scripts for stages 1 and 2 of this bootloader. - - src/arm/PL1/kernel - Contains sources used exclusively in the kernel, as well as linker scripts for stages 1 and 2 of this kernel. - - src/arm/PL1/PL1_common - Contains sources used in both: kernel and bootloader. -## Boot Process -When RaspberryPi boots, it searches the first partition on SD card (which should be formatted FAT) for its firmware and configuration files, loads them and executes them. The firmware then searches for the kernel image file. The name of the looked for file can be kernel.img, kernel7.img, kernel8.img (for 64-bit mode) or something else, depending on configuration and firmware used (rpi-open-firmware looks for zImage). - -The image is then copied to some address and jumped to on all cores. Address should be 0x8000 for 32-bit kernel, but in reality is 0x2000000 in rpi-open-firmware and 0x10000 in qemu (version 2.9.1). 3 arguments are passed to the kernel: first (passed in r0) is 0; second (passed in r1) is machine type; third (passed in r2) is the address of FDT or ATAGS structure describing the system or 0 as default. - -PIs that support aarch64 can also boot directly into 64-bit mode. Then, the image gets loaded at 0x80000. We're not using 64-bit mode in this project. - -Qemu can be used to emulate RaspberryPi, in which case kernel image and memory size are provided to the emulator on the command line. Qemu can also load kernel in the form of an elf file, in which case its load address is determined based on information in the elf. - -Our kernel has been executed on qemu emulating RaspberryPi 2 as well as on real RaspberryPi 3 running rpi-open firmware (although not every functionality works everywhere). To quicken running new images of the kernel on the board, a simple bootloader has been written by us, which can be run from the SD card instead of the actual kernel. It reads the kernel image from uart, and executes it. The bootloader can also be used within qemu, but there are several problems with passing keyboard input to the kernel once it's running. - -Both bootloader and kernel are split into 2 stages. - -### Loader -In case of the loader it is due to the fact, that the the actual kernel read by it from UART is supposed to be written at 0x8000. If the loader also ran from 0x8000 or a close address, it could possibly overwrite it's own code while writing kernel to memory. To avoid this, the first stage of the loader first copies its second stage embedded in it to address 0x4000. Then, it jumps to that second stage, which reads kernel image from uart, writes it at 0x8000 and jumps to it. Arguments (r0, r1, r2) are preserved and passed to the kernel. Second stage of the bootloader is intended to be kept small enough to fit between 0x4000 and 0x8000. Atags structure, if present, is guaranteed to end below 0x4000, so it should not get overwritten by loader's stage2. - -The loader protocol is simple: first, size of the kernel is sent through UART (4 bytes, little endian). Then, the actual kernel image. Our program pipe_image is used to prepend kernel image with its size. -### Kernel -In case of kernel, it is desired to have image run from 0x0, because that's where the interrupt vector table is under default settings. This is also achieved by splitting it into 2 stages. -#### Stage 1 -Stage 1 is loaded at some higher address. It has second stage image embedded in it. It copies it to 0x0 and jumps to it. What gets more complicated compared to loader, is the handling of ATAGS structure. Before copying stage 2 to 0x0, stage 1 first checks if atags is present and if so, it is copied to some location high enough, that it won't be overwritten by stage 2 image. Whenever the memory layout is modified, it should be checked, if there is a danger of ATAGS being overwritten by some kernel operations before it is used. In current setup, new location chosen for ATAGS is always below the memory later used as the stack and it might overlap memory later used for translation table, which is not a problem, since kernel only uses ATAGS before filling that table. - -When stage 1 of the kernel jumps to second stage, it passes modified arguments: first argument (r0) remains 0 if ATAGS was found and is set to 3 to indicate, that ATAGS was not found. Second argument (r2) remains unchanged. Third argument (r2) is the current address of ATAGS (or remains unchanged if no ATAGS was found). -If support for FDT is added in the future, it must also be done carefully, so that FDT doesn't get overwritten. -#### Stage 2 -At the start of the stage 2 of the kernel, there is the interrupt vector table. It's first entry is the reset vector, which is not normally unused. In our case, when stage 1 jumps to 0x0, first instruction of stage 2, it jumps to that vector, which then calls the setup routine. - -### Some more notes -In both loader and the kernel, at the beginning of stage1 it is ensured, that only one ARM core is executing. - -It's worth noting, that in first stages the loop that copies the embedded second stage is intentionally situated after the blob in the image. This way, this loop will not overwrite itself with the data it is copying, since the stage 2 is always copied to some lower address. It copies to 0x0 in case of kernel and to 0x4000 in case of loader - we assume stage 1 won't be loaded below 0x4000. - -Qemu, stock RaspberryPi firmware and rpi-open-firmware all load image at different addresses. Although stock firmware is not used in this project, our loader loads kernel at 0x8000, where the stock firmware would. Because of that, it is desired, that image is able to run, regardless of where it was loaded at. This was realized by writing first stages of loader and kernel in careful, position-independent assembly. The starting address in corresponding linker scripts is irrelevant. The stage 2 blobs are embedded using .incbin assembly directive. Second stages are written normally in C and compiled as position-dependent for their respective addresses. - -## MMU -Here's an explanation of steps we did to enable the MMU and how the MMU works in general. - -MMU stands for Memory Management Unit. It does 2 important things: - -1. It allows programs to use virtual memory addressing. Virtual addresses are translated by the MMU to physical addresses with the help of translation table. -2. It guards against unallowed memory access. Element that only implements this functionality is called MPU (Memory Protection Unit) and is also found in some ARM cores. - -Without MMU code executing on a processor sees the memory as it really is. - -When it tries to load data from address 0x00AA0F3C it indeed loads data from 0x00AA0F3C. This doesn't mean address 0x00AA0F3C is in RAM: RAM can be mapped into the address space in an arbitrary way. - -MMU can be configured to "redirect" some range of addresses to some other range. Let's assume we configured the MMU to translate address range 0x00A00000 - 0x00B00000 to range 0x00200000 - 0x00300000. Now, code trying to perform operation on address 0x00AA0F3C would have the address transparently translated to 0x002A0F3C, on which the operation would actually take place. - -The translation affects all (stack and non-stack) data accesses as well as instruction fetches, hence an entire program can be made to work as if it was running from some memory address, while in fact it runs from a different one! - -The addresses used by program code are referred to as virtual addresses, while addresses actually used by the processor - as physical addresses. - -This aids operating system's memory management in several ways - -1. A program may by compiled to run from some fixed address and the OS is still free to choose any physical location to store that program's code - only a translation of program's required address to that location's address has to be configured. A problem of simultaneous execution of multiple programs compiled for the same address is also avoided in this way. -2. A consecutive memory region might be required by some program. For example: due to earlier allocations and deallocactions there isn't a big enough (no pun intended) free consecutive region of physical memory. Smaller regions can be mapped to become accessible as a single region in virtual address space, thus avoiding the need for defragmentation. - -A given mapping can be made valid for only one execution mode (i.e. region only accessible from privileged mode) or only certain types of accesses . A memory region can be made non-executable, which guards against accidental jumping there by program code. That is important for countering buffer-overflow exploits. An unallowed access triggers a processor exception, which passes control to an appropriate interrupt service routine. - - -In RaspberryPi environments used by us, there are ARMv7-A compatible processors, which we currently use only in 32-bit mode. Information here is relevant to those systems (there are Pi boards with both older and newer processors, with more or less functionality and features available). - -If MMU is present, general configuration of it is done through registers of the appropriate coprocessor (cp15). Translations are managed through translation table. It is an array of 32-bit or 64-bit entries (also called descriptors) describing how their corresponding memory regions should be mapped. A number of leftmost bits of a virtual address constitutes an index into the translation table to be used for translating it. This way no virtual addresses need to be stored in the table and MMU can perform translations in O(1) time. - -### Coprocessor 15 -Coprocessor 15 contains several registers, that control the behaviour of the MMU. They are all accessed through mcr and mrc arm instructions. - -1. SCTLR, System Control Register - "provides the top level control of the system, including its memory system". Bits of this register control, among other things, whether the following are enabled: - - 1. the MMU - 2. data cache4. TEX remap - 3. instruction cache - 4. TEX remap (changes how some translation table entry bit fields (called C, B and TEX) are used - not in the project) - 5. access flags (enabling causes one translation table descriptor bit normally used to specify access permissions of a region to be used as access flag - not used either) - -2. DACR, Domain Access Control Register - "defines the access permission for each of the sixteen memory domains". Entries in translation table define which of available 16 memory domains a memory region belongs to. Bits of DACR specify what permissions apply to each of the domains. Possible settings are to allow accesses to regions based on settings in translation table descriptor or to allow/disallow all accesses regardless of access permission bits in translation table. - -3. TTBR0, Translation Table Base Register 0 - "holds the base address of translation table 0, and information about the memory it occupies". System mode programmer can choose (with respect to some alignment requirements) where in the physical memory to put the translation table. Chosen address (actually, only a number of it's leftmost bits) has to be put in TTBR for the MMU to know where the table lies. Other bits of this register control some memory attributes relevant for accesses to table entries by the MMU - -3. TTBR1, Translation Table Base Register 1 - simillar function to TTBR0 (see below for explaination of dual TTBR) -4. TTBCR, Translation Table Base Control Register, which controls: - - 1. How TLBs (Translation Lookaside Buffers) are used. TLBs are a mechanism of caching translation table entries. - 2. Whether to use some extension feature, that changes traslation table entries and TTBR* lengths to 64-bit (we're not using this, so we won't go into details) - 3. How a translation table is selected. - -There can be 2 translation tables and there are 2 cp15 registers (TTBR0 and TTBR1) to hold their base addresses. When 2 tables are in use, then on each memory access some leftmost bits of virtual address determine which one should be used. If the bits are all 0s - TTBR0-pointed table is used. Otherwise - TTBR1 is used. This allows OS developer to use separate translation tables for kernelspace and userspace (i.e. by having the kernelspace code run from virtual addresses starting with 1 and userspace code run from virtual addresses starting with 0). A field of TTBCR determines how many leftmost bits of virtual address are used for that (and also affects TTBR0 format). In the simplest setup (as in our project) this number is 0, so only the table specified in TTBR0 is used. - -### Translation table - -Translation table consists of 4096 entries, each describing a 1MB memory region. An entry can be of several types: - -1. Invalid entry - the corresponding virtual addresses can not be used -2. Section - description of a mapping of 1MB memory region -3. Supersection - description of a mapping of 16MB memory region, that has to be repeated 16 times in consecutive memory sections . This can be used to map to physical addresses higher than 2^32. -4. Page table - no mapping is given yet, but a page table is pointed. See below. - -Besides, translation table descriptor also specifies: - -1. Access permissions. -2. Other memory attributes (cacheability, shareability). -3. Which domain the memory belongs to. - -### Page Table - -Page table is something simillar to translation table, but it's entries define smaller regions (called, well - pages). When a translation table descriptor describing a page table gets used for translation, then entry in that page table is fetched and used along with some middle bits of the virtual address used as index. This allows for better granularity of mappings, as it doesn't require the page tables to occupy space if small pages are not needed. We could say, that 2-level translations are performed. On some versions of ARM translations can have more levels than that. This means the MMU might sometimes need to fetch several entries from different level tables to compute the physical address. This is called a translation table walk. - -As of 15.01.2020 page tables and small pages are not used in the project (although programming them is on the TODO list). - -### Project specific info - -Despite the overhelming amount of configuration options available, most can be left deafult and this is how it's done in this project. Those default settings usually make the MMU behave like it did in older ARM versions, when some options were not yet available and hence, the entire system was simpler. - -Our project uses C bitfield structs for operating on SCTLR and TTBCR contents and translation table descriptors. With DACR - bit shifts are more appropriate and with TTBCR - our default configuration means we're writing '0' to that register. This is an elegant and readable approach, yet little-portable across compilers. Current struct definitions work properly with GCC. - -Structs describing SCTLR, DACR and TTBCR are defined in src/arm/PL1/kernel/cp_regs.h. -Structs describing translation table descriptors are defined in src/arm/PL1/kernel/translation_table_descriptors.h. - -Before the MMU is enabled, all memory is seen as it really is. Therefore, the only feasible way of enabling it is by initially setting the descriptors in translation table to map all addresses (mapping just addresses used by the kernel would be enough) to themselves. It is called a flat map. - -### Setting up MMU and FlatMap - -How setting up a flat map and turning on the MMU and management of memory sections is done in our project: - -1. Translation table is defined in the linker script src/arm/PL1/kernel/kernel_stage2.ld as a NOLOAD section. C code gets the table's start and end addresses from symbols defined in that linker script (see arm/PL1/kernel/memory.h). -2. Function setup_flat_map() defined in arm/PL1/kernel/paging.c enables MMU with a flat map. It prints relevant information to uart while performing the following procedure: - - 1. In a loop write all descriptors to the translation table, set them as sections, accessible from PL1 only, belonging to domain 0. - 2. Set DACR to allow domain 0 memory accesses, based on translation table descriptor permissions and block accesses to other domains, as only domain 0 is used in this project. - 3. Make sure TEX remap, access flag, caches and the MMU are disabled in SCTLR. Disabling some of them might be unnecessary, because MMU is assumed to be disabled from the start and enabled caches might cause no problems as long as only flat map is used. Still, the way it is done right now is known to work well and optimizations are not needed. - 4. Clear all caches and TLBs (again, it is suspected that some of this is unnecessary). - 5. Write TTBCR setting such that only 32-bit translation table is used. - 6. Make TTBR0 point to the start of translation table. Rest of attributes in TTBR0 (concerning how table entries are being accessed) are left as 0s (defaults). - 7. Enable the MMU and caches by setting the appropriate bits in SCTLR. - -After some cp15 register writes, the isb assembly instruction is used, which causes ARM core to wait until changes take effect. This is done to prevent some later instructions from being executed before the changes are applied. - -In arm/PL1/kernel/paging.c the function claim_and_map_section() can be used to modify an entry in translation table to create a new mapping. Memory allocation also done in that source file uses some lists to describe free and taken sections, but has nothing to do with with the MMU. -## Program Status Register -CPSR (Current Program Status Register) is a register, bits of which contain and/or determine various aspects of execution, i.e. condition flags, execution state (arm, thumb or jazelle), endianness state, [execution mode](./link_to_doc_explaining_modes) and interrupt mask. This register is readable and writeable with the use of mrs and msr instructions from any PL1 mode, thus it is possible to change things like mode or interrupt mask by writing to this register. - -Additionally, there are other registers with the same or simillar bit fields as CPSR. Those PSRs (Program Status Registers) are: - -1. APSR (Application Program Status Register) -2. SPSRs (Saved Program Status Registers) - -APSR is can be considered the same as CPSR or a view of CPSR, with some limitations - some bit fields from CPSR are missing (reserved) in APSR. APSR can be accessed from PL0, while CPSR should only be accessed from PL1. This was an application program executing in user mode can learn some of the settings in CPSR without accessing CPSR directly. - -SPSR is used for exception handling. Each exception-taking mode has it's own SPSR (they can be called SPSR_sup, SPSR_irq, etc.). On exception entry, old contents of CPSR are backed up in entered mode's SPSR. Instructions used for exception return (subs and ldm ^), when writing to the pc, have the important additional effect of copying the SPSR to CPSR. This way, on return from an exception, processor returns to the state from before the exception. That includes endianess settings, execution state, etc. - -In our project, the structure of PSRs is defined in terms of C bitfield structs in src/arm/PL1/kernel/psr.h. -#Ramfs - -A simple ram file system has been introduced to avoid having to embed too many files in the kernel in the future. - -The ram filesystem is created on the development machine and then embedded into the kernel. Kernel can then parse the ramfs and access files in it. - -Ramfs contains a mapping from file's name to it's size and contents. Directories, file permissions, etc. as well as writing to filesystem are not supported. - -Currently this is used to access the code of PL0 test program by the kernel, which it then copies to the appropriate memory location. In case more user mode programs are later written, they can all be added to ramfs to enable the kernel to access them easily. - -## Specification -When ramfs is accessed in memory, it MUST be aligned to a multiple of 4. - -The filesystem itself consists of blocks of data, each containing one file. Blocks of data in the ramfs come one after another, with the requirement, that each block starts at a 4-aligned offset/address. If a block doesn't end at a 4-aligned address, there shall be up to 3 null-bytes of padding after it, so that the next block is properly aligned. - -Each block start with a C (null-terminated) string with the name of the file it contains. At the first 4-aligned offset after the string, file size is stored on 4 bytes in little endian. Null-bytes are used for padding between file name and file size if necessary. Immediately after the file size reside file contents, that take exactly the amount of bytes specified in file size. - -As obvious from the specification, files bigger than 4GB are not supported, which is not a problem in the case of this project. - -## Implementations - -Creation of ramfs is done by the makefs program (src/host/makefs.c). The program accepts file names as command line arguments, creates a ramfs containing all those files and writes it to stdout. As makefs is a very simple tool (just as our ramfs is a simple format), it puts files in ramfs under the names it got on the command line. No stripping or normalizing of path is performed. In case of errors (i.e. io errors) makefs prints information to stderr and exits. - -Parsing/reading of ramfs is done by a kernel driver (src/arm/PL1/kernel/ramfs.c). The driver allows for finding a file in ramfs by name. File size and pointers to file name string and file contents are returned through a structure from function find_file. - -As ramfs is embedded in kernel image, it is easily accessible to kernel code. The alignment of ramfs to a multiple of 4 is assured in kernel's linker script (src/arm/PL1/kernel/kernel_stage2.ld). -## Exceptions -Whenever some illegal operation (attempt to execute undefined instruction, attempt to access memory with insufficient permission, etc.) happens or some peripheral device "messages" the ARM core, that something important happened, an exception occurs. Exception is something, that pauses normal execution and passes control to the (specific part of) operating system. -Upon an exception, several things happen: - -1. Change of [Processor mode](./processor-modes-explained.txt). -2. CPSR gets saved into new mode's [SPSR](./PSRs-explained.txt). -3. pc (incremented by some value) is saved into new mode's lr. -4. Execution jumps to an entry in the exception vectors table specific to the exception. - -Each exception type is taken to it's specific mode. Types and their modes are: - -1. Reset and supervisor mode. -2. Undefined instruction and undefined mode. -3. Supervisor call and supervisor mode. -4. Prefetch abort and abort mode. -5. Data abort and abort mode. -6. Hypervisor trap and hypervisor mode (not used normally, only with extensions). -7. IRQ and IRQ mode. -8. FIQ and FIQ mode. - -The new value of the pc (the address, to which the exception "jumps") is the address of nth instruction from exceptiom base address, which, under simplest settings, is 0x0 (bottom of [virtual address space](./MMU-explained.txt)). N depends on the exception type. It is: - -1. reset -2. undefined instruction -3. supervisor call -4. prefetch abort -5. data abort -6. hypervisor trap (not used here) -7. IRQ -8. FIQ - -Those 8 instructions constitute the exception vectors table. As the instruction follow one another, each of them should be a branch to some exception-handling routine. In fact, on other architectures often the exception vector table holds raw addresses of where to jump instead of actual instructions, as here. - -Bottom of virtual address space can be changed to some other value by manipulating the contents of SCTLR and VBAR coprocessor registers. - -On exception entry, the registers r0-r12 contain values used by the code that was executing before. In order for the exception handler to perform some action and return to that code, those registered can be preserved in memory. Some compilers can automatically generate appropriate prologue and epilogue for handler-functions, that will preserve the right registers (see <link to some gcc doc about it>; we're not using this feature in our project). - -Having old CPSR in SPSR and old pc in lr is helpful, when after handling the exception, the handler needs to return to the code that was executing before. There are 2 special instructions, subs and ldm ^ (load multiple with a dash ^), that, when used to change the pc (and therefore perform a jump) cause the SPSR to be copied into CPSR. As bits of CPSR determine the current execution mode, this causes the mode to be change to that from before the exception. In short, subs and ldm ^ are the instructions to use to return from exceptions. - -As noted eariler, upon exception entry an incremented value of pc is stored in lr. By how much it is incremented, depends on exception type and execution state. For example, entering undefined instruction exception for thumb state places in undef's lr the problematic instruction's address + 2, while taking this exception from ARM state places in undef's lr that instruction's address + 4 (see full table in paragraph B1.8.3 of [armv7-ar_arm](https://static.docs.arm.com/ddi0406/c/DDI0406C_C_arm_architecture_reference_manual.pdf)). - -It's worth noting, that while our [implementation of exception handlers](src/arm/PL1/kernel/interrupt_vector.S) also sets the stack pointer (sp) upon each exception entry, a kernel could be written, where this wouldn't be done, as each mode enterable by exception has it's own sp. -## IRQ -2 of possible exceptions in ARM are IRQ (Interrupt Request) and FIQ (Fast Interrupt Request). The can be caused by external source, such as peripheral devices and they can be used to inform the kernel about some action, that happened. - -Interrupts offer an economic way of interacting with peripheral devices. For example, code can probe UART memory-mapped registers in a loop to see whether transmitting/receiving of a character finished. However, this causes the processor needlessly execute the loop and makes it impossible or difficult to perform another tasks at the same time. Interrupt can be used instead of probing to "notify" the kernel, that something it was waiting for just happened. While waiting for interrupt, the system can be put to halt (i.e. wfi instruction), which helps save power, or it can perform other actions without wasting processor cycles in a loop. - -An interrupt, that is normally IRQ, can be made into FIQ by ARM system dependent means. FIQ is meant to be able to be handled faster, by not having to back up registers r8-r12, that FIQ mode has it's own copies of. This project only uses IRQ. - -Some peripheral devices can be configured (through their memory-mapped registers) to generate an interrupt under certain conditions (i.e. UART can generate interrupt when received characters queue fills). The interrupt can then be either masked or unmasked (sometimes in more than one peripheral register). If interrupts are enabled in CPSR and a peripheral device tries to generate one, that is not masked, IRQ (or FIQ) exception occurs (which causes interrupts to be temporarily masked in CPSR). The code can usually check, whether an interrupt of given kind from given device is **pending**, by looking at the appropriate bit of the appropriate peripheral register (mmio). As long as an interrupt is pending, re-enabling interrupts (for example via return from IRQ handler) shall cause the exception to occur again. Removing the source of the interrupt (i.e. removing characters from UART fifo, that filled) doesn't usually cause the interrupt to stop pending, in which case a pending-bit has to be cleared, usually by writing to the appropriate peripheral register (mmio). - -IRQs and FIQs can be configured as vectored - the processor then, upon interrupt, jumps to different location depending on which interrupt occured, instead of jumping to the standard [IRQ/FIQ vector](./Exception-vector-explained.txt). This can be used to speed up interrupt handling. Our simple project does not, however, use this feature. - -Currently, IRQs from 2 sources are used: [ARM timer IRQ](https://www.raspberrypi.org/app/uploads/2012/02/BCM2835-ARM-Peripherals.pdf) and UART IRQs. The kernel makes sure, that timer IRQ only occurs when processor is in user mode. IRQ handler does not return in this case - it calls scheduler. The kernel makes sure, that UART IRQ only occurs, when a process is blocked and is waiting for UART IO operation. The interrupt handler, when called, checks what type of UART action happened and tries (through calling of appropriate function from scheduler.c) to handle that action and, possibly, to unblock the waiting process. UART IRQ might occur when another process is executing (not possible now, with only one process, but shall be possible when more processes are added to the project), in which case it the handler returns, or when kernel is explicitly waiting for interrupts (because all processes are blocked), in which case it calls schedule() instead of returning. -# Processor modes - -ARMv7-A core can be executing in one of several modes (not to be confused with instruction set states or endianness execution state). Those are: - -1. User -2. FIQ -3. IRQ -4. Supervisor -5. Abort -6. Undefined -7. System - -In fact, there are more if the processor implements some extensions, but this is irrelevant here. - -Current processor mode is encoded in the lowest five bits of the [CPSR](./PSRs-explained.txt) register. - -Processor can operate in one of 2 privilege levels (although, again, extensions exist, that add more levels): - -1. PL0 - privilege level 0 -2. PL1 - privilege level 1 - -Processor modes have their assigned privilege levels. User mode has privilege level 0 and all other modes have privilege level 1. Code executing in one of privileged modes is allowed to do more things, than user mode code, i.e. writing and reading some of the coprocessor registers, executing some privileged instructions (i.e. mrs and msr, when used to reference CPSR, as well as other modes' registers), accessing privileged memory and changing the mode (without causing an interrupt). Attempts to perform those actions in user mode result either in undefined (within some limits) behaviour or an exception (depending on what action is considered). - -User mode is the one, in which application programs usually run. Other modes are usually used by the operating system's kernel. Lack of privileges in user mode allows PL1 code to control execution of PL0 code. - -While code executing in PL1 can freely (except switching from system to user mode, which produces undefined behaviour) change mode by either writing the CPRS or executing cps instruction, user mode can only be exitted by means of an interrupt. - -Some ARM core registers (i.e. r0 - r7) are shared between modes, while some are not. In this case, separate modes have their private copies of those registers. For example, lr and sp in supervisor mode are different from lr and sp in user mode. For full information about shared and not shared (banked) registers, see paragraph B9.2.1 in [armv7-a manual](https://static.docs.arm.com/ddi0406/c/DDI0406C_C_arm_architecture_reference_manual.pdf). The most important things are that user mode and system mode share all registers with each other and they don't have their own SPSR (which is used for returning from [exceptions](./Exception-vector-explained.txt) and exceptions are never taken to those 2 modes) and that all other modes have their own SPSR, sp and lr. - -The reason for having multiple copies of the same register in different modes is that it simplifies writing interrupt handlers. I.e. supervisor mode code can safely use sp and lr without destroying the contents of user mode's sp and lr. - -The big number of PL1 modes is supposed to aid in handling of interrupts. Each kind of interrupt is taken to it's specific mode, as detailed in [here](./Exception-vector-explained.txt). - -Supervisor mode, in addition to being the mode supervisor calls are taken to, is the mode the processor is in when the kernel boots. - -System mode, which uses the same registers as user mode, is said to have been added to ARM architecture to ease accessing the unprivileged registers. For example, setting user mode's sp from supervisor mode can be done by switching to system mode, setting the sp and switching back to supervisor mode. Other modes' registers can alternatively be accessed with the use of mrs and msr assembly instructions (but not from user mode). - -Despite the name, system mode doesn't have to be the mode used most often by operating system's kernel. In fact, prohibition of direct switching from system mode to user mode would make extensive use of system mode impractical. This project, for example, uses supervisor mode for most of the privileged tasks. -## Scheduler -An operating system has to manage user processes. Our system only has one process right now, but usual actions, such as context saving or context restoring, are implemented anyways. The following few paragraphs contain information on how process management looks like in operating systems in general. - -Process might return control to the system by executing the svc (eariler called swi) instruction. System would then perform some action on behalf of the process and either return from the supervisor call exception or attempt to schedule another process to run, in which case context of the old process would need to be saved for later and context of the new process would need to be restored. - -Process has data in memory (such as it's stack, code) as well as data in registers (r0-r15, CPSR). Together they constitute process' context. From process' perspective, context should not unexpectedly change, so when control is taken away from user mode code (via an exception) and later (possibly after execution of some other processes) given back, it should be transparent to the process (except when kernel does something for the process in terms of supervisor call). In particular, the contents of core registers should be the same as before. For this to be achievable, the operating system has to back up process' registers somewhere in memory and later restore them from that memory. - -Operating system kernel maitains a queue of processes waiting for execution. When a process blocks (for example by waiting for IO), it is removed from the queue. If a process unblocks (for example because IO completed) it is added back to the queue. In general, some systems might complicate it, for example by having more queues, but discussing those variations is out of scope of this documentation. When processor is free, one of the processes from the queue (determined by some scheduling algorithm <link to wikipedia??> implemented in the kernel) gets chosen and run on the processor. - -As one process could never use a supervisor call, it could occupy the processor forever. To remedy this, timer interrupts can be used by the kernel to interrupt the execution of a process after some time. The process would then have it's context saved and go to the end of the queue. Another process would be scheduled to run. - -Other exceptions might occur when process is running. Depending on kernel design, handler of an exception (such as IRQ) might return to the process or cause another one to be scheduled. - -If at some time all processes are blocked waiting, the kernel can wait for some interrupt to happen, which could possibly unblock some process (i.e. because IO completed). - -While not mentioned earlier, switching between processes' contexts involves not only saving and restoring of registers, but also changing the translation table entries to properly map memory regions used by current process. - -In our project, process management is implemented in src/arm/PL1/kernel/scheduler.c. - -A "queue" contains data of the only process (variables PL0_regs[], PL0_sp, PL0_lr and PL0_PSR). - -### Functions -Function setup_scheduler_structures is supposed to be called before scheduler is used in any way. - -Function schedule_new() creates and runs a new process. - -Function schedule_wait_for_output() causes the current process to have it's context saved and get blocked waiting for UART to send data. It is called from supervisor call handler. Function schedule_wait_for_input() is similar, but process waits for UART to receive data. - -Function schedule() attempts to select a process (currently the only one) and run it. If process cannot be run, schedule() waits for interrupt, that could unblock the process. The interrupt handler would not return in this case, but rather call schedule() again. - -Function scheduler_try_output() is supposed to be called by IRQ handler when UART is ready to transmit more data. It can cause a process to get unblocked. scheduler_try_input() is simillar, but relates to receiving data. - -The following are assured in our design: - -1. When processor is in user mode, interrupts are enabled. -2. When processor is in system mode, interrupts are disabled, except when explicitly waiting for the interrupt when process is blocked. -3. When a process is waiting for input/output, the corresponding IRQ is unmasked. Otherwise, that IRQ is masked. -4. If an interrupt from UART occurs during execution of user mode code (not possible here, as we only have one process, but shall become possible when proper processes are implemented), the handler shall return. If that interrupt occurs during execution of PL1 code, it means it occured in scheduler, that was implicitly waiting for it and the handler calls scheduler() again instead of returning. -5. Interrupt from timer is unmasked and set to come whenever a process gets scheduled to run. Timer interrupt is disabled when in PL1 (when scheduler is waiting for interrupt, only UART one can come). -6. A supervisor call requesting an UART operation, that can not be completed immediately, causes the process to block. -## Linking - -Linking is a process of creating an executable, library or another object file out of object files. [wikipedia](https://en.wikipedia.org/wiki/Linker_%28computing%29) -During linking, values previously unknown to the compiler (i.e. what will be the addresses of external functions/variables, from what address will the code be executing) might be injected into the code. - -Linker script is, among others, used to tell the linker, where in memory the specific parts of the executable should lie. - -In a hosted environment (when building a program to run under an full-featured operting system, like GNU/Linux), a linker script is usually provided by the toolchain and used if no other script is provided. In a bare-metal project, the developer usually has to write their own linker script, in which they specify the binary image's **load address** and section layout. - -Contents of an object code file or executable (our .o or .elf) are grouped into sections. Sections have names. Common named are .text (usually contains code), .data (usually contains statically-allocated variables initialized to non-zero values), .bss (usually used to reserve memory for statically allocated variables initialized to zero), .rodata (usually contains statically-allocated variables, that are not going to be modified). - -In a hosted environment, when an executable (say, of elf format) is executed, contents of it's sections are usually placed in different memory segments with different access privileges, so that, for example, code is not writable and variable contents are not executable. This helps reduce the risk of buffer overflow exploits. - -In a bare-environment like ours, we don't execute an elf file directly (except in qemu, which is the unpreferred approach anyway), but rather a raw binary image created from an elf file. Still, the notion of section is used along the way. - -During link, one or more object code files are combined into one file (in our case an executable). Section contents of input files land in some sections of the output file, in a way defined in the linker script. In a hosted environment, a linker script would likely put contents of input .text sections in a .text section, contents of input .data sections in a .data section, etc. The developer can, however, use sections with different names (although weird behaviour of some linkers might occur) and assign their contents in their preferred way using a linker script. - -In linker script it is possible to specify a section as NOLOAD (usually used for .bss), which, in our case, causes that section not to be included in the binary image later created with objcopy. - -It is also possible to treat same-named input sections differently depending on what file they came from and even use wildcards when specifying file names. - -Variables can be created, as well as new symbols, which can then be references from C code. - -Defining alignment of specific parts of future image is also easily achievable. - -We made use of all those possibilities in our scripts. - -In src/arm/PL1/kernel/kernel_stage2.ld the physical memory layout of thkernel is defined. Symbols defined there, such as _stack_end, are referenced in C header src/arm/PL1/kernel/memory.h. - -While src/arm/PL1/kernel/kernel.ld and src/arm/PL1/loader/loader.ld define the starting address, it is irrelevant, as the assembly-written position-independent code for [first stages of loader and kernel](./Boot_explained.txt) does not depend on that address. - -At the beginning of this project, we had very little understanding of linker scripts' syntax. [This article](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/sections.html#OUTPUT-SECTION-DESCRIPTION) proved useful and allowed us to learn the required parts in a short time. As discussing the entire syntax of linker scripts is beyond the scope of this documentation, we refer the reader to that resource. -## Problems we faced - -Problems we've faced (I mentioned to Metal (is it ok to use that pseudonym? I think it's ok), I'd include those and he seemed very enthusiastic about us describing them) (+ some stuff qualifiable as probles is already in HISTORY.md) - -* Our ramfs needs to be 4-aligned in memory, but when objcopy creates the embeddable file, it doesn't (at least by default) mark it's data section as requiring 2**2 alignment. There has to be .=ALIGN(4) in linker script before ramfs_embeddable.o, but I forgot about it, which caused the ramfs to misbehave -* VERY NAUGHTY PROBLEM · Many sources mention /COMMON/ as the section, that contains some specific kind of uninitialized (0-initialized) data. Obviously, it has to be included in the linker script. Unfortunately, gcc names it differently, mainly - /COM/. This caused our linker script to not include it in the image. Instead, it was placed somewhere after the last section specified in the linker script. This happened to be after our NOLOAD stack section, where first free MMU section is (which happens to always get allocated to the first process, which gets it's code copied there). Do You imagine sitting for hours in front of radare2, searching for a bug in scheduler.c or PL0_test.c, that causes the userspace code to fail with either some kind of weird abort or undefined instruction, always on the second PL0 instruction!? -* VERY NAUGHTY PROBLEM · I wanted to make our bootloader and kernel able to run no matter what address they are loaded at (see comment in kernel's stage1 linker script). To achieve that, I added -fPIC to compilation options of all arm code. With this, I decided I can, instead of embedding code in other code using objcopy, just put that code in separate linker script section with section_start and section_end symbols defined, so that I can copy it to some other address in runtime. I did it and it worker with interrupt vector and libkernel (see point below). But once I changed EVERYTHING to use linker symbols/sections instead of objcopy embedding it turned out it doesn't really work... and I had to make it back the old way :( The thing is -fPIC requires code to be loaded by some os or bootloader, that will fill the global offset table with symbols. I knew it's possible to generate bare-metal position-independent code, that shall work without got, but it turned out this is not implemented in gcc (it is in arm compiler, but only in 32-bits and who would like to use arm compiler anyway). I ended up writing stage1 of both bootloader and the kernel in careful position-independent assembly, thus achieving my goal (jut with a bit more of effort). -* Linker behaves weird when section names don't start with .text, .data, etc. -* Not strictly a problem, but a funny mistake of mine, that is worth mentioning... At first I didn't know about special features of SUBS pc, lr and ldm rn {pc} ^ instructions. So I would switch to user mode by first branching to code in PL0-accessible section and having it execute isb instruction. This worked, but was not good, because code executed by the kernel was in memory section writable by userspace code. So i separated that into "libkernel", that would be in a PL0-executable but non-writable section and would perform the switch... Well, it did work. Still, I was happy when I learned how to achieve the same with subs/ldm and could remove libkernel, making the project a bit simpler. -* system mode has separate stack pointer from supervisor mode, so when going from supervisor to system we need to set it... We didn't know that and we were getting weeeird bugs (where changing something little in one place would make the bug occur or not occur somewhere completely else); also, it's not allowed (undefined behaviour) to switch from system mode directly to user mode... (at least this didn't cause such weird things to happen...) -* both bcm arm peripherals manual and the manual to uart itself say, that writing 0s to PL011_UART_IMSC unmasks interrupts; its the opposite: writing 1s enables specific interrupts and writing 0s disables them. wiki.osdev code also got it the way it's written in those docs, but this didn't cause problems, since uart irq was not enabled in ARM_ENABLE_IRQS_2 (using #define names from our code), so, as intended, no irq was occuring -* STILL UNFIXED · The very simple pipe_image program somehow manages to break stdin, so that even other programs run in that same (bash) shell can't read from it... (in zsh other interactively run commands work ok, but commands following pipe_image inside a shell function still have that problem) - -## Sources of Info - -* wiki.osdev (good for starting off, we could also (in a polite way) mention the things, that were broken there) - + getting uart irq masking wrong (not really their fault, see above) - + zeroing bss even tho it was placed in a section, that wasn't marked NOLOAD (I need to verify that yet) - + switch inside an enum!?!?!? - + There is already some more mentioned in HISTORY.md -* ARM Architecture Reference Manual® ARMv7-A and ARMv7-R edition (man, this has 2720 pages! But I think this was the most useful document of all) -* dwelch67 -* http://www.simtec.co.uk/products/SWLINUX/files/booting_article.html - very good description of atags -* BCM2835-ARM-Peripherals.pdf and https://elinux.org/BCM2835_datasheet_errata -* https://buildmedia.readthedocs.org/media/pdf/devicetree-specification/latest/devicetree-specification.pdf - perhaps You've found another source for that... but if not, this seems to be good! -* online ARM Compiler toolchain Assembler Reference -* Christina Brook's rpi-open-firmware -* http://infocenter.arm.com/help/topic/com.arm.doc.ddi0183g/DDI0183G_uart_pl011_r1p5_trm.pdf -* GNU make documentation -* description of linker scripts: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Using_ld_the_GNU_Linker/sections.html#OUTPUT-SECTION-DESCRIPTION @@ -5,5 +5,5 @@ rm -f document.md echo "//TODO insert [TOC] here" >> document.md for file in "${array[@]}" do - cat $file >> document.md + cat docs/$file >> document.md done |