difference between a segment and a page

In this thread, which was unfortunately closed, brewbuck states:

Quote:

---

Speaking of page tables and memory protection, some processors (e.g. x86) also have a memory protection mechanism called "segmentation" which uses segments instead of pages for access control. This shouldn't be confused with the kind of "segments" we are talking about here, although the two things are not entirely unrelated.

---

What's the difference between a segment and a page? And what kind of access controls do we have? Read/write? Anything else? How is memory protected?

Thanks, guys.

And a question of mine on segments but not related to cs32 is:

How does one obtain write access to the code segment, so that one may re-write parts of the generated machine language instructions at run-time? It just poped into my head when i read that the code segment gets protected by the OS...

The x86 architecture supports something called segments. A segment, as a basic principle, is a "chunk of memory". In the early x86, segments were introduced to allow the processor, that was 16-bit only, to access memory beyond 64k (2^16 == 64K). A segment base address gets added to the register value. In the early days, segments were only limited by the 64K architectural limit, and could start on any even 16-byte address in memory. The segment content was exactly the base-address shifted 4 bits down (2^4 = 16 - thus the limit of 16 byte alignment for the segments).

There are three kinds of segments in x86: CS (Code Segment), DS (Data Segment) and SS (Stack Segment). There is also a second data-segment register, ES, that can be used for example when copying from one segment to another.

To improve protection within the processor and at the same time extend the addressability of the processor, the 80286 introduced "Protected mode segments". In this mode, the segment cotnent isn't the base address of the segment, but an index into a "segment descriptor table". The segment descriptor table contains not only a base-address, but also the limit of the segment (how large the segment is). This means that a segment that is smaller than 64K can not overwrite into another segment. The segment base address was now 24 bits, giving a total address range of (a whopping) 16MB, but segments are still 64KB each (and registers are still 16 bit).

Then came the 386, which extends the processor registers to 32 bits. It's not much point in having 32-bit registers if the processor can't use 32 bits to address memory, so the descriptor entries also got an extension to 32-bit base address and a 20-bit limit that can either be "bytes" or "multiply by 4KB" - the latter giving full 32-bit segment size.

The 386 also introduces two new "spare" segment registers: FS, GS - they have the same sort of purpose as ES.

A segment has several attributes, such as "Writable", "Executable". It can also be "not present", so it's possible to use segments to swap in and out data from disk. There is also bits to indicate if the code is "user-mode" or "kernel mode" [actually, there is 4 levels, but most OS's would only use level 3 (user) and level 0 (kernel)]. Thus, each segment can be accessible by kernel or user-mode.

I'm sure this is clear as mud. A bit more in a bit.

Now for paging.
A page is a 4KB section of memory that is handled by the MMU (Memory Management Unit). It is another approach to protecting various bits of memory from being abused by apps that shouldn't be using that memory. The pages have attributes such as Read/Write, User/Kernel access. A page can be "present" or "not present". If the page is "not present", a "page fault" happens. This is used for two things: When the swapping process puts pages of memory to disk to make space for other apps/data, it marks that particular page as "not present". The other use is to mark memory inaccessible, so that for example NULL-pointers can be caught as "invalid memory access".

Page-tables are used to keep track of which 4KB section of memory is where, what attributes it has, etc. In the traditional case, a page-table-entry (PTE) is 32-bits, and part of that is the PHYSICAL address in memory, the remaining part is used for attributes.

In modern OS's page-tables are used in preference over segments. Segments still exist, because they are a basic component of the processor architecture, but the segment descriptors are there only because they must be, and the approach is to set all segment descriptors (CS, DS, SS, etc) to have base address of 0 and limit of 4GB. That gives all segments access to all of the memory, and no difference is made to what memory section is used for what purpose at the segment level.


As to xuftugulus question: Since the page-tables is what determines where you can write and where you can't, you have to ask the OS (nicely) if it will allow you write access to the code-section.

--
Mats

In Windows, you can actually ask VirtualAlloc to give you some pages where you can execute code.

Moved to Tech Board.

Quote:

---

Originally Posted by Elysia

In Windows, you can actually ask VirtualAlloc to give you some pages where you can execute code.

---

Yes, if you want to generate code at runtime, you can ask for pages that have execute attribute.

--
Mats

Sorry, but segments and pages are obsolete with the standard flat-memory model used now. I can't think why anyone would wish to learn that malformed and contorted system when flat memory model is so easy?

Pages very much exist still today.
Some functions, such as Window's VirtualAlloc allocates memory rounded to pages, for example.
And you get only access violations when writing to a page you don't own. That's why you won't get an access violation when writing one byte outside of your array.
This is relative to the x86, of course.

Quote:

---

Originally Posted by MacNilly

Sorry, but segments and pages are obsolete with the standard flat-memory model used now. I can't think why anyone would wish to learn that malformed and contorted system when flat memory model is so easy?

---

As Elysia says, pages are definitely still in use. That is how your Linux or Windows can run several applications that are all loaded at the same address, but in their own virtual space.

Yes, flat memory model, using the segments as I described above (base=0, limit=4GB) is the convention used by most OS's.

--
Mats

Ugh. I shudder to think of pages and segments. Damn that assembly language class. ;p

Normally, though, you don't really need to mess around with them unless you use specific functions. So it's fine not to learn them, I think.

Quote:

---

Originally Posted by MacNilly

Ugh. I shudder to think of pages and segments. Damn that assembly language class. ;p

---

Just wait 'til you try to understand the system of page-tables when you are using AMD Virtualization which has nested page-tables, where the virtual machine has it's page-table, but to make sure that the VIRTUAL machine doesn't clobber other virtual machines, the Hypervisor/VMM (Virtual Machine Monitor) has ANOTHER set of page-tables, so every page-table access of the virtual machine itself is also redirected through the page-table of the VMM. It gets a bit "interesting" to understand where the actual physical memory for a particular page in the virtual page that the guest is using.

--
Mats

There is something called the "NX" bit or the "XD" bit (depending on if it is AMD or Intel) that forbids executing code in certain segments of memory, but this is not always supported. In Windows it came online with XP SP2, but not all programs know about it or use it. The Linux 2.6 kernel supports it, but lots of times you have to compile a program specifically with the option specified that you want to use that feature.

Lots of times this is how a buffer overflow attack works. All you have to do is overwrite the return address in your stack frame, and then when the function returns, it will go to the new address that you wrote in memory. Voila! Now whatever is in that return address that it went to will be run as code. Hackers use this technique a lot in order to gain access and execute code on a compromised machine. But we shouldn't be talking about what hackers do....I just mentioned that because that is one way to execute code generated on the fly if you want to do so.

We have been talking about paging systems, virtual memory, and caches a lot in my operating systems class right now. There are ways of paging that are both software-based and hardware-based. The best option, obviously, is when you have hardware support to do paging.

Like matsp already mentioned, the is something called the MMU (Memory Management Unit). A virtual address gets passed to the MMU, and the MMU spits out the physical address which then gets passed to main memory. Each virtual address is split into two portions: a virtual page number and a page offset. So imagine you have some memory address. The high "n" bits represent a virtual page number, while the lower "p" bits represent an offset into that page. When the MMU does address translation, the page offset stays the same. The virtual page number, however, gets translated into a physical page number.

Address translation in the MMU happens via a page table. The MMU will index into the page table when it is given the virtual page number. The page table will say if that page is valid (in other words, if that page is currently in main memory....it could be that the page is not valid, or in other words, it is stored on disk).

Usually a hardware paging system will have a TLB (Translation Look-aside Buffer) so that it doesn't have to access main memory twice in order to access one thing. The TLB is a small cache on the chip which holds a small portion of the page table (recently accessed pages in memory). By doing this, most virtual addresses can be translated to physical addresses just by looking at the pages contained in the TLB, and we don't have to go to the physical memory to look at the full page table. If the page we want isn't in the TLB, then we have to go to physical memory and find the page in the page table there (that is one memory access), and then go to physical memory again to get the actual data we were looking for (that is a second memory access).

Then there are multilevel page tables which have been mentioned...but I won't go into.

Quote:

---

Originally Posted by DavidP

There is something called the "NX" bit or the "XD" bit (depending on if it is AMD or Intel) that forbids executing code in certain segments of memory, but this is not always supported. In Windows it came online with XP SP2, but not all programs know about it or use it. The Linux 2.6 kernel supports it, but lots of times you have to compile a program specifically with the option specified that you want to use that feature.

---

Yes, the NX/XD bit is bit 63 in a PAE-page-table entry (PAE is an extension to support more than 4GB of RAM in a 32-bit machine). This bit only means "don't execute here" if the correct bit is also set in another register (EFER, I think, or it's CR4) - and of course, only on machines that actually have this feature (all 64-bit AMD processors and most of the 64-bit Intel models support this).

The problem with some applications is that they do what I described earlier: generate code at run-time, which requires writable memory that is also executable. Applications that were developed before the OS support for "No Execute/Execute disable" got in there would of course just use the regular "malloc()" or some such to create memory space for execution, and then fail miserably when it tried to execute the code with the Execute Disable bit set...

--
Mats

--
Mats

Quote:

---

Originally Posted by matsp

The problem with some applications is that they do what I described earlier: generate code at run-time, which requires writable memory that is also executable. Applications that were developed before the OS support for "No Execute/Execute disable" got in there would of course just use the regular "malloc()" or some such to create memory space for execution, and then fail miserably when it tried to execute the code with the Execute Disable bit set...

---

With OS support, the processor could set up alternate page tables pointing to the same region of memory, but with different execute/write bits.

Failing that, on a POSIX-like operating system you can call mprotect() to cause the pages to become writable, write the generated code into them, then call mprotect() again to flip them back to read-only. That's a lot of thunking in and out of the kernel, though.

Quote:

---

Originally Posted by matsp

As Elysia says, pages are definitely still in use. That is how your Linux or Windows can run several applications that are all loaded at the same address, but in their own virtual space.

Yes, flat memory model, using the segments as I described above (base=0, limit=4GB) is the convention used by most OS's.

---

On that note, the NX problem could always be solved all the way as early as the 80386, but you have to use the segmentation architecture to do it. Under segmentation, only code within a "code segment" can execute. Since segments are stupid when compared with paging, nobody ever took advantage of this. I've written code in both spheres, and I don't blame them, either.

Quote:

---

Originally Posted by brewbuck

With OS support, the processor could set up alternate page tables pointing to the same region of memory, but with different execute/write bits.

Failing that, on a POSIX-like operating system you can call mprotect() to cause the pages to become writable, write the generated code into them, then call mprotect() again to flip them back to read-only. That's a lot of thunking in and out of the kernel, though.

---

Actually, there is no need to alias the same bit of memory - you just have to make sure that data-space that also need to be executable DOES NOT have the NX/XD bit set, whilst normal data-space not intended for execution has said bit set. Aliasing the same location in at different virtual locations just causes all sorts of confusion.

--
Mats

Quote:

---

Originally Posted by brewbuck

On that note, the NX problem could always be solved all the way as early as the 80386, but you have to use the segmentation architecture to do it. Under segmentation, only code within a "code segment" can execute. Since segments are stupid when compared with paging, nobody ever took advantage of this. I've written code in both spheres, and I don't blame them, either.

---

Yes, indeed, as long as CS and DS or SS don't overlap, there is no risk of execution anywhere outside the code-segment. But as you say, it gets a bit clunky, and doesn't work well with traditional pointer architectures.

--
Mats

Quote:

---

Originally Posted by matsp

Aliasing the same location in at different virtual locations just causes all sorts of confusion.

---

It happens quite often, whenever processes share a memory map or a SHM segment...

You could probably do it even without special OS support, simply by opening /proc/self/mem and mmapping it straight back into your own address space. Yeah, it kinda makes my head hurt.

Quote:

---

Originally Posted by brewbuck

It happens quite often, whenever processes share a memory map or a SHM segment...

You could probably do it even without special OS support, simply by opening /proc/self/mem and mmapping it straight back into your own address space. Yeah, it kinda makes my head hurt.

---

Yes, but shared memory isn't mapping the same memory twice IN THE SAME PROCESS.

And yes, you can trick around with mmaping your own memory, but that's rarely meaningful.

--
Mats

Quote:

---

Originally Posted by matsp

Yes, but shared memory isn't mapping the same memory twice IN THE SAME PROCESS.

And yes, you can trick around with mmaping your own memory, but that's rarely meaningful.

---

I have to do it as part of the startup process of my own toy kernel. When you flip paging on, you have to make sure that CS:EIP is still pointing to a valid instruction. I map the kernel way up at 0xC0100000, but I also have to make sure the first megabyte of memory is mapped to itself, so that when you flip the switch you're still okay. Of course I immediately jump up to 0xC0100000 and then remove the identity map. The first meg actually ends up mapped at 0xC0000000, right beneath the kernel in VM space.

I guess it's a kernel, not a "process," but it's still something I have to do.

Also, there's a little catch you have to deal with when calling or gating between TSS segments -- the processor fills in the TSS link field of the switched-to-TSS using the page tables of the switching-from TSS. But when the TSS call returns, it restores the link field using the page tables of the switched-to process. This implies that the TSS segments themselves have to be placed somewhere in memory and mapped to the same virtual address in both of the involved tasks.

It's all so "fun..."

Yes, startup code is kind of special case too. And kernels sometimes map user-space memory into kernel mode too, which is another special case. But it's a good idea not to do that unless there is a GOOD reason for it. [E.g. kernel needs to keep a reference to the user processes memory after the system call has been finished. Since the user can free that memory back to the kernel and then do other things with it if we juse keep the user-space mapping around, there is a reason to map it "again"].

TSS belongs in the kernel - and by the way, most of the time, you don't want to use the x86 task-switching mechanisms. [We're getting away from the subject, but ...]. I have ported/written/worked on several different kernels, and none of them use Task-Gates or Task Switches using the x86 mechanisms except for two special cases:
1. Stack fault.
2. Double fault.

Since both of these tend to happen when "things have gone really wrong", you may need a fresh stack, CR3, etc to be able to continue anything useful at all. Stack-fault can of course happen in user-mode, so you detect that case and continue to a kernel function to grow the stack or kill the user-mode app, whichever makes most sense. But in kernel mode, if you ran out of stack, it's bad news - and normal fault handling don't work well when you have "no workable stack", so Task-gate is the only viable option. Double-fault is either a consequence of completely blowing the stack, or something else horribly wrong, so same thing there.

You may need a TSS for switching user/kernel stack, but that's all it does in the normal case. But in modern processors there are "syscall" or "sysenter" (which is which I can never remember) that avoids that for system calls, which leaves interrupts and traps/exceptions. It may be that you have to support multiple TSS so that multiple threads can be in the kernel at the same time.

For general task-switching, you are much better off just writing the code to switch from one process to another (save all registers on stack, save stack in process control block, and fetch page-table map from the new proces into CR3, then reverse the save-process).

FPU state is lazy-saved "on demand".

--
Mats