Hmm, that's strange. Are you compiling with optimizations?

Yes. I'm compiling with --release. I somehow thought you are doing so too, but apparently that is not the case.

BTW, I also had same issue with paging, because code has no side-effects, and some of it was getting optimized away.

Yes. I'm compiling with --release. I somehow thought you are doing so too, but apparently that is not the case.

I don't use --release at the moment, since it makes debugging more difficult and the kernel is still fast enough without it. The problem with --release is that the compiler is clever. For example, the compiler might turn your modified stack_overflow function into loop { print!("."); } (through tail call optimization), which no longer causes a stack overflow. A do-not-optimize attribute for functions would be great for this case…

However, the code should also work with --release, of course. I'll try to find a different way to cause stack overflows, that also works with optimizations.

BTW, I also had same issue with paging, because code has no side-effects, and some of it was getting optimized away.

That sounds really bad 😟.

Thanks a lot for reporting these issues! I'm quite busy at the moment, but I'll try to investigate and fix them as soon as I have some time again.

Does the following work?

#[inline(never)]
fn stack_overflow(){
    print!(".");
    stack_overflow();
    print!(".");
    stack_overflow();
}

Or

#[inline(never)]
fn a(){
    b();
}
#[inline(never)]
fn b(){
    print(".");
    a();
}

So there are two issues here:

1. Tail call optimization might turn an endless recursion into a loop. I think the proper way to avoid this is to add an inline(never) annotation.
2. The optimizer removes endless loops without side effects. This is a known bug caused by undefined behavior in LLVM: rust-lang/rust#28728

My test for this works with --release optimisations on:

#[allow(unconditional_recursion)]
fn stack_overflow(n: u64) -> u64 {
    // We need to indicate to the compiler that it should disable optimisations by black_boxing `n` 
   stack_overflow(black_box(n + 1))
}

/// lifted from: https://doc.rust-lang.org/src/core/hint.rs.html#111
fn black_box<T>(dummy: T) -> T {
    unsafe {
        asm!("": : "r"(&dummy));
        return dummy;
    }
}

Can you elaborate on the paging issues? It's a known issue that all of the accesses to the page tables should be volatile and ideally non-tearing-guaranteed (probably through inline asm), but I haven't heard of any problems with these things in practice.

woops, didn't see the original post date - ignore (although an answer would still be nice 😄)

@64 Unfortunately, I never really solved the issues. I didn't know how to use bochs debugger then. My paging code was a huge mess anyways, so I have rewritten it twice since then. I'm now using PageTable struct from x86_64 crate with my own custom mapper, and it seems to work. No idea why, though. I access the page tables like this:

let mut table: &mut PageTable = unsafe { &mut *virt_addr.as_mut_ptr() };

And I make sure the reference is dropped before I flush changes to TLB.