This post has been republished via RSS; it originally appeared at: MSDN Blogs.
Writing in assembly language doesn't absolve you from
adhering to the software conventions.
Certainly you have more flexibility in assembly language,
but that doesn't give you freedom to do anything you want.
We saw earlier that
you must not use any stack memory below the red zone,
because a page fault or other exception could occur,
and the system needs to know which parts of the stack are
available for use by exception handlers.
This also means that you need to adhere to the software
conventions so that the system can unwind the stack in
order to find and execute the exception handlers.
In particular, this means that if your function is not
a lightweight leaf function,
you must register unwind codes on systems that require them
(which is pretty much everything other than x86).
If you fail to register unwind codes, then the system will
assume that you are a lightweight leaf function,
which means that it will assume that all nonvolatile registers
are unmodified from the calling function,
the stack pointer has not been changed from its value
at function entry,
and that the return address is in the default location.
For x64, this means that the return address is at the top
of the stack;
for RISC, it means that the return address is in the standard
return address register.
If the system thinks that you are a lightweight leaf function,
but you didn't follow the rules for a lightweight leaf function,
then bad things will happen.
During unwinding, the system will not restore
nonvolatile registers to the values from the calling function.
(It can't do that even it wanted to,
seeing as you didn't tell the system where to restore them from.)
The system will assume that the caller's address is
whatever is stored in the default return address location.
(And again, it can't do any better even if it wanted to,
since you didn't tell the system where the return address is.)
If the nonvolatile registers are improperly restored,
then if the system needs to unwind to the caller,
the caller will have incorrect registers,
and that will lead to unpredictable behavior.
If the return address is improperly restored,
then the system won't be able to unwind the stack at all.
If you're lucky, it'll realize that the return address
is not in a registered code segment and terminate your
process immediately (and flag the crash as a potential
security vulnerability).
If you're not lucky, the incorrect return address
will happen to be in a registered code segment,
and the system will continue unwinding from that point,
even though it has nothing to do with anything on the
call stack.
This may end up calling exception handlers in ways
that they never expected to run,
which is also
something that an attacker could take advantage of.
Anyway, the point of the story is that you need to
adhere to the Windows software conventions even when
writing assembly language.
Because the system may at times need to
reason over
the state of your program,
and you need to keep it in a state that the system
can understand.