Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Why do i get `** stack smashing detected **` error?

I am trying to link this assembly function with this c++ code:

.text 

.global _Z7scambiaRiS_

_Z7scambiaRiS_:
    pushq %rbp
    mov %rsp, %rbp

    mov (%rdi), %rax
    mov (%rsi), %rcx
    mov %rcx, (%rdi)
    mov %rax, (%rsi)

    leave
    ret

.data
#include <iostream>

void extern scambia(int &a, int &b);

int main()
{ 
    int a, b;
    std::cin>>a>>b;
    std::cout<<"a: "<<a<<" b: "<<b<<"\n";
    
    scambia(a, b);

    std::cout<<"a: "<<a<<" b: "<<b<<"\n";
    
    return 0;
};

Assembly is a intel x86-64 version.

The program works, but then it stops with segmentation fault,
here is the output, given as input 3 4:

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

3 4 
a: 3 b: 4
a: 4 b: 3
*** stack smashing detected ***: terminated
Aborted (core dumped)

I don’t really understand how i managed to compromise the stack, and even debugging with gdb i couldn’t find out what went wrong

>Solution :

scambia is declared as taking references to int, which is 4 bytes on this platform. However

    mov (%rdi), %rax
    mov (%rsi), %rcx
    mov %rcx, (%rdi)
    mov %rax, (%rsi)

is doing 8-byte loads and stores, because rax and rcx are 64-bit registers. So each of them writes 4 bytes beyond the int that you passed in, which means that other stack space is overwritten, triggering the stack smashing detector.

To use the assembly function as is, it should be declared in C++ as taking references to int64_t or some other 64-bit type, and the types of a,b adjusted accordingly. (long int is 64 bits on many x86-64 platforms, but not all).

To keep the declaration as swapping two ints, change the assembly code to use 32-bit registers for the data being moved to and from memory:

    mov (%rdi), %eax
    mov (%rsi), %ecx
    mov %ecx, (%rdi)
    mov %eax, (%rsi)

Note that rdi and rsi should stay as is, since they are pointers, which are 64 bits.

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading