Using LD_PRELOAD libraries and glibc backtrace function for debugging

Recently, I had a user app causing messages from the kernel about … “program xxx is using a deprecated SCSI ioctl, please conver it to SG_IO”.

The ioctl, I knew was SCSI_IOCTL_SEND_COMMAND. So, I grepped through all the source code I had looking for SCSI_IOCTL_SEND_COMMAND. No hits. Hmmm. Where the heck is this SCSI_IOCTL_SEND_COMMAND ioctl coming from? There are a ton of libraries and other code in here which I didn’t write, and don’t understand too well. How to find this?

The ioctl() system call takes 3 parameters, a file descriptor, an ioctl number, and a pointer. The ioctl number is SCSI_IOCTL_SEND_COMMAND. strace reveals ioctl() happens to be called zillions of times, and finding just where this one is coming from with only strace is problematic. If only there were a way to sort of trap the ioctl() call, and triggering on the ioctl number, produce a backtrace…

Turns out there is a way to do just this.

Grepping through header files revealed that the value of SCSI_IOCTL_SEND_COMMAND is 1.

There is a shell environment variable, LD_PRELOAD, which will allow arbitrary shared libraries to be loaded prior to running any program. These shared libraries can override functions in glibc, or other libraries, and do other things, including calling the original library function.

Additionally, glibc provides some functions to do backtraces.

So, all I have to do is write a shared library which overrides ioctl, looks for the ioctl number of 1, and calls the glibc backtrace functions when that happens, and in any case then calls the “real” ioctl glibc system call.

Simple in theory… how about in practice? Devil’s in the details.

First off, how to override an existing library function:

First you need to include a few things relating to shared libs:

#define _GNU_SOURCE
#include <dlfcn.h >

Then, declare a function pointer to hold the value
of the “real” ioctl() function from glibc:

static int (*next_ioctl)(int fd, int request, void *data) = NULL;

Then declare your replacement ioctl function:

int ioctl(int fd, int request, void *data)
{
        char *msg;

        if (next_ioctl == NULL) {
                fprintf(stderr, "ioctl : wrapping ioctl\n");
                fflush(stderr);
                // next_ioctl = dlsym((void *) -11, /* RTLD_NEXT, */ "ioctl");
                next_ioctl = dlsym(RTLD_NEXT, "ioctl");
                fprintf(stderr, "next_ioctl = %p\n", next_ioctl);
                fflush(stderr);
                if ((msg = dlerror()) != NULL) {
                        fprintf(stderr, "ioctl: dlopen failed : %s\n", msg);
                        fflush(stderr);
                        exit(1);
                } else
                        fprintf(stderr, "ioctl: wrapping done\n");
                fflush(stderr);
        }
        return next_ioctl(fd, request, data);
}

That makes a wrapper function for ioctl(). On the first call, it will use dlsym() to find the “real” ioctl function. Then it just calls this function with the same parameters with which it was called. Well, that’s not very useful. We want to actually do something when “request” has the value of 1. Specifically, we want to print a stack backtrace, to see the trail of functions leading up to this call, so we can find out where the hell somebody is using SCSI_IOCTL_SEND_COMMAND.

So how to get a backtrace?

#include < signal.h>
#include < execinfo.h>

static void show_stackframe() {
  void *trace[16];
  char **messages = (char **)NULL;
  int i, trace_size = 0;

  trace_size = backtrace(trace, 16);
  messages = backtrace_symbols(trace, trace_size);
  printf("[bt] Execution path:\n");
  for (i=0; i < trace_size; ++i)
        printf("[bt] %s\n", messages[i]);
}

Then, we just modify our ioctl wrapper to call this when “request” == SCSI_IOCTL_SEND_COMMAND:

int ioctl(int fd, int request, void *data)
{
        char *msg;

        if (next_ioctl == NULL) {
                fprintf(stderr, "ioctl : wrapping ioctl\n");
                fflush(stderr);
                // next_ioctl = dlsym((void *) -11, /* RTLD_NEXT, */ "ioctl");
                next_ioctl = dlsym(RTLD_NEXT, "ioctl");
                fprintf(stderr, "next_ioctl = %p\n", next_ioctl);
                fflush(stderr);
                if ((msg = dlerror()) != NULL) {
                        fprintf(stderr, "ioctl: dlopen failed : %s\n", msg);
                        fflush(stderr);
                        exit(1);
                } else
                        fprintf(stderr, "ioctl: wrapping done\n");
                fflush(stderr);
        }
        if (request == 1) { /* SCSI_IOCTL_SEND_COMMAND ? */
                /* call back trace */
                fprintf(stderr, "SCSI_IOCTL_SEND_COMMAND ioctl\n");
                fflush(stderr);
                show_stackframe();
        }
        return next_ioctl(fd, request, data);
}

How to compile this?

 #
 # Makefile
 #

all:    libs test_ioctl

libs:   libwrap_ioctl.so

libwrap_ioctl.so:       wrap_ioctl.c
        rm -f libwrap_ioctl.so*
        gcc -fPIC -shared -Wl,-soname,libwrap_ioctl.so.1 -ldl -o libwrap_ioctl.so.1.0  wrap_ioctl.c
        ln -s libwrap_ioctl.so.1.0 libwrap_ioctl.so.1
        ln -s libwrap_ioctl.so.1 libwrap_ioctl.so

clean:
        rm -f libwrap_ioctl.so* test_ioctl

To use it, just set LD_PRELOAD to load the library before executing your program.

e.g.: running a test_ioctl program that just calls the ioctl:

myhost:/home/me/wrap_ioctl # LD_PRELOAD=./libwrap_ioctl.so ./test_ioctl
ioctl : wrapping ioctl
next_ioctl = 0x20000000000472a8
ioctl: wrapping done
SCSI_IOCTL_SEND_COMMAND ioctl
[bt] Execution path:
[bt] ./libwrap_ioctl.so [0x200000000004ce30]
[bt] ./libwrap_ioctl.so(ioctl+0x6028) [0x200000000004d340]
[bt] ./test_ioctl [0x4000000000000840]
[bt] /lib/libc.so.6.1(__libc_start_main+0x50308) [0x2000000000097630]
[bt] ./test_ioctl [0x4000000000000660]
myhost:/home/me/wrap_ioctl #

You can modify wrap_ioctl.c to trigger a backtrace upon whatever condition you wish

Currently it triggers upon the 2nd argument of ioctl being 1, which is SCSI_IOCTL_SEND_COMMAND. Just modify this if statement to trigger on whatever condition you wish.

Likewise, you can wrap any library function you want… printf(), read(), write(), open(), whatever.

If you don’t want the back trace, take it out the call to show_stackframe(), and do whatever you want instead.

If you ever has some debugging problem that makes you ask “where the heck is this getting called from?” this technique might be some help.

In my particular case, the code which was calling SCSI ioctl send command turned out to look like:

        ioctl(fd, 1, some_pointer);

which was why grepping for SCSI_IOCTL_SEND_COMMAND turned out to be futile.

Hope it’s some help to someone.

~ by scaryreasoner on November 17, 2007.

24 Responses to “Using LD_PRELOAD libraries and glibc backtrace function for debugging”

  1. Question: How could the call to ioctl be detected, when called from inside the libc?

    For ioctl() this is not a realistic scenario, but what if we wanted to trace the behavior of low-level I/O function like read() or write()? There are a lot of high-level functions in the libc that simply end up using the low-level ones (printf-family, putc, puts, …).
    Is it possible to override a library function for the library itself?

  2. I’m not sure. Try it and see. I guess it depends on when those symbols get resolved.

    I haven’t been so unfortunate as to be in a position of having to debug libc.

    OTOH, strace can intercept read and write, and other system calls, but by a different mechanism (ptrace() sys call — which looks to be black magic). There’s also ltrace, which might be of some use.

  3. I found two interesting approaches:
    – using “ld -wrap”: http://okmij.org/ftp/syscall-interpose.html
    – “PLT infection”: http://www.phrack.org/issues.html?issue=56&id=7

  4. Neato. That last link… yikes.

  5. […] technique comes in handy again Today, my LD_PRELOAD trick, which I wrote about before (clicky) came in handy […]

  6. Hi,
    I was just thinking about doing a similar thing, except in my case I want to trace back from socket system calls. I am going to follow your instructions and see how it goes.

  7. Thanks for the code and Makefile, very helpful. I used a similar method to wrap some time related functions (time, gettimeofday, nanosleep and select) to make time seem to pass more quickly for a process. Why? Because I was facing a huge test suite that took hours to run, yet spent much of that time waiting for things and checking timeouts etc. Now it runs much more quickly and no changes were needed to the code under test.

  8. thanks for the tip, it really helped.

  9. Good guide for LD_PRELOAD, though in this particular case, if you’re using Solaris or MacOS, you could use this DTrace one-liner from the shell:

    # dtrace -n syscall::ioctl:entry/pid == $target && arg1 == 1/{ustack();}’ -c

    This would print out a userland stack trace every time your program generated an ioctl with ‘1’ as the second argument.

  10. Thanks for the tip on MacOs and Solaris. I used SunOS 4.x a bunch, back in the day, used to be sysadmin (back when that meant something) of a bunch of Suns, Hp-ux, etc. boxes, including a Sun E5000, back when that was a half-million dollar machine (er… I think it was a half-million, though that seems absurd when I write it now… it had half a GB or RAM, and I vaguely remember thinking, hmm, a dollar a gigabyte… but might be wrong. Was a long time ago.) Done a little Solaris (SunOS 5.x and above) but it’s been ages. Nowadays, it’s almost entirely linux, with some annoying detours into the dreaded, nasty, horrible vmware beast from time to time. 🙂 I’ve never really tried MacOS, although I’m told that a game which I wrote ( http://wordwarvi.sourceforge.net ) seems to work alright on MacOS.

  11. Er, no, reviewing my so-called “math”, the “dollar a gigabyte” figure makes no sense. Ok, I have no idea how much the e5000 cost back in 1993 or so.

  12. No, probably not 1993… probably 1995, or 1996. I give up. Too long ago. Damn I’m old.

  13. Hi I want trap write system call using ptrace mechanism. And I want get all the parameters are passed to write system call. Any help would be greatly appreciated.


    Patel.

  14. I can trap write system call, but I am not able to print the parameters which were passed to write system call…

  15. I’ve never used ptrace, so I don’t know about that.

    You ought to be able to use the technique I talked about above… the parameters to write are an int, a void *, and a size_t.

    Also, if all you want to do is print the arguments to write(), you might try the -e option of strace, eg:

    #strace -e write echo hello
    write(1, “hello\n”, 6hello
    ) = 6

    See the strace man page.

  16. BTW the first google hit is an article which tells how to do exactly what you want, printing the arguments of write, using ptrace. See: http://www.linuxjournal.com/article/6100

    It’s about half way down the page. It’s i386 specific, but anything that uses ptrace will be architecture specific.

    Well, I’m guessing you probably already found that, and it’s not working for you for some reason. But, like I said, I’ve never used ptrace, so I’m afraid I can’t help on that.

    It also occurs to me that, as one poster noted above, calls to write() from within libc likely won’t be intercepted by the LD_PRELOAD technique.

    You might take a look at the source for strace:
    http://sourceforge.net/project/showfiles.php?group_id=2861
    and see what it’s doing. (I’m not sure that’s the canonical source for strace, it’s just what showed up when I typed “strace source” into google.)

  17. Hi sorry for late reply, I was not feeling well so could not reply to this. I done through LD_PRELOAD. But now I am looking for ptrace because of its portability. I want to print all the parameters which are passed to the selected system call, say for ex : Open(), execve(), Write()…

  18. Hi thanks for your reply, I got it…

  19. Again : I am not able to trap all Open() calls.
    say for ex : 30 times open() is called,
    1) Only few I am able to trap.
    2) While printing arguments of this call, I am not able to print every time this call is trapped. I am getting Segmentation fault.

  20. Hi,
    Finally I done it! Its working fine. When you go for LD_PRELOAD it can trap system calls only which are called by parent process not the child process. So if you want to trap all the syscalls which are called by all forked processes, then you need to use ptrace. Get the child pid (forked process pid)by the waitid syscall(Third arg : siginfo_t structure(.si_pid)).

    Good luck guys!!!
    Thanks and regards,
    Patel.

    • Hi Patel,

      Can you show an example or your code intercepting syscalls w/ ptrace? Can it intercept syscalls that are called from within libc?

      Thanks

  21. Very useful article! One minor note: The man page says that you have to call dlerror() prior to calling dlsym(), in order to clear any old error conditions. So the code should look like:

    dlerror(); /* Clear any existing error */
    next_ioctl = dlsym(RTLD_NEXT, “ioctl”);

    if ((msg = dlerror()) != NULL) {
    // …error handler…
    }

  22. On system calls vs. library calls: “open()” is actually a library function implemented in libc. It is a wrapper to the platform specific kernel trap mechanism as in

    /usr/include/asm-generic/unistd.h:__SYSCALL(__NR_open, sys_open)

    This means you can intercept calls to “open()” with the technique described. See my example code for wrapper.c:

    /*
    * Wrapper für Bibliotheksfunktionen
    * am Beispiel von “open()”.
    *
    * Von Nils Magnus (magnus@linuxtag.org)
    *
    * nach einer Beschreibung von Greg Kroah-Hartmann und einem Hinweis
    * von Alexander Gabert.
    *
    * Mehr Doku:
    * http://tldp.org/HOWTO/Program-Library-HOWTO/dl-libraries.html
    */

    #define RETTYPE int
    #define FUNC open
    #define FUNCSTRING “open”
    #define ARGS const char *pathname, int flags
    #define ARGSVARS pathname, flags
    #define LIB “/lib/x86_64-linux-gnu/libc-2.13.so”

    #include
    #include
    #include
    // #include
    #include
    #include

    RETTYPE FUNC(ARGS) {

    /* Static pointer to the original function. it is initialized to
    NULL but can be recycled during subsequent calls: */
    static RETTYPE (*orig)(ARGS) = NULL;

    /* Handle to get access to the original library */
    void *handle;

    /* Real return value: */
    RETTYPE rv;

    /* In case loading the library fails: */
    char *error;

    /* Are we called for the first time? If so, grab the handle: */
    if (!orig) {
    handle = dlopen(LIB, RTLD_LAZY);
    if (!handle) {
    fputs(dlerror(), stderr);
    exit(1);
    }
    orig = dlsym(handle, FUNCSTRING);
    if ((error = dlerror()) != NULL) {
    fprintf(stderr, “%s\n”, error);
    exit(1);
    }
    }

    /* Do something with the ARGS */

    /* … */

    /* Call the original function: */

    rv = orig(ARGSVARS);

    printf(“%d = open(\”%s\”, %d);\n”, rv, ARGSVARS);

    /* Do somthing with the return value */

    return rv;
    }

    Matching Makefile:
    CC = gcc
    CFLAGS = -fPIC -shared -ldl

    wrapper.so: wrapper.c
    $(CC) $(CFLAGS) -o $@ $<

    Test run:
    $ LD_PRELOAD=./wrapper.so head -1 /etc/passwd
    3 = open("/etc/passwd", 0);
    root:x:0:0:root:/root:/bin/bash

  23. […] So I decided to dig more into the data camera app passes through ioctl. I found just what I needed: Using LD_PRELOAD libraries and glibc backtrace function for debugging by Scary Reasoner. This will allow not only camera app to be investigated but… mmm… much, much […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: