Kprobes (Kernel Probes) is a powerful feature in the Linux kernel that allows developers and system administrators to dynamically intercept and monitor any kernel function. It provides a mechanism for tracing and debugging the kernel by enabling you to inject custom code into almost any point in the kernel, allowing you to collect information, modify data, or even create entirely new behaviors. Kprobes are particularly useful for diagnosing kernel issues, performance tuning, and understanding kernel behavior without needing to modify the kernel source code or reboot the system.
Background on Kprobes:
Kprobes were introduced in the Linux kernel as a way to enable non-disruptive kernel tracing. The main use case is dynamic instrumentation, which allows developers to investigate how the kernel behaves at runtime without modifying or recompiling the kernel.
How Kprobes Work
Kprobes allow you to place a "probe" at a specific point in the kernel, known as a probe point. When the kernel execution reaches this probe point, the probe is triggered, and a handler function that you define is executed. Once the handler is done, the normal execution of the kernel resumes.
There are two types of handlers in Kprobes:
1. Pre-handler: This is executed just before the probed instruction.
2. Post-handler: This is executed after the probed instruction completes.
Key Components of Kprobes:
1. Kprobe Structure : Defines the probe, including the symbol name (function to be probed) and pointers to pre- and post-handlers.
- Example:
static struct kprobe kp = {
.symbol_name = "do_fork", // Name of the function to probe
};
2. Pre-Handler: Executed before the instruction at the probe point. It can be used to capture the state of the system (e.g., register values).
- Example:
static int handler_pre(struct kprobe *p, struct pt_regs *regs) {
printk(KERN_INFO "Pre-handler: register value is %lx\n", regs->ip);
return 0;
}
3. Post-Handler: Executed after the instruction at the probe point. This is useful for gathering information after the instruction has executed.
- Example:
static void handler_post(struct kprobe *p, struct pt_regs *regs, unsigned long flags) {
printk(KERN_INFO "Post-handler: instruction completed\n");
}
Inserting a Kprobe:
Once the Kprobe structure is set up, you register the probe using the `register_kprobe()` function, which activates the probe at the desired location in the kernel.
Example of inserting a probe:
int ret = register_kprobe(&kp);
if (ret < 0) {
printk(KERN_ERR "Kprobe registration failed\n");
} else {
printk(KERN_INFO "Kprobe registered successfully\n");
}
When you're done with the probe, it should be unregistered using `unregister_kprobe()`.
Use Cases for Kprobes:
1. Debugging: Inspect kernel function behavior and parameters at runtime without recompiling the kernel.
2. Performance Monitoring: Collect detailed performance statistics at various points in the kernel.
3. Dynamic Analysis: Understand kernel module or driver behavior in real-time.
4. Fault Injection: Inject faults at specific points in the kernel to test how the kernel reacts to errors.
5. Security Auditing: Monitor suspicious or unauthorized kernel activities.
Kprobes vs. Other Tracing Mechanisms:
- Ftrace: Another kernel tracing framework, but more focused on function-level tracing. Kprobes are more versatile as they allow you to probe any instruction.
- SystemTap**: Provides a higher-level interface that uses Kprobes under the hood.
- eBPF: A more modern, flexible, and performant tracing framework that has overlapping functionality with Kprobes.
Kprobe Variants:
1. Jprobes**: A variant that allows you to specify the exact function signature for the probe. This feature is deprecated in modern kernels.
2. Kretprobes**: A specialized form of Kprobes that hooks into the return path of functions, allowing you to trace function exits and the values returned by kernel functions.
Limitations of Kprobes:
- Probes introduce overhead, so excessive probing can impact system performance.
- Probing certain sensitive or timing-critical functions can lead to system instability.
- The handler code should be minimal and non-blocking to avoid disrupting the kernel execution flow.
Example Code:
Below is a basic example of how Kprobes can be used to monitor the `do_fork` function in the kernel, which is responsible for process creation:
#include <linux/kernel.h>#include <linux/module.h>
#include <linux/kprobes.h>
static struct kprobe kp = {
.symbol_name = "do_fork", // The function to probe
};
static int handler_pre(struct kprobe *p, struct pt_regs *regs) {
printk(KERN_INFO "do_fork() called, IP = %lx\n", regs->ip);
return 0;
}
static void handler_post(struct kprobe *p, struct pt_regs *regs, unsigned long flags) {
printk(KERN_INFO "do_fork() completed\n");
}
static int __init kprobe_init(void) {
kp.pre_handler = handler_pre;
kp.post_handler = handler_post;
if (register_kprobe(&kp) < 0) {
printk(KERN_ERR "Kprobe registration failed\n");
return -1;
}
printk(KERN_INFO "Kprobe registered successfully\n");
return 0;
}
static void __exit kprobe_exit(void) {
unregister_kprobe(&kp);
printk(KERN_INFO "Kprobe unregistered\n");
}
module_init(kprobe_init);
module_exit(kprobe_exit);
MODULE_LICENSE("GPL");
This will print information to the kernel log each time the `do_fork()` function is invoked.
To compile and run the Kprobe example you provided, you need to follow these steps:
1. Prerequisites
- You need to have the Linux kernel headers installed.
- Make sure you have root (superuser) privileges, as you'll be loading kernel modules.
- You need `gcc` and `make` installed for compiling the kernel module.
yum search glibc-static
yum install glibc-static
yum update --allowerasing
2. Write the Kprobe Kernel Module
wget https://raw.githubusercontent.com/torvalds/linux/master/samples/kprobes/kprobe_example.c
Save the Kprobe code into a file, for example, `kprobe_example.c`:
3. Create a Makefile
Create a `Makefile` to automate the compilation of the kernel module. This Makefile should look like this:
makefile
obj-m += kprobe_example.o
all:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
clean:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
4. Compile the Kprobe Kernel Module
$ make
This will use the kernel headers and create a module file `kprobe_example.ko`.
5. Insert the Kernel Module
To insert the module into the running kernel, use the `insmod` command. You need root privileges to load kernel modules:
$ insmod kprobe_example.ko
6. Check Kernel Logs for Output
You can monitor the kernel logs to see the output of the Kprobe:
$ dmesg | tail
[18084.207866] kprobe_init: Planted kprobe at 00000000207be762
This will show you the success or failure of inserting the Kprobe and print any trace outputs when the kernel function (like `do_fork`) is invoked.
7. Trigger the Probed Function
You can manually trigger the `do_fork()` function by starting a new process, such as running any command: $ ls
Since `do_fork()` is involved in creating new processes, every time a new process is created, the Kprobe pre- and post-handlers will execute, and you'll see the output in `dmesg`.
8. Remove the Kernel Module
Once you're done with the Kprobe, you can remove the kernel module using `rmmod`:
$ rmmod kprobe_example
[root@myhost]# lsmod | grep probe
kprobe_example 3569 0
[root@myhost]# ls
[root@myhost]# rmmod kprobe_example
[root@myhost]# lsmod | grep probe
[root@myhost]#
Check the kernel log again to see the output confirming the Kprobe has been removed:
$ dmesg | tail
9. Clean Up
To clean the build directory and remove compiled files, you can run:
$ make clean
--------------------------------------------------------------------------------------------------------
Example Workflow:
$ vim kprobe_example.c # Write the module code
$ vim Makefile # Create the Makefile
$ make # Compile the module
$ insmod kprobe_example.ko # Insert the module
$ dmesg | tail # Check kernel logs for output
$ ls # Trigger the do_fork() function
$ dmesg | tail # Check logs again to see Kprobe output
$ sudo rmmod kprobe_example # Remove the module
$ make clean # Clean up the build files
