RAID Demystified: A Beginner’s Guide to Data Protection and Performance

In today’s digital world, data is invaluable and losing  data/files due to disk failure can be devastating. This is where RAID (Redundant Array of Independent Disks) comes into play. RAID is a powerful technology that enhances data protection, storage performance, and fault tolerance, ensuring your data remains safe and accessible.

What is RAID?

RAID combines multiple hard drives into a single logical unit, providing benefits such as:

✅ Improved Performance – Faster read/write speeds by distributing data across multiple disks.

✅ Higher Availability – Ensures continuous access to data even if a drive fails.

✅ Fault Tolerance – Protects against hardware failures by storing data redundantly.

RAID is widely used by businesses, IT professionals, and individuals who need secure and efficient storage solutions.

How RAID Works: A Simple Example

Imagine you have a storage server with three disks where your data is stored.

📌 Without RAID: If one disk fails, all data on it is lost, potentially leading to downtime or permanent data loss.

📌 With RAID: The system keeps functioning because the data is mirrored or distributed across other disks, ensuring redundancy and seamless recovery.

RAID Controller: The Brain Behind the System

A RAID controller manages the disks, making them appear as a single storage unit to the operating system. This improves redundancy, performance, and reliability without manual intervention.

Different RAID types/configurations cater to different needs. Let’s break down the most commonly used RAID levels:

🔹 RAID 0 (Striping) – Enhances speed by distributing data across multiple drives. 🚀 Pros: High performance. ❌ Cons: No redundancy—if one drive fails, all data is lost.

🔹 RAID 1 (Mirroring) – Creates an exact copy of data on two drives. 🔄 Pros: Excellent data protection. ❌ Cons: Requires double the storage space.

🔹 RAID 5 (Striping with Parity) – Balances performance and redundancy by distributing parity data. 🔄 Pros: Can survive a single drive failure. ❌ Cons: Slower write speeds due to parity calculations.

🔹 RAID 6 (Double Parity) – Similar to RAID 5 but with extra fault tolerance. 🔄 Pros: Can survive two simultaneous drive failures. ❌ Cons: Requires more storage space.

🔹 RAID 10 (RAID 1 + RAID 0) – Combines speed and redundancy by mirroring data and striping it across multiple drives.  Pros: Best of both worlds—fast and secure. ❌ Cons: Requires at least four drives.

 Choosing the Right RAID Level

Your choice depends on your needs:

✅ Need speed? RAID 0

✅ Need redundancy? RAID 1, RAID 5, or RAID 6

✅ Need both? RAID 10

Advantages of Using RAID

✔ Data Redundancy – Protects against hardware failures.

✔ Fault Tolerance – Keeps systems running even if a drive fails.

✔ High Performance – Improves read/write speeds.

✔ Scalability – Easily expand storage by adding more drives.

✔ Faster Recovery – Restores lost data quickly compared to traditional backups.

Limitations of RAID

❌ Additional Cost – Requires multiple hard drives.

❌ Setup Complexity – Requires technical knowledge.

❌ Not a Backup Solution – Protects against drive failures, but not accidental deletions or cyberattacks.

Final Thoughts: Is RAID Right for You?

RAID is an excellent data protection solution that improves performance and prevents data loss from hardware failures. Whether you’re a business storing critical files or a home user safeguarding personal data, RAID can provide peace of mind and reliability.

However, remember that RAID is not a replacement for backups—it only protects against hardware failures, not accidental deletions or cyber threats.

Linux Memory Management and Layout

In Linux, RAM is divided into kernel space and user space to manage memory and protect the kernel from user applications. Here's a breakdown of how this division works, along with details on high memory, low memory, and crashkernel allocation:

Kernel Space vs. User Space

Kernel Space: This area is reserved for running the kernel, kernel extensions, and most device drivers. User-space applications cannot directly access this memory.

User Space: This is the memory area where application software and some drivers execute, typically with one address space per process.

The separation ensures that processes are protected from each other and that the kernel is protected from user-space applications.

Memory Division

1.  Address Space: On some architectures like x86-64 ppc64, the virtual address space is split into two halves. The bottom half is for user-space allocations, and the top half is for kernel allocations. For example, on x86, there can be a 3G/1G split, where 3GB is for user space and 1GB for the kernel.

2.  Memory Allocation: Memory is allocated as needed within the address space. The address space split determines the use of virtual addresses but doesn’t dictate physical memory use. The kernel allocates memory for its own binary and any additional needs, which cannot be swapped out.

3.  Memory Mapping: The kernel manages memory mapping with the help of hardware (MMU - memory management unit). The kernel maintains its own mappings shared by all processes, and each process gets its own user-space mapping.

High Memory vs. Low Memory

Applicability: The "high/low memory" split is primarily relevant to 32-bit architectures with relatively large physical RAM (more than ~1 GB). On 64-bit architectures or systems with smaller physical address spaces, the entire physical space can be accessed from the kernel virtual memory space, and all physical memory is considered "low memory".

Low Memory: In Linux systems, low memory is typically used for the kernel. A portion of the kernel virtual address space can be mapped as a single contiguous chunk into physical "low memory".

High Memory: High memory is often used for application space. It is a range of the kernel's memory space where data to be accessed is placed.

In essence, high memory is a region of kernel memory used to map physical memory that cannot be contiguously mapped with the rest of the kernel's virtual memory.

Crashkernel Memory Allocation:

Crashkernel is a reserved memory region that can be used to boot a second kernel in case of a system crash, enabling crash analysis and debugging. Its is design that some part of the this allocated in Low memory and other part at high memory

GRUB (GRand Unified Bootloader) is a boot loader that loads the Linux kernel, presenting a menu of operating systems or kernels to choose from at system startup. It allows users to select which OS to boot and pass arguments to the kernel. GRUB's RAM usage occurs in the real mode area (RMA), which is a part of RAM used by GRUB for its operations and to load boot components. The current size of the RMA is 768 MB. However, GRUB2 might fail with an out-of-memory error depending on bigger size of  kernel image or the initramfs

The memory layout of a program is divided into segments, each with a specific purpose. These segments include the text segment, data segment (initialized and uninitialized), heap, and stack. In the context of memory allocation, the parameters you've provided describe the boundaries and limits of memory usage for a program. Here's an explanation of each term:

memory_limit: This sets the maximum amount of memory in bytes that a script is allowed to allocate. It helps prevent poorly written scripts from using up all available memory. A value of 0 here (0000000000000000) likely means there is no memory limit. However, in some contexts, it might represent the initial memory limit which can be changed. In PHP, setting `memory_limit` to -1 means the script can use all the memory that is left over from the operating system and other important processes running.

alloc_bottom: This indicates the starting address of the memory region available for allocation. In your example, it's 0000000010350000.

alloc_top: This represents the highest address up to which memory can be allocated. Here, it's 0000000030000000.

alloc_top_hi: This could indicate a higher boundary for memory allocation, possibly in a system that uses segmented memory management. In this case, it's 0000000100000000.

rmo_top: This might refer to the top of the read-only memory region. It is 0000000030000000 in your layout.

ram_top: This likely indicates the top address of the available RAM. It's 0000000100000000 in your example.

The stack segment is near the top of memory with a high address, while the text, data, and heap segments have lower addresses. When a function is called, stack memory is allocated for it, and when a new local variable is declared, more stack memory is allocated, causing the stack to grow downwards. Stack memory allocation and deallocation are done automatically. The heap is where dynamic memory allocation takes place using functions like `malloc()` and `calloc()`. Unlike the stack, heap memory allocation is not continuous, and users can free heap memory, causing fragmentation.

Here's the conversion for each of your memory parameters:

memory_limit: 0000000000000000 (hex) = 0 bytes. It means there is no memory limit imposed.

alloc_bottom: 0000000010350000 (hex) = 271,151,104 bytes.

alloc_top: 0000000030000000 (hex) = 805,306,368 bytes.

alloc_top_hi: 0000000100000000 (hex) = 4,294,967,296 bytes.

rmo_top:0000000030000000 (hex) = 805,306,368 bytes.

ram_top:0000000100000000 (hex) = 4,294,967,296 bytes.

Now, calculating the sizes of memory ranges:

1) Allocation Range (alloc_bottom to alloc_top):
    alloc_top - alloc_bottom + 1 = 805,306,368 - 271,151,104 + 1 = 534,155,265$$ bytes
    In MB: $$534,155,265 / 1,048,576 \approx 509.41MB
2) Upper Allocation Range (alloc_bottom to alloc_top_hi):
    alloc_top_hi - alloc_bottom + 1 = 4,294,967,296 - 271,151,104 + 1 = 4,023,816,193$$ bytes
    In GB: 4,023,816,193 / 1,073,741,824 approx 3.75GB
3) RMO Size (from alloc_bottom to rmo_top):
    rmo_top - alloc_bottom + 1 = 805,306,368 - 271,151,104 + 1 = 534,155,265 bytes
    approx 509.41MB
4) RAM Size (from alloc_bottom to ram_top):
    ram_top - alloc_bottom + 1 = 4,294,967,296 - 271,151,104 + 1 = 4,023,816,193bytes
    approx 3.75GB

-----------------------------------------

The Real Mode Area (RMA) in the context of the Linux kernel and boot process refers to the region of memory where the system initially operates in real mode before transitioning to protected mode. In real mode, the processor can address only 1 MB of memory.

Here's a breakdown of how the RMA fits into the boot process:

1.  Real Mode Operation: When the computer starts, it boots into real mode. In this mode, the processor has a limited addressing capability (1MB).

2. GRUB Loading: GRUB (or any bootloader) operates initially in real mode. It loads the kernel image into memory using BIOS disk I/O services.

3.  Memory Arrangement: In real mode, RAM is organized such that the kernel image is loaded into memory by the boot loader. A small part of the kernel containing real-mode code is loaded below the 640K barrier, while the larger part that runs in protected mode is loaded after the first megabyte.

4.  Transition to Protected Mode: After loading the necessary components, the system switches from real mode to protected mode, which allows access to more memory and advanced features.

5. Memory Addressing: In Real Mode, memory access is done using Segmentation via a segment:offset system.

Key Components in IBM Power Systems Boot Process

Key Components in IBM Power Systems Boot Process

Stage 1 : OPAL Firmware: Initializes hardware and provides runtime services. Passes control to Petitboot as the default bootloader.

Stage 2 : Petitboot: Functions as the primary bootloader. Uses kexec to load the Linux kernel and initramfs directly.

• Scans available devices for bootable options.
• Detects core.elf (GRUB binary) as a bootable ELF file.
• Loads and executes core.elf.

Stage 3: GRUB: May be involved as an intermediate bootloader for Linux distributions that rely on GRUB configuration (e.g., RHEL, SLES, or Ubuntu Server).Works as part of the core.elf file, loaded by Petitboot in some scenarios.

• GRUB reads its configuration file (e.g., /boot/grub/grub.cfg).
• Presents a boot menu (if configured) or selects the default kernel.
• Loads the Linux kernel and initramfs into memory.
• Passes control to the kernel.

Steage 4: Linux Kernel : The kernel initializes the system and starts the init process.

----------------------------------------------------------------------------------------------------------------

Advantages :

1. Compatibility: Supports Linux distributions with GRUB-based boot processes.
2. Flexibility: Allows advanced boot scenarios (e.g., multiple kernels, chainloading).
3. Optimization: Petitboot handles hardware initialization efficiently, while GRUB adds cross-distro compatibility.

While Petitboot is the default and primary bootloader in IBM Power Systems, GRUB can be used as part of the boot process, particularly through the core.elf file. Petitboot loads and executes GRUB when required, allowing Linux distributions to leverage GRUB's flexibility and maintain consistent boot processes across architectures. This combination ensures optimal performance and compatibility for enterprise-grade Linux distributions on Power Systems.

---------------------------- Power Firmware -------------------

In IBM's PowerPC architecture (commonly used in IBM Power Systems), firmware plays a critical role in managing hardware resources, initializing the system, and providing runtime services. Here's how it works and where firmware resides:

Key Firmware Components in IBM Power Systems

Hostboot:

Responsible for the low-level initialization of the system, such as memory controller setup and processor initialization. Resides in non-volatile storage (e.g., flash memory) on the system board.

Runs on the main processor during the very early stages of boot.

OpenPOWER Firmware (OPAL):

Acts as the interface between the hardware and the operating system.

Provides services such as interrupt handling, power management, and hardware abstraction.

Resides in non-volatile memory (flash storage) but is loaded into main system RAM for execution during boot.

Petitboot:

A Linux-based bootloader that uses the kexec mechanism to load the Linux kernel.

Petitboot itself resides on the system's non-volatile storage and is executed in main system RAM during the boot process.

System Management Firmware:

Manages system-level operations such as monitoring, diagnostics, and recovery.

Runs on a dedicated service processor (e.g., BMC) and resides in the BMC's non-volatile storage.

Where Firmware Resides

Non-Volatile Storage (Flash Memory):

Core firmware components like Hostboot, OPAL, and Petitboot are stored in the system's flash memory on the motherboard or a separate chip. This ensures persistence even when the system is powered off.

System RAM:

During the boot process, firmware like Hostboot, OPAL, and Petitboot are copied from flash memory into main system RAM for execution.

The Linux kernel uses OPAL calls to interact with hardware, and these services are available as long as the system runs.

Service Processor (BMC):

The BMC firmware resides in its own dedicated non-volatile memory on the service processor.

The BMC operates independently of the main system and manages power-on, firmware updates, and error reporting. Interaction with Linux OS on Power Systems

Firmware-to-OS Handoff:

OPAL firmware initializes hardware and performs diagnostics before handing control to the Linux kernel. Petitboot (running on top of OPAL) loads the Linux kernel via kexec.

Runtime Services:

OPAL continues to provide runtime services to the Linux kernel, such as hardware interrupts, error handling, and power state management. Linux interacts with firmware using the OPAL API and device tree structures.

Firmware Location on Filesystem:

Firmware blobs for devices (e.g., network cards, GPUs) are stored in /lib/firmware. Core system firmware (e.g., OPAL, Hostboot) does not reside in the Linux filesystem but in the system's non-volatile memory. Summary for Power Systems

Primary Firmware (Hostboot, OPAL, Petitboot): Resides in non-volatile storage (flash memory) on the system board. Executed in system RAM during boot and runtime.

Service Processor Firmware (BMC): Resides in dedicated non-volatile storage on the service processor. Operates independently of the main CPU and Linux OS.

Device Firmware: Resides in /lib/firmware on the Linux filesystem and is loaded into specific hardware devices by their drivers.

This modular design ensures separation between core firmware, runtime services, and device-specific firmware, enabling robust and scalable operations on IBM Power Systems.

------------------------------------------HMC-----------------------------------

The Hardware Management Console (HMC) is a critical component in managing IBM server systems, including IBM Power Systems and IBM Z mainframes. It serves as a physical or virtual appliance that provides a unified interface for system administrators to control and monitor multiple servers and their partitions.

Key Features of HMC

- Management Interface: The HMC offers both command line (SSH) and web-based interfaces, allowing for flexible access and management of the systems it oversees. This includes functionalities for monitoring system health, configuring hardware, and managing logical partitions.

- Multi-System Management: One of the significant advantages of the HMC is its ability to manage multiple servers simultaneously. This capability is essential for organizations with complex IT infrastructures, as it simplifies administration tasks and enhances operational efficiency.

- Virtualization Support: The HMC plays a crucial role in virtualization by enabling the creation and management of logical partitions (LPARs). This allows for better resource utilization and flexibility in deploying applications across different environments.

- Monitoring and Diagnostics: Administrators can quickly identify hardware issues through the HMC's monitoring tools. It provides real-time status updates and alerts, facilitating proactive maintenance and reducing downtime.

- Redundancy and Reliability: The HMC can be configured in redundant setups to ensure high availability. Dual HMCs can manage the same systems, providing backup capabilities in case one console fails.

- Security Features: The HMC is designed with security in mind, featuring a closed platform that restricts unauthorized software installations and limits access to essential functions. It is firewalled by default, with minimal open ports to enhance security against external threats.

Connection Between mkvterm and Virtual Serial Adapters

The mkvterm command on IBM's Hardware Management Console (HMC) is used to open a virtual terminal connection to a logical partition (LPAR). This command is closely associated with virtual serial adapters, which facilitate the connection between the HMC and the LPAR.

1. Virtual Serial Adapters:

   - Each LPAR typically has two virtual serial server adapters, allowing for console connections. These adapters are configured within the Virtual I/O Server (VIOS) environment.

   - The mkvtermerm command allows users to connect to these virtual serial adapters, effectively opening a console session for managing the LPAR

2. Usage of mkvterm:

   - The mkvterm command on the HMC corresponds directly to commands used in the VIOS for managing virtual terminal sessions. When you execute mkvterm, you specify the LPAR ID to establish a connection through an available virtual serial adapter.

   - If the HMC is unavailable, users can still access the LPAR console via VIOS using a similar command (mkvterm), which underscores the flexibility of managing LPARs through different interfaces.

3. Session Management:

   - It is crucial to manage these sessions properly. For instance, if a console session is active through VIOS, attempting to start another session via HMC will result in an error indicating that a terminal session is already open. This emphasizes the need for careful session handling to avoid conflicts.

4. Command Examples:

   - To create a virtual serial client adapter on VIOS, one might use commands like:

     

     chhwres -m ms02 -r virtualio --rsubtype serial -o a -p ms02-vio1 -s 45 -a adapter_type=client,remote_lpar_name=Machine02,remote_slot_num=0,supports_hmc=0

     ```

   - To start a console session for an LPAR using mkvterm:

     ```bash

     mkvterm -id <LPAR-ID>

     ```

The mkvterm command serves as a bridge between the HMC and virtual serial adapters, allowing administrators to manage LPARs efficiently through console connections. Proper configuration and management of these connections ensure seamless operations within IBM's Power Systems environment.

Comprehensive Guide to Linux Kernel Selftests

The Linux kernel includes a powerful suite of self-tests found under the `tools/testing/selftests/` directory. These self-tests provide a framework for developers to verify specific code paths within the kernel, ensuring the robustness and reliability of individual components. This guide covers how to set up, run, and customize these tests to suit different testing needs.

Overview of Kernel Self-Tests

Kernel self-tests serve as essential tools for validating new code additions or changes to the Linux kernel. They can be run post-kernel build, installation, and boot to validate that code changes do not introduce regressions or unexpected behavior. One critical feature within these self-tests is the hot-plug testing, which allows developers to simulate and verify the dynamic plugging and unplugging of CPUs and memory.

Hot-Plug Tests: Limited and Full Modes

Hot-plug testing, by nature, can be complex due to dependencies on hardware readiness for CPU and memory offlining. To mitigate potential system hangs during these tests, the kernel provides:

Limited Mode: CPU hot-plug is tested on a single CPU, and memory hot-plug is restricted to 2% of available hot-plug-capable memory.

Full Mode: This extensive test mode performs hot-plug actions across all hot-plug capable CPUs and 10% of the memory. This mode can be useful for systems where comprehensive validation of hot-plug readiness is required.

Building and Running Self-Tests

$ make -C tools/testing/selftests

$ make -C tools/testing/selftests run_tests

Alternatively, to both build and run in a single command, use:

$ make kselftest

Note: Some tests, particularly those requiring hardware access, may need root privileges.

Running Specific Self-Tests

Kernel self-tests offer flexibility for targeting specific components or subsystems. To run only the tests related to a specific subsystem, use the `TARGETS` variable:

For example, to run only `ptrace` tests:

$ make -C tools/testing/selftests TARGETS=ptrace run_tests

$ make TARGETS="size timers" kselftest

Refer to the `Makefile` located at `tools/testing/selftests/Makefile` for a complete list of possible test targets.

Running Full Range Hot-Plug Self-Tests

For systems requiring full-scale hot-plug testing, the following commands build and execute the tests across all hot-plug-capable resources.

$ make -C tools/testing/selftests hotplug

$ make -C tools/testing/selftests run_hotplug

 Installing Kernel Self-Tests

To simplify running self-tests on other systems, the `kselftest_install.sh` script allows installation to a default or user-specified location.

Default Installation

$ cd tools/testing/selftests

$ ./kselftest_install.sh

Custom Installation Location

Specify an installation directory for easier management and accessibility:

$ cd tools/testing/selftests

$ ./kselftest_install.sh <install_dir>

Running Installed Self-Tests

After installing, run the installed tests using the provided `run_kselftest.sh` script, which executes all self-tests in the specified installation directory.

$ cd kselftest

$ ./run_kselftest.sh

---------------------------------------

Testing BPF (Berkeley Packet Filter) and fprobe features within Linux kernel self-tests involves several additional steps because these tests require specific kernel configurations and privileges to ensure proper operation. Here’s a breakdown of the steps required:

Enable Required Kernel Configurations

To run BPF and fprobe self-tests, ensure the kernel has the necessary configurations enabled. These settings are often found in the kernel's `.config` file before building the kernel.

CASE 1 : For BPF Self-Tests

Enable the following configurations:

- `CONFIG_BPF`: Enable BPF support

- `CONFIG_BPF_SYSCALL`: Enable BPF syscall interface

- `CONFIG_BPF_JIT`: Enable BPF Just-In-Time (JIT) compiler

- `CONFIG_HAVE_EBPF_JIT`: Enable extended BPF JIT support

- `CONFIG_DEBUG_INFO_BTF`: Enable BPF Type Format (BTF), which is often required for BPF self-tests

- `CONFIG_NET`: Enable networking stack if testing network-related BPF functionality

If running BPF tracing , make sure the following options are also enabled:

- `CONFIG_TRACING`: Enable kernel tracing

- `CONFIG_FUNCTION_TRACER`: Enable function tracing (for eBPF tracing features)

- `CONFIG_BPF_EVENTS`: Enable BPF events for tracing

CASE2: For fprobe Self-Tests

fprobe relies on certain configurations, particularly in the tracing subsystem:

- `CONFIG_KPROBES`: Enable kernel probes

- `CONFIG_FPROBE`: Enable fprobe functionality, which is typically under kernel debugging and tracing options

- `CONFIG_KPROBE_EVENTS`: Enable kprobe events (needed for fprobe usage and testing)

Use `make menuconfig` or `make nconfig` in your kernel source directory to enable these options interactively.

--------------------------------------------

Recompile and Boot into the Custom Kernel

After setting the configurations, recompile the kernel to include the BPF and fprobe functionality, then reboot into this kernel.

$ make -j$(nproc) && make modules_install && make install

$ reboot

Install Required User-Space Utilities

Some BPF and fprobe tests may require user-space tools for BPF program management or tracing. Ensure these tools are installed:

1. bpftool: Used to inspect and manage BPF programs and maps.
2. iproute: Provides tools like `tc` for attaching BPF programs to network traffic.

 

 Run BPF and fprobe Self-Tests : Navigate to the `tools/testing/selftests/` directory in your kernel source and specify the `TARGETS` for BPF and fprobe:

Running BPF Tests  

$ make -C tools/testing/selftests TARGETS=bpf run_tests

BPF self-tests are designed to cover various aspects, such as BPF JIT, network socket filters, and tracing. Some BPF tests may require root privileges, so consider running with `sudo` if you encounter permission errors.

Running fprobe Tests

$ make -C tools/testing/selftests TARGETS=fprobe run_tests

The fprobe tests are generally under the tracing or ftrace section. Since fprobe works closely with kprobes and tracing functionalities, you may see additional output related to probe events.

Troubleshooting Common Issues

Missing Kernel Configurations: If tests fail with messages about unsupported configurations, recheck that all necessary kernel options are enabled.

Permission Issues: Some BPF programs and fprobes require privileged access. Run the tests with `sudo` if needed.

BTF (BPF Type Format) Requirements: Some BPF tests require BTF data, which may need to be enabled explicitly in `.config` (`CONFIG_DEBUG_INFO_BTF`).

By following these steps, you should be able to run BPF and fprobe tests as part of the Linux kernel self-tests, verifying their functionality and stability within your custom-built kernel.

Reference:

https://www.kernel.org/doc/Documentation/kselftest.txt

Overview of the Linux Test Project (LTP) and Submitting Patches to the Mailing List

The Linux Test Project (LTP) is a collaborative initiative aimed at enhancing the reliability, robustness, and stability of the Linux kernel. This project was initiated by SGI, OSDL, and Bull, and is currently developed and maintained by a consortium of major technology companies, including SUSE, Red Hat, Fujitsu, IBM, Cisco, and Oracle. The primary objective of LTP is to provide a comprehensive suite of tests that serve the open-source community.

Goals and Objectives

LTP's primary objective is to provide a collection of automated tools to rigorously test the Linux kernel and associated system libraries. The test suites focus on various kernel components, including file systems, networking, memory management, and system calls. By using LTP, developers can gain valuable insights into potential bugs, stability issues, or areas for performance optimization.

Key Components of LTP:

1. Test Suites: LTP offers several specialized test suites, including:
2.  Syscalls: Verifies the implementation and response of different system calls.
3.  File Systems: Tests various file systems for integrity, performance, and stability.
4.  Networking: Assesses network protocol handling, performance, and error management.
5.  Security: Evaluates the kernel’s security features, enforcing secure data handling and user permissions.

The main goals of LTP include:

1. Testing Automation: Automating the testing process to improve the quality of the Linux kernel and its associated system libraries.
2. Reliability Validation: Ensuring that the Linux kernel can handle various workloads without failure.
3. Robustness Assessment: Testing the kernel's ability to recover from errors and handle unexpected situations gracefully.

Testing Suites : LTP consists of a collection of tools designed specifically for testing the Linux kernel and related features. These testing suites cover a wide range of scenarios to validate different aspects of kernel performance. 

 Key Features:

1. Comprehensive Coverage: The tests are designed to cover various functionalities within the kernel.
2. Stress Testing: Some tests are intended to stress specific components of the system, which can help identify potential issues before they affect production environments.

How LTP Works: The LTP testing workflow follows these primary steps:

1. Initialization: Set up the test environment and prerequisites, configuring the necessary resources.

2. Test Selection: Choose relevant tests based on the kernel area under evaluation—such as file systems, network protocols, or memory management.

3. Test Execution: Run the selected tests on the system to validate the kernel's behavior and performance.

4. Result Collection: Gather data from each test run, noting any failures, errors, or performance metrics.

5. Reporting: Summarize test results with detailed logs and reports for analysis.

Important Warning

It is crucial to note that LTP tests should not be run on production systems. Certain tests like "growfiles, doio, and iogen" are particularly aggressive as they stress the I/O capabilities of the system. These tests are designed to discover or even induce problems, making them unsuitable for environments where stability is critical.

Contribution and Community

LTP is an open-source project that encourages contributions from developers around the world. Users can report issues, submit pull requests, and participate in discussions regarding improvements and new features. The project maintains an active GitHub repository where users can access the latest code, documentation, and testing results.

How to Subscribe: Visit the Mailing List Subscription Page: Go to the LTP Mailing List Info Page to find details about the mailing list. Fill Out Subscription Form: You will need to provide your email address and follow the instructions to complete your subscription.

https://lists.linux.it/listinfo/ltp

This is a mailing list for the Linux Test Project. The list is used for patches, bug reports as well as general discussion about LTP and Linux testing.

Sending Patches: Prepare Your Patch. Make sure your patch is ready for submission. It's advisable to test your patch using GitHub Actions to ensure it compiles cleanly across various architectures and distributions. Use git send-email: The preferred method for sending patches is using git send-email. This will format your patch correctly for the mailing list. Once prepared, send your patch to the mailing list (ltp@lists.linux.it).

To post a message to all the list members, send email to ltp@lists.linux.it

LWN.net is a subscriber-supported publication that focuses on providing in-depth coverage of the Linux and free software development communities. LWN relies on subscriptions to fund its operations, as traditional advertising models have proven ineffective for their needs. Subscribing helps ensure the continuity of their publication. https://lwn.net/Articles/708182/#t

Test case tutorial:

This is a step-by-step tutorial on writing a simple C LTP test, where topics of the LTP and Linux kernel testing will be introduced gradually using a concrete example. Please refer this link

https://linux-test-project.readthedocs.io/en/latest/developers/test_case_tutorial.html

The list of syscalls which are tested under testcases/kernel/syscalls:

• kernel syscalls: 365
• tested syscalls: 341

https://linux-test-project.readthedocs.io/en/latest/users/stats.html

source

NOTE:

Cross-Compilation Process: 

1. Source Code Compilation: process begins with compiling source files (e.g., .c or .cpp) into object files (.o). This is done using a cross-compiler that targets the architecture of the device for which the code is intended
2. Linking: After generating object files, a linker combines them into a single executable file, such as  .ELF (or main.axf). This step may also involve using a linker script that defines memory regions and other configurations specific to the target device1.
3. Conversion: Once the ELF(or .axf) file is created, it can be converted into other formats like .bin, which are suitable for uploading to the target hardware.

For the ppc64le (PowerPC 64-bit Little Endian) architecture ( the equivalent of the .axf** format used in ARM cross-compilation) is typically the ELF (Executable and Linkable Format). The ELF format is widely used for executable files, object code, shared libraries, and core dumps across various architectures, including ppc64le. When cross-compiling for ppc64le, the output binary files are usually in ELF format, which can be identified by the .elf extension or simply no extension at all.

Toolchain Setup: To cross-compile for ppc64le, you need to install a suitable toolchain. A toolchain is a set of programming tools that work together to facilitate the development, building, and deployment of software applications. This typically includes compilers, linkers, debuggers, and libraries that are executed in a sequence where the output of one tool serves as the input for the next. This architecture, developed by the AIM alliance (Apple, IBM, and Motorola), is based on the RISC (Reduced Instruction Set Computing) principles and supports both 32-bit and 64-bit processing.

 Example : install gcc-powerpc64le-linux-gnu

Compiling and Linking: Use the cross-compiler to compile source files:

 powerpc64le-linux-gnu-gcc -o output_binary source.c

This command generates an ELF executable named "output_binary".

 file output_binary

The output should indicate that it is an ELF file for PowerPC architecture and the resulting ELF binaries can be transferred to a ppc64le system for execution. This format allows developers to create executables that are compatible with PowerPC systems, facilitating software development in embedded and server environments.

Tests setup:

The internal LTP library provides a set of features that permits to customize tests behavior by setting environment variables and using specific tests arguments.

Library environment variables:

Following environment variables are expected to be set by LTP users. Therefore, with some exceptions, they have LTP_ prefix. Environment variables with TST_ prefix are used inside LTP shell API and should not be set by users.

https://linux-test-project.readthedocs.io/en/latest/users/setup_tests.html

Installation and tests execution:  Basics requirements to build LTP are the following:

• git
• autoconf
• automake
• make
• gcc
• m4
• pkgconf / pkg-config
• libc headers
• linux headers
• git-email

git clone --recurse-submodules https://github.com/linux-test-project/ltp.git

cd ltp

make autotools

./configure

Running single tests: 

cd testcases/kernel/syscalls/foo

make

PATH=$PATH:$PWD ./foo01

To compile all tests :

make

# install LTP inside /opt/ltp by default

make install

Running tests: To run all the test suites

cd /opt/ltp

# run syscalls testing suite

./kirk -f ltp -r syscalls

Note: Many test cases have to be executed as root.

Test suites (e.g. syscalls) are defined in the runtest directory. Each file contains a list of test cases in a simple format.

Each test case has its own executable or script that can directly executed:

testcases/bin/abort01

# some tests have arguments

testcases/bin/mesgq_nstest -m none

# vast majority of tests have a help

testcases/bin/ioctl01 -h

# Many require certain environment variables to be set

LTPROOT=/opt/ltp PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/wc01.sh

Most commonly, the PATH variable needs to be set and also LTPROOT, but there are a number of other variables which usually kirk sets for you.

How to send a patch:

Here are the detailed steps for cloning the Linux Test Project (LTP), adding tests, building, running test cases, and sending patches to the community

Pre-requisite : subscribe to mailing list as mentioned in above section.

1. Check Your Email ID and Username:

   - Verify your Git configuration by running:  

     cat .gitconfig

2. Navigate to the Patch Directory:

   - Change to the directory where your patch is located    

     cd LTP_vxyz_patch/

3. Clone the LTP Repository:

   - Clone the LTP repository from GitHub:

     git clone git@github.com:linux-test-project/ltp.git

4. Change Directory to LTP:

   - Move into the cloned LTP directory    

     cd ltp

5. Build the Project:

   - Execute the build script to compile the project:   

     ./build.sh

6. Navigate to the Test Case Directory:

   - Go to the specific test case directory you want to modify or add tests  

     cd testcases/kernel/mem/hugetlb/hugeshmget

7. Modify/Create a Test Case:

   - Open the test case file in a text editor (e.g., `vi`) and write or modify your test case:

     vi hugeshmget06.c

8. Compile and Check the Test Case:

   - Build and check your test case for any errors:

     make && make check

9. Run the Test Case:

   - Execute your test case to verify its functionality:

     ./hugeshmget06

10. Run with Input Argument:

    - Optionally, run your test case with an input argument for loop of count=5  (e.g., `-i 5`):

      ./hugeshmget06 -i 5

11. Check Git Status:

    - Check which files have been modified or added in your Git repository    

      git status

12. Stage Your Changes:

    - Add your modified files to the staging area for commit:

      git add ../../../../../runtest/hugetlb hugeshmget06.c ../../.gitignore

13. Check Git Status Again:

    - Verify that your changes are staged correctly:   

      git status

14. Commit Your Changes:

    - Commit your changes with a sign-off message:   

      git commit -s

15. Create a Patch File:

    - Generate a patch file for your commit:    

      git format-patch -1

16. Edit the Patch File Before Sending:

    - Open the generated patch file and edit it as necessary (e.g., update version, email IDs):

      vi 0001-Migrating-the-libhugetlbfs-testcases-shm-gettest.c-t.patch

17. Send the Patch via Email:

    - Use `git send-email` to send your patch to the community mailing list and reviewers:

      git send-email --to=ltp@lists.linux.it --cc=reviewer1@xyz.co --cc=reviewer2@abc.co ./0001-Migrating-the-libhugetlbfs-testcases-shm-gettest.c-t.patch

    - Ensure that you fill in the subject line and other email details as prompted.

18. Respond to Review Comments via Email:

    - After receiving feedback, reply to the email addressing any review comments and mention any new versions of patches based on those comments.

By following these steps, you can effectively contribute to the Linux Test Project by adding new tests, verifying their functionality, and submitting patches for review by the community.

Search your Patch in mailing list :

To search details on patch with search key  for example - "libhugetlbfs" or "shmget-test" or by username "sachinpb"

Example 1 : https://lore.kernel.org/ltp/?q=libhugetlbfs

Example 2 : https://lore.kernel.org/ltp/?q=sachinpb

Example 3: https://lore.kernel.org/ltp/ZvqQxP9KVW6PqFOo@yuki.lan/

[LTP] [ANNOUNCE] The Linux Test Project has been released for SEPTEMBER 2024

Let us test the newly added testcase "hugeshmget06 A hugepage shm test"

Step 1 : Clone the LTP repo from https://github.com/linux-test-project/ltp

Step 2 : Confirm the testcase merged with master branch

Step 3 : Check the new testcase "hugeshmget06.c" existing in "/root/ltp/testcases/kernel/mem/hugetlb/hugeshmget"

Step 4 : Build test cases by running script ./build.sh as shown below:

Step 5 : Do make install  and check the binaries at "/root/ltp-install/"

Step 6 : Execute  the testcase ./hugeshmget06 as shown below 

Step 7: Execute the above testcase in loop of any count . For example count=3 

Conclusion

The Linux Test Project plays a vital role in ensuring that the Linux kernel remains reliable and robust for users across various industries. By providing a structured approach to testing, LTP helps developers identify potential issues early in the development process, ultimately contributing to a more stable operating environment for all Linux users.

For more information or to get involved with LTP, you can visit their [GitHub repository](https://github.com/linux-test-project/ltp).

In this blog, we've explored the Linux Test Project (LTP), covering how to add tests and submit patches to the mailing list. I hope you found this information useful and that it enhances your understanding and contributions to LTP. Best of luck on your journey with LTP! 

Reference :

1) https://linux-test-project.readthedocs.io/en/latest/
2) https://linux-test-project.readthedocs.io/en/latest/index.html
3) https://jitwxs.cn/51dc9e04

Git Unlocked:: A Beginner's Guide to Version Control

Git is source code management systems. Its distributed version control system (DVCS) which facilitates multiple developers and teams to work in parallel.  Git’s primary emphasis is on providing speed while maintaining data integrity. Git also provides ability to perform almost all the operations offline, when network is not available. All the changes can be pushed to the server on network’s availability.

source: GIT workflow structure

Git is designed as a distributed version control system, which means:

Data Structure:

Git uses a data structure called a Merkle tree, where each commit points to its parent commit, forming a chain of versions. Each commit has a unique SHA-1 hash that identifies it and its content.

Objects:

There are four main types of objects in Git:

1. Blobs: Store file data.
2. Trees: Represent directories and contain blobs or other trees.
3. Commits: Point to trees and contain metadata such as author, date, and commit message.
4. Tags: Reference specific commits for marking release points24.

Branches:

Branches are pointers to commits, allowing multiple lines of development within the same repository. The default branch is usually called master or main.

Staging Area:

Before committing changes, files are added to the staging area using git add. This allows users to review changes before finalizing them in a commit.

Remote Repositories:

Git allows synchronization between local repositories and remote ones (like GitHub), enabling collaboration among multiple users.

Some of the commonly used terms are listed below :

1. Repository - is directory that contains all the project files.
2. Clone - it creates a working copy of local repository.
3. Branch - is created to encapsulate the development of new features or to fix bugs.
4. HEAD - points to the last commit.
5. Commit - commits changes to HEAD and not to remote repository.
6. Pull - Gets the changes from the remote repository to the local repository.
7. Push - Commits the local repository changes to the remote repository.

---------------------------------------------------------------------------------------------------------

To understand when to use "git fetch" instead of "git pull", let's break it down into simpler terms with an example.

git fetch: Think of this as checking for updates. It retrieves the latest changes from a remote repository (like GitHub) but does not apply those changes to your current working files. It’s like downloading new episodes of your favorite show but not watching them yet.

git pull: This command does two things at once: it fetches the latest changes and  immediately merges them into your current files. It’s like downloading those episodes and then watching them right away.

When to Use "git fetch"

1. You Want to Review Changes First: If you're working on a project and you want to see what others have done before you integrate their changes, `git fetch` allows you to check for updates without making any changes to your own work right away.

2. Avoiding Conflicts : If you have uncommitted changes in your local files, using `git pull` could create conflicts because it tries to merge changes automatically. By using `git fetch`, you can first see what has changed and then decide how to handle it.

3. Working Offline or Later: If you're in a situation where you can't deal with merging right away (like being on a beach without Wi-Fi), you can fetch the updates now and merge them later when you're ready.

Example : Imagine you're working on a team project.

1. You’re writing code on your laptop.

2. Your teammate pushes some updates to the shared repository while you’re still working.

3. You want to see what they changed without disrupting your current work.

  git fetch origin

  This command checks for any updates from the remote repository named "origin" and downloads the latest information about those changes, but it doesn’t change any of your files yet.

4. After fetching, you can review what your teammate did:

   git log origin/main

   This shows you the commit history of the remote branch so you can see what’s new.

5. Once you're ready and have made sure there are no conflicts, you can merge those changes:

   git merge origin/main

   Now your local files are updated with your teammate’s changes.

This way , use "git fetch" when you want to keep your local repository updated without immediately affecting your working files. This gives you more control over how and when to integrate changes from others, especially useful in collaborative environments or when you're not ready to merge yet.

--------------------------------------------------------------------

The recursive clone of a Git repository:

This refers to the process of cloning a repository along with all its submodules. Submodules are essentially repositories nested inside another repository, allowing you to include external libraries or components as part of your project.

How It Works : When you perform a standard clone using the command:

git clone [repository-url]

It only clones the main repository. If that repository contains submodules, those submodule directories will be created but will remain empty until you initialize and update them.

To clone a repository along with its submodules, you use the `--recurse-submodules` option:

git clone --recurse-submodules [repository-url]

This command does the following:

1. Clones the Main Repository: It creates a local copy of the main repository.

2. Initializes Submodules: It automatically initializes any submodules defined in the ".gitmodules" file.

3. Updates Submodules: It fetches the content of those submodules, ensuring they are populated with the correct data.

Example: Imagine you have a project that relies on a library hosted in another Git repository. If you want to include this library as a submodule in your project, your main repository might look like this:

- Main Repository: "MyProject"

  - Submodule: "ExternalLibrary"

If you were to clone MyProject without the recursive option, you'd end up with:

MyProject/

├── .git/

├── ExternalLibrary/  (empty)

└── other files...

with that option 

git clone --recurse-submodules [repository-url]

You would get:

MyProject/

├── .git/

├── ExternalLibrary/  (with content)

└── other files...

Why use Recursive Clone?

1. Convenience: It saves time by automatically fetching and setting up all necessary components for your project.
2. Consistency: Ensures that all required libraries or modules are at the correct version specified in the main repository.

Using a recursive clone is essential when working with projects that depend on multiple repositories to ensure everything is correctly set up from the start.

$ git clone --recursive git@github.xyz.com:smpi/xyz-tests.git

Cloning into 'xyz-tests'...

remote: Enumerating objects: 24, done.

remote: Counting objects: 100% (24/24), done.

remote: Compressing objects: 100% (21/21), done.

remote: Total 18759 (delta 13), reused 7 (delta 3), pack-reused 18735

Receiving objects: 100% (18759/18759), 125.23 MiB | 33.33 MiB/s, done.

Resolving deltas: 100% (12772/12772), done.

Updating files: 100% (5233/5233), done.

Submodule 'smpi-ci/mtt' (https://github.com/open-mpi/mtt-legacy.git) registered for path 'smpi-ci/mtt'

Submodule 'tests/hpc-smpi-fvt' (git@github.xyz.com:smpi/hpc-smpi-fvt.git) registered for path 'tests/hpc-smpi-fvt'

Submodule 'tests/ompi-tests' (git@github.xyz.com:smpi/ompi-tests.git) registered for path 'tests/ompi-tests'

Cloning into '/data/nfs_smpi_ci/xyz-tests/smpi-ci/mtt'...

remote: Enumerating objects: 1abc, done.        

remote: Counting objects: 100% (21/21), done.        

remote: Compressing objects: 100% (18/18), done.        

remote: Total 13184 (delta 7), reused 10 (delta 3), pack-reused 13163        

Receiving objects: 100% (13184/13184), 3.94 MiB | 20.67 MiB/s, done.

Resolving deltas: 100% (8686/8686), done.

Cloning into '/data/nfs_smpi_ci/xyz-tests/tests/hpc-smpi-fvt'...

remote: Enumerating objects: 15511, done.        

remote: Total 15511 (delta 0), reused 0 (delta 0), pack-reused 15511        

Receiving objects: 100% (15511/15511), 81.99 MiB | 28.47 MiB/s, done.

Resolving deltas: 100% (10318/10318), done.

Cloning into '/data/nfs_smpi_ci/xyz-tests/tests/ompi-tests'...

remote: Enumerating objects: 36104, done.        

remote: Total 36104 (delta 0), reused 0 (delta 0), pack-reused 36104

Receiving objects: 100% (36104/36104), 103.97 MiB | 26.81 MiB/s, done.

Resolving deltas: 100% (23820/23820), done.

Submodule path 'smpi-ci/mtt': checked out 'mnccd42d37c232883d3a600ac4151868a3327b7'

Submodule path 'tests/hpc-smpi-fvt': checked out 'abc5ab4afacfacccc04dacb2c55a41477d2c02'

Submodule path 'tests/ompi-tests': checked out 'xyx7f6e194df5bbc16e836f8d63556be363a94ca5'

$ 

------------

NOTE: With version 2.13 of Git and later, --recurse-submodules can be used instead of --recursive

With  older Git , you can use:

git clone --recursive git://github.com/foo/bar.git

For already cloned repos, or older Git versions, use:

git clone git://github.com/foo/bar.git

cd bar

git submodule update --init --recursive

------------

The "git merge" :

The "git merge" command is a fundamental feature in Git that allows developers to combine changes from different branches into a single branch. This process is essential for integrating various lines of development, such as feature branches or bug fixes, into the main codebase.

What git merge Does?

1. Combines Changes: The primary function of "git merge" is to integrate the histories of two branches. When you execute "git merge <branch-name>", Git takes the changes from the specified branch and merges them into your current branch, which is often referred to as the "receiving branch".

2. Creates a Merge Commit: If the branches have diverged (i.e., both have new commits since they last shared a common ancestor), Git creates a new commit known as a merge commit. This commit has two parent commits: one from each branch being merged. This allows Git to maintain a complete history of changes.

3. Finds the Common Base: Before merging, Git identifies the common ancestor (or merge base) of the two branches. It then computes the differences (diffs) between this common ancestor and each of the branches to apply those changes simultaneously.

4. Handling Conflicts: If there are conflicting changes in the same part of a file from both branches, Git will pause the merge process and prompt you to resolve these conflicts manually. Once resolved, you can finalize the merge with a commit.

Types of Merging

1) Fast-Forward Merge: If the current branch is directly behind the branch being merged (meaning it has no new commits since their last common ancestor), Git can simply move the pointer of the current branch forward to the latest commit of the other branch without creating a new commit 

2) Three-Way Merge : When both branches have diverged, a three-way merge occurs. Here, Git uses three points (the two branch tips and their common ancestor) to create a new merge commit that reconciles changes from both branches.

Example: To illustrate how "git merge" works, let's go through a practical example involving two branches: main and feature.

1. Initial Setup: You start with a repository that has a `main` branch and you create a new branch called `feature` for development.

   git checkout -b feature main

2. Making Changes in the Feature Branch:

   You make some changes in the `feature` branch and commit them.

   # Edit some files

   git add <file>

   git commit -m "Add new feature"

3. Switching Back to Main:

   After completing your work on the "feature" branch, you switch back to the "main" branch.

   git checkout main

4. Making Additional Changes in Main: While you were working on the "feature", someone else made changes to the "main" branch. You can update it before merging:

   git pull origin main

5. Merging the Feature Branch into Main: Now, you can merge the changes from the "feature" branch into the "main" branch.

  

   git merge feature

If there are conflicting changes (e.g., both branches modified the same line in a file), Git will pause and prompt you to resolve these conflicts manually before completing the merge:

# Resolve conflicts in your editor, then stage the resolved files

git add <resolved-file>

# Complete the merge after resolving conflicts

git commit -m "Merge feature into main with conflict resolution"

Using "git merge", developers can effectively integrate changes from multiple branches, making it easier to collaborate on projects. Understanding how to handle different types of merges and resolve conflicts is crucial for maintaining a clean project history and ensuring smooth collaboration among team members.

-----------

The "git rebase" :

Git rebase is a powerful command in Git that allows you to integrate changes from one branch into another by moving the base of your current branch to a different commit. This process effectively rewrites the commit history, making it appear as though you started your work from a different point in the project history.

What Does Git Rebase Do?

1. Changing the Base: When you perform a rebase, you change the base of your current branch to a specified commit from another branch. This means that all the commits in your current branch will be reapplied on top of the new base commit, creating new commits in the process.

2. Linear History: One of the main advantages of rebasing is that it helps maintain a clean and linear project history. This is particularly useful for simplifying the commit log and making it easier to follow changes over time. Instead of having a branching structure, rebasing makes it look like all changes were made sequentially.

3. Interactive Rebasing: Git rebase can be performed in two modes: standard and interactive. The interactive mode allows you to modify commits during the rebase process, such as editing commit messages, squashing multiple commits into one, or even removing commits altogether.

To perform a standard rebase, you would typically use:

git rebase <base-branch>

For example, if you want to rebase your feature branch onto the main branch:

git checkout feature

git rebase main

This command will take all commits from "feature" and apply them on top of "main", effectively updating "feature" with the latest changes from `main`.

 Example: 

1. Initial Setup

   - You have two branches: main and feature

   - You make several commits on feature while others continue to update main.

2. Rebasing:

   - To incorporate the latest changes from main into feature, you would execute:

   git checkout feature

   git rebase main

3. Resolving Conflicts:

   - If there are conflicts during the rebase, Git will pause and allow you to resolve them. After resolving conflicts in the affected files, you would continue the rebase with:

   git add <resolved-file>

   git rebase --continue

Advantages of Git Rebase

1. Cleaner Commit History: By avoiding unnecessary merge commits, rebasing results in a more straightforward project history that is easier to read and understand.
2. Easier Debugging: A linear history simplifies tracking down bugs and understanding how features were developed over time.
3. Better Collaboration: Rebasing helps keep feature branches up-to-date with ongoing changes in the main branch, which can reduce conflicts when it's time to merge back into main.

What Happens During Rebase?

- Updating Your Work: When you run this command, Git looks at the latest commits in the `main` branch and applies your feature branch's commits as if you started working on them after the latest changes in main. This effectively updates your feature branch with any new changes that have been made in main.

- Cleaner History: Unlike merging, which creates a new commit that combines both branches (often making the history look complicated), rebasing keeps your project history neat and linear. It looks like all your work was done after the latest changes from main, even if it wasn't.

 Example : Imagine you are working on a project:

1. You create a branch called feature to add a new feature.

2. While you're working, someone else adds new updates to the main branch.

3. To ensure your feature includes those updates, you use git rebase main.

4. After running this command, your commits from feature will be placed on top of the latest commits from main, making it look like you developed your feature with all the latest updates in mind

NOTE:

While rebasing can be very useful, it is important to note that rewriting history can lead to complications if not handled carefully—especially if you've already pushed your changes to a shared repository. It’s generally advised not to rebase public branches that others may be using. 

Git rebase is a powerful tool for managing project history and integrating changes across branches while maintaining clarity and organization in your commit log.

Git rebase and the combination of `git pull` followed by `git merge` are related but not equivalent operations. Here’s a breakdown of their differences and how they function:

Git Pull vs. Git Rebase

1. Git Pull:

   - The `git pull` command is essentially a combination of two commands: git fetch followed by git merge. When you run git pull, Git fetches the latest changes from the remote branch and then merges those changes into your current branch, creating a new merge commit if necessary. This can lead to a more complex commit history with multiple merge commits, especially in collaborative environments where many changes are being made concurrently.

   git pull origin main

2. Git Rebase:

   - The git rebase command, on the other hand, takes the commits from your current branch and re-applies them on top of another branch (often the updated main branch). This effectively rewrites the commit history, creating a linear progression of commits without additional merge commits. This can make the project history cleaner and easier to follow

   git rebase main

Key Differences

History Structure: 

  - Merge: Results in a branching history with merge commits that reflect the integration points between branches.

  - Rebase: Produces a linear history that appears as if all changes were made sequentially on top of the base branch.

Conflict Resolution:

  - During a merge, if conflicts arise, you resolve them and then create a new merge commit.

  - During a rebase, conflicts must be resolved at each commit being reapplied, and you continue the rebase process after resolving each conflict

Use Cases:

  - Use git pull when you want to quickly integrate remote changes into your local branch, accepting the additional merge commit.

  - Use git rebase when you want to keep a clean history, especially in feature branches that have not yet been shared with others 

That way,  use git fetch when you want to check for updates without altering your current work, git pull for quickly updating your branch with remote changes (but be cautious of conflicts), and git merge when you want to combine different branches of work. 

Git Fork : A fork is a complete copy of a repository that allows you to make changes independently without affecting the original repository. It is often used in collaborative projects, especially on platforms like GitHub.

Purpose: Forking is typically used when you want to contribute to someone else's project. You can create your own version of the repository to experiment with or develop features without needing direct access to the original repository.

Isolation: A fork creates an entirely separate repository, meaning that changes made in your fork do not impact the original repository unless you explicitly submit a pull request.

Fprobe: Efficient Kernel Function Tracing in Linux

Fprobe is a function entry/exit probe mechanism in the Linux kernel, built on top of the ftrace framework. It allows developers to attach callbacks to function entry and exit points, similar to kprobes and kretprobes, but with improved performance for multiple functions through a single handler. This makes fprobe particularly useful for tracing and debugging kernel functions without the overhead associated with full ftrace features.

Key Features of Fprobe:

Performance: Provides faster instrumentation for multiple functions compared to traditional kprobes and kretprobes.

Structure: The fprobe structure includes fields for entry and exit handlers, allowing customized behavior when functions are entered or exited.

Registration: Fprobes can be registered using various methods, including function-name filters or direct address registration.

The usage of fprobe :

The fprobe is a wrapper of ftrace (+ kretprobe-like return callback) to attach callbacks to multiple function entry and exit. User needs to set up the struct fprobe and pass it to register_fprobe(). Typically, fprobe data structure is initialized with the entry_handler and/or exit_handler as below.

NOTE: callbacks refer to functions that are executed in response to specific events, particularly when a function is entered or exited

A typical fprobe setup might look like this:

struct fprobe fp = {
    .entry_handler = my_entry_callback,
    .exit_handler = my_exit_callback,
};
register_fprobe(&fp, "func*", "func2");

-----------------------------------------------------

Simple C program that demonstrates how to use fprobe in the Linux kernel. This program sets up an fprobe to monitor the entry and exit of a specific kernel function. For this example, we'll assume that you want to probe a function named "target_function".

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/fprobe.h>
#include <linux/init.h>

// Function prototypes
int my_entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data);
void my_exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data);
void target_function(void); // Declare target_function prototype

// Define the fprobe structure
static struct fprobe fp = {
    .entry_handler = my_entry_callback,
    .exit_handler = my_exit_callback,
};

// The function we want to probe
void target_function(void) {
    printk(KERN_INFO "Inside target_function\n");
}

// Entry callback
int my_entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data) {
    printk(KERN_INFO "Entered target_function\n");
    return 0; // Return an int as expected
}

// Exit callback
void my_exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data) {
    printk(KERN_INFO "Exited target_function\n");
}

// Module initialization
static int __init my_module_init(void) {
    // Register the fprobe for the target function
    if (register_fprobe(&fp, "target_function", NULL)) {
        printk(KERN_ERR "Failed to register fprobe\n");
        return -1;
    }

    printk(KERN_INFO "Fprobe registered successfully\n");

    // Call the target function to see the callbacks in action
    target_function();

    return 0;
}

// Module cleanup
static void __exit my_module_exit(void) {
    // Unregister the fprobe
    unregister_fprobe(&fp);
    printk(KERN_INFO "Fprobe unregistered successfully\n");
}

module_init(my_module_init);
module_exit(my_module_exit);

MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Fprobe Example Module");
MODULE_AUTHOR("SACHIN P BAPPALIGE");

--------------------------------------------------------------------------

Compilation and Loading:

Step 1 : Save the code in a file named fprobe_example.c.

Step 2 : Create a Makefile as shown below

obj-m += fprobe_example.o

all:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

--------------------------------------

# make
make -C /lib/modules/6.12.0-rc2+/build M=/root/fprobe modules
make[1]: Entering directory '/root/linux'
  CC [M]  /root/fprobe/fprobe_example.o
  MODPOST /root/fprobe/Module.symvers
  CC [M]  /root/fprobe/fprobe_example.mod.o
  CC [M]  /root/fprobe/.module-common.o
  LD [M]  /root/fprobe/fprobe_example.ko
  BTF [M] /root/fprobe/fprobe_example.ko
make[1]: Leaving directory '/root/linux'

# ls

Makefile  Module.symvers  fprobe_example.c  fprobe_example.ko  fprobe_example.mod  fprobe_example.mod.c  fprobe_example.mod.o  fprobe_example.o  modules.order

--------------------------------------------------

Step 3 : Insert module-->  insmod fprobe_example.ko

Step 4: Check kernel logs with dmesg to see output from the callbacks.

Step 5: Remove the module using  -->  rmmod fprobe_example

Here’s a C program that demonstrates how to use fprobe to monitor the do_fork function in the Linux kernel. This example sets up entry and exit callbacks to log when the do_fork function is called and when it returns.

-------------------------------------

#include <linux/module.h>

#include <linux/kernel.h>

#include <linux/fprobe.h>

#include <linux/init.h>

#include <linux/sched.h>

// Function prototypes for the entry and exit callbacks

int my_entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data);

void my_exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data);

// Define the fprobe structure

static struct fprobe fp = {

    .entry_handler = my_entry_callback,

    .exit_handler = my_exit_callback,

};

// Entry callback for do_fork

int my_entry_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data) {

    printk(KERN_INFO "do_fork called: PID = %d\n", current->pid);

    return 0; // Return an int as required

}

// Exit callback for do_fork

void my_exit_callback(struct fprobe *fp, unsigned long entry_ip, unsigned long args, struct pt_regs *regs, void *data) {

    printk(KERN_INFO "do_fork returned: PID = %d\n", current->pid);

}

// Module initialization function

static int __init my_module_init(void) {

    // Register the fprobe for the do_fork function

    if (register_fprobe(&fp, "do_fork", NULL)) {

        printk(KERN_ERR "Failed to register fprobe for do_fork\n");

        return -1;

    }

    printk(KERN_INFO "Fprobe registered successfully for do_fork\n");

    return 0;

}

// Module cleanup function

static void __exit my_module_exit(void) {

    // Unregister the fprobe

    unregister_fprobe(&fp);

    printk(KERN_INFO "Fprobe unregistered successfully for do_fork\n");

}

module_init(my_module_init);

module_exit(my_module_exit);

MODULE_LICENSE("GPL");

MODULE_DESCRIPTION("Fprobe Example for do_fork");

MODULE_AUTHOR("SACHIN P BAPPALIGE");

------------------------------------------------------------------------------

NOTE: Fprobe and kprobe are both mechanisms used in the Linux kernel for tracing and debugging, but they have distinct characteristics and use cases.

Kprobe :

Purpose: Kprobes allow developers to insert probes at almost any kernel function entry or exit point. They enable dynamic instrumentation by setting breakpoints on specified functions.

Structure: Each kprobe consists of a handler that is invoked when the probe is hit, allowing for custom actions or logging.

Performance: Kprobes can introduce overhead, especially when multiple probes are registered, as they rely on software breakpoints (SWBP) which can be slower compared to other methods.

Recursion Handling: Kprobes maintain a per-CPU variable (`current_kprobe`) to manage recursion safely, which prevents re-entry into the same probe handler.

Developers commonly probe various Linux kernel functions to gather insights about system performance, debug issues, and monitor behavior. Here are some typical functions that are often probed:

Commonly Probed Kernel Functions : 

1)  do_fork: Used for process creation. Probing this function can help track process spawning and resource allocation.

2)  sys_read / sys_write: These functions handle system calls for reading from and writing to files. Probing them allows developers to monitor file I/O operations.

3)  schedule: Responsible for context switching between processes. Probing this function can provide insights into scheduling behavior and CPU usage.

4)  kmalloc / kfree: Functions for memory allocation and deallocation in the kernel. Probing these can help identify memory usage patterns and potential leaks.

5)  vfs_read / vfs_write:Virtual filesystem layer functions that manage file operations. Probing these functions helps in understanding file access patterns across different filesystems.

7)  tcp_sendmsg / tcp_recvmsg:Used for sending and receiving TCP messages. Probing these can assist in analyzing network performance and behavior.

8)  netif_receive_skb: This function processes incoming packets in the network stack. Probing it can help monitor network traffic handling.

9)  exit_notify: Handles the cleanup of a process upon exit. Probing this function can track process termination and resource cleanup.

10) __wake_up: Used to wake up processes waiting on a condition variable. Probing this can provide insights into synchronization and concurrency issues.

11) File System Operations: Functions like lookup, create, and unlink are also commonly probed to monitor filesystem interactions

In summary, while both kprobe and fprobe serve the purpose of tracing kernel functions, fprobe is tailored for efficiency and ease of use in scenarios requiring multiple function tracing. Kprobes offer broader capabilities but at the cost of performance overhead and complexity.

Reference :

1) https://docs.kernel.org/trace/fprobe.html