Linux security and kernel Lockdown - kernel image access prevention feature

Linux has a long history of security-focused development and has been used in many high-security environments, such as military and government organizations. Linux is highly customizable, which allows administrators to tailor security configurations to their specific needs. For example, security modules like SELinux and AppArmor can be configured to enforce highly granular access control policies. Many Linux distributions include security-focused features, such as hardening patches and secure boot support, by default.The open-source nature of Linux allows for community-driven development and auditing, which can help to uncover security vulnerabilities and improve the overall security of the system. Linux containers, such as Docker and Kubernetes, have become increasingly popular in recent years and offer a more secure alternative to traditional virtualization solutions. Linux is widely used in cloud environments and has many built-in features for secure cloud deployments, such as network isolation and encryption. It is constantly being updated and improved with new security features and bug fixes, making it one of the most secure operating systems available.

The Kernel Lockdown feature is designed to prevent both direct and indirect access to a running kernel image, attempting to protect against unauthorized modification of the kernel image and to prevent access to security and cryptographic data located in kernel memory, whilst still permitting driver modules to be loaded. This is security feature, the Linux Security Module (LSM, nicknamed “lockdown”). It does promise to bring additional security to one of the most widely-used and hardened kernels on the market. The lockdown feature’s aim is to restrict various pieces of kernel functionality.  There are two modes available to the lockdown module: Integrity and Confidentiality. When in Integrity mode, kernel features which would allow userland code to modify the running kernel are disabled. When in Confidentiality mode, userland code to extract confidential information from the kernel will be disabled.First off, it will restrict access to kernel features that may allow arbitrary code execution by way of code supplied by any application or service outside of the kernel (aka “userland”). The new feature will also block processes from reading/writing to /dev/mem and /dev/kmem memory, as well as block access to opening /dev/port (as a means to prevent raw ioport access). Other features include:

1. Enforcing kernel module signatures.
2. Prevents even the root account from modifying the kernel code.
3. Kexec reboot (in case secure boot being enabled does not keep the secure boot mode in new kernel).
4. Lockdown of hardware that could potentially generate direct memory addressing (DMA).
5. Lockdown of KDADDIO, KDDELIO, KDENABIO and KDDISABIO console ioctls.

where

• The KDADDIO, KDDELIO, KDENABIO, and KDDISABIO console ioctls are used to manage console input/output (I/O) on Linux systems. Here's a brief overview of each of these ioctls:
• KDADDIO: This ioctl is used to add a new input/output device to the console. When a new device is added using KDADDIO, it can be used to send input to or receive output from the console.
• KDDELIO: This ioctl is used to remove an input/output device from the console. When a device is removed using KDDELIO, it is no longer able to send input to or receive output from the console.
• KDENABIO: This ioctl is used to enable input/output from a specific device on the console. When a device is enabled using KDENABIO, it can be used to send input to or receive output from the console.
• KDDISABIO: This ioctl is used to disable input/output from a specific device on the console. When a device is disabled using KDDISABIO, it is no longer able to send input to or receive output from the console.

NOTE: The "KD" in these console ioctls stands for Keyboard Display. The term "keyboard display" is used to refer to the console on a computer system, which includes the keyboard and screen used to interact with the system

If a prohibited or restricted feature is accessed or used, the kernel will emit a message that looks like:

        Lockdown: X: Y is restricted, see man kernel_lockdown.7

where X indicates the process name and Y indicates what is restricted. On an EFI-enabled x86 or arm64 machine, lockdown will be automatically enabled if the system boots in EFI Secure Boot mode.

Coverage: When lockdown is in effect, a number of features are disabled or have their use restricted.  This includes special device files and kernel services that allow direct access of the kernel image:

              /dev/mem
              /dev/kmem
              /dev/kcore
              /dev/ioports
              BPF
              kprobes

and the ability to directly configure and control devices, so as to prevent the use of a device to access or modify a kernel image:The use of module parameters that directly specify hardware parameters to drivers through the kernel command line or when loading a module.

The term "lockdown" refers to a set of security features in the Linux kernel that are designed to prevent even privileged users, such as the root user, from bypassing certain security restrictions. These features are intended to provide an additional layer of protection against malicious software and unauthorized access to sensitive information.

There are two main components to the lockdown feature:

Integrity measurement: This feature prevents changes to the kernel's security settings, such as disabling secure boot or loading unsigned kernel modules, even by users with root privileges.

Confidentiality protection: This feature prevents user space processes from accessing certain sensitive information, such as kernel memory or hardware resources, even if the processes are running with root privileges.

The lockdown feature is a powerful tool for enhancing the security of Linux systems, particularly in high-security environments or those where data privacy is a top concern. However, it can also limit the flexibility of the system, so it's important to carefully consider the trade-offs before enabling this feature.

---------------------------------------------------------

The lockdown feature and SELinux are both security features in the Linux kernel, but they serve different purposes and work independently of each other.

SELinux is a mandatory access control (MAC) system that enforces a set of security policies to determine what processes and users can access specific resources, such as files or network ports. It operates by labeling resources with a security context and assigning labels to users and processes. The security policies defined in SELinux are enforced by the kernel and can prevent unauthorized access and other security breaches.

The lockdown feature, on the other hand, is designed to prevent even privileged users, including those with root privileges, from bypassing certain security restrictions. It achieves this by restricting access to certain kernel features and preventing modifications to the kernel's security settings.

When both SELinux and the lockdown feature are enabled, they work together to provide a comprehensive security solution. SELinux enforces mandatory access controls to restrict access to resources, while the lockdown feature ensures that even privileged users cannot bypass certain security restrictions. This can help to prevent security breaches caused by malicious software or unauthorized access to sensitive information.

The combination of SELinux and the lockdown feature provides a powerful security solution for Linux systems. It's important to carefully configure and manage these features to ensure that they do not interfere with normal system operations or cause unintended consequences.

The idea of effectively rendering the root account less capable of working with a system (on a kernel level), might be considered (to some) a disservice to Linux (and Linux administrators). However, in the realm of business, absolute security is a necessity — especially on machines that house sensitive business/customer data. When the root account is under a form of strict lockdown, malicious code would be significantly more challenging to run rampant on a system. This could lead to fewer data breaches. And because the kernel developers are making the lockdown feature “optional,” it is possible for enterprise admins to enable the feature on production machines that store such sensitive data. Conversely, on standard desktop machines (or developer machines) the feature can remain disabled.

Linux kernel has several security features built into it to protect against various types of security threats. Some of the additional security features of Linux kernel include:

1) AppArmor: AppArmor is a mandatory access control (MAC) system that restricts the capabilities of individual applications or processes. It can be used to enforce security policies that limit the actions of individual applications, such as restricting access to certain files or network resources.

2) Control Groups (cgroups): cgroups provide a way to organize and manage system resources, such as CPU, memory, and I/O bandwidth, among different processes. This helps to prevent individual processes from monopolizing system resources, which can improve system performance and stability.

3) Kernel SamePage Merging (KSM): KSM allows multiple identical memory pages to be merged into a single page, reducing memory usage and improving system performance. However, this feature also presents a potential security risk, as it could allow an attacker to create a malicious page that looks like a legitimate page, thereby bypassing memory protection measures.

4) Executable Space Protection: Executable Space Protection is a security feature that prevents execution of code from memory pages that are marked as data or stack. This helps to prevent buffer overflow and other types of attacks that rely on executing code in memory regions that are not intended for code execution.

5) Secure Boot: Secure Boot is a security feature that ensures that only trusted software is executed during system boot-up. It uses cryptographic signatures to verify the authenticity of boot loaders and other critical components of the system, preventing unauthorized or malicious software from running at boot time.

6) Address Space Layout Randomization (ASLR): This feature randomizes the memory layout of user space programs, making it more difficult for attackers to exploit vulnerabilities in the program's code.

7) Seccomp: This feature provides a mechanism for filtering system calls that can be made by a process, allowing administrators to restrict the system calls that can be made by certain programs.

8) Trusted Platform Module (TPM): This is a hardware-based security feature that provides a secure storage area for cryptographic keys and other sensitive data. It can be used to enhance the security of system booting, disk encryption, and other security-related functions.

9)  SELinux similar to AppArmor. These are two popular security modules that provide mandatory access control (MAC) enforcement in the Linux kernel. They use security policies to define what resources, such as files and network ports, can be accessed by which processes and users.

-----------------------------

The Linux kernel includes a variety of cryptography algorithms that can be used to provide secure communication, storage, and other security-related functions. Here are some of the key cryptography algorithms and features in the Linux kernel:

1) Advanced Encryption Standard (AES): AES is a symmetric encryption algorithm that is widely used for data encryption. The Linux kernel includes an implementation of AES that can be used by applications and other kernel subsystems.

2) RSA: RSA is an asymmetric encryption algorithm that is used for digital signatures and key exchange. The Linux kernel includes an implementation of RSA that can be used by applications and other kernel subsystems.

3) SHA: SHA (Secure Hash Algorithm) is a family of cryptographic hash functions that are used for digital signatures, data integrity checking, and other security-related functions. The Linux kernel includes implementations of several SHA algorithms, including SHA-1, SHA-256, and SHA-512.

4) Random Number Generation: Random number generation is a critical component of many cryptographic functions. The Linux kernel includes several sources of entropy that are used to generate high-quality random numbers for use in cryptography algorithms.

5) Cryptographic API: The Linux kernel includes a Cryptographic API that provides a standard interface for using cryptographic functions in kernel modules and applications. The API includes support for a wide range of cryptographic algorithms and features, including those listed above.

6) Filesystem Encryption: The Linux kernel includes support for encrypting filesystems using the dm-crypt subsystem. This allows for encrypted storage of sensitive data and can be used to protect against data theft in the event of a system breach.

7) IPSec: IPSec is a protocol suite for securing IP communications, including VPNs and other types of network connections. The Linux kernel includes support for IPSec, which can be used to secure network communications between Linux systems.

Linux kernel has a wide range of built-in security features that can help to protect against various types of security threats. Linux-based security offers a wide range of benefits, including customizability, community-driven development, and a long history of use in high-security environments. These factors have helped to make Linux a popular choice for organizations looking to enhance the security of their systems and data.

Reference:

https://man7.org/linux/man-pages/man7/kernel_lockdown.7.html

Non-Uniform Memory Access Architecture

Non-Uniform Memory Access (NUMA) is a computer memory design used in multiprocessors, where the memory access time depends on the distance between the CPU and the memory.  In a NUMA system, each CPU has access to its own local memory as well as remote memory, which can cause performance issues if not managed properly.

In the Non-Uniform Memory Access (NUMA) architecture, the path from processor to memory is non-uniform. This organization enables the construction of systems with a large number of processors, and hence the association with very large systems. A NUMA system with cache-coherent memory running a single OS image is still an SMP system. A general representation of the NUMA system is shown below.

NUMA Architecture

The system is comprised of multiple nodes each with 2-4 processors, a memory controller, memory and perhaps IO. There might be a separate node controller, or the MC and NC could be integrated. The nodes could be connected by a shared bus, or may implement a cross-bar.

 By classifying memory location bases on signal path length from the processor to the memory, latency and bandwidth bottlenecks can be avoided. This is done by redesigning the whole system of processor and chipset.  AMD Opteron family was introduced featuring integrated memory controllers with each CPU owning designated memory banks. Each CPU has now its own memory address space. A NUMA optimized operating system such as ESXi allows workload to consume memory from both memory addresses spaces while optimizing for local memory access. Let’s use an example of a two CPU system to clarify the distinction between local and remote memory access within a single system.

Source

The memory connected to the memory controller of the CPU1 is considered to be local memory. Memory connected to another CPU socket (CPU2)is considered to be foreign or remote for CPU1. Remote memory access has additional latency overhead to local memory access, as it has to traverse an interconnect (point-to-point link) and connect to the remote memory controller. As a result of the different memory locations, this system experiences “non-uniform” memory access time.

HP Prema:

A node comprises four processors, memory, IO and a pair of node controllers. There are three IOH devices in the system. The processors and memory on the CPU board are connected to the XNC board with the node controllers.

Source

The "isolcpus" kernel parameter is used to isolate one or more CPUs from the kernel scheduler.  This is typically used for running real-time or high-performance applications that require dedicated CPU resources.  However, it does not have any direct relationship with NUMA nodes.

vi  /etc/default/grub

Find the line that starts with "GRUB_CMDLINE_LINUX_DEFAULT" and add "numa=off" to the end of the line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash numa=off"

Regenerate the GRUB configuration file by running the following command:

update-grub

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the system for the changes to take effect

Note that after disabling NUMA, the system will treat all memory as a single, uniform memory pool. This may not always improve performance

---------------

Certainly, here's an example of how to use the "isolcpus" command to isolate CPU cores from the kernel scheduler:

Find out the number of available CPU cores by running:

cat /proc/cpuinfo | grep processor | wc -l

To isolate one or more CPU cores from the kernel scheduler, append the "isolcpus" parameter to the kernel boot command in the GRUB configuration file. For example, to isolate CPU core 0, edit the GRUB configuration file by running:

/etc/default/grub

Add the "isolcpus" parameter followed by the CPU core number(s) to the end of the "GRUB_CMDLINE_LINUX_DEFAULT" line. For example:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=0"

If you want to isolate multiple cores, separate them with commas. For example:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=0,2"

save and update-grub

Reboot the system for the changes to take effect.

After isolating the specified CPU cores with the "isolcpus" parameter, you can assign them to a specific process using the "taskset" command. For example, to run a process on CPU core 0, run:

taskset -c 0 <command>

Note that isolating CPU cores can affect system performance, so it's important to test your application's performance before and after isolating CPU cores to see if it has any effect on performance.

# cat /proc/cpuinfo | grep processor | wc -l
16
# taskset -c 0 hostname
host1
# taskset -c 18 hostname
taskset: failed to set pid 11896's affinity: Invalid argument
# taskset -c 16 hostname
taskset: failed to set pid 12131's affinity: Invalid argument
# taskset -c 15 hostname
host1
# taskset -c 11,15 hostname
host1
 #

on PPC arch :  /etc/grub.conf

Find the line that starts with "append" and add "isolcpus" parameter followed by the CPU core number(s) to the end of the line. For example

append="quiet splash isolcpus=0,2"

Ater isolating the specified CPU cores with the "isolcpus" parameter, you can assign them to a specific process using the "taskset" command

check your  kernel supports isolcpus:

grep -i isolcpus /boot/config-$(uname -r)

This command searches for the "isolcpus" parameter in the kernel configuration file for the currently running kernel.

If the output of the command shows a line that looks like this:

CONFIG_ISOLCPU_PROC=y

then your kernel supports isolating CPU cores with the "isolcpus" parameter.

Some Linux distributions may not include the kernel configuration file in the /boot directory. In that case, you may need to install the "kernel-devel" or "kernel-source" package to access the kernel configuration file

NOTE: Another way of making CPU offline:
echo 0 > /sys/devices/system/cpu/cpu7/online (to offline cpu)
change cpu number with required cpu

Source

Simultaneous multithreading (SMT) is a processor design that combines hardware multithreading with superscalar processor technology. Simultaneous multithreading can use multiple threads to issue instructions each cycle.

Example: How enable SMT  and check  on power architecture(PPC): 

# cat smt.sh
while [ 1 ]
do
        ppc64_cpu --smt=off
        ppc64_cpu --smt
        ppc64_cpu --smt=on
        ppc64_cpu --smt
        ppc64_cpu --smt=2
        ppc64_cpu --smt
        ppc64_cpu --smt=4
        ppc64_cpu --smt
done

-----------------------------------END--------------------------------------------------------------

High Performance Network Adapters and protocols

High performance network adapters are designed to provide fast and efficient data transfer between servers, storage systems, and other devices in a data center or high-performance computing environment. They typically offer advanced features such as high bandwidth, low latency, RDMA support, and offload capabilities for tasks such as encryption and compression. These adapters are often used in high-performance computing, cloud computing, and data center environments to support large-scale workloads and high-speed data transfer. Some examples of high-performance network adapters include:

• Mellanox ConnectX-6 and ConnectX-6 Dx
• Intel Ethernet Converged Network Adapter X710 and X722
• Broadcom BCM957810A1008G Network Adapter
• QLogic QL45212HLCU-CK Ethernet Adapter
• Solarflare XtremeScale X2522/X2541 Ethernet Adapter
• Chelsio T6 and T6E-CR Unified Wire Adapters

High-performance network adapters typically use specialized protocols that are designed to provide low-latency and high-bandwidth communication between systems. Some examples of these protocols include:

1. Remote Direct Memory Access (RDMA): A protocol that allows data to be transferred directly between the memory of one system and another, without involving the CPU of either system.
2. RoCE (RDMA over Converged Ethernet): An extension of RDMA that allows RDMA traffic to be carried over Ethernet networks.
3. iWARP: A protocol that provides RDMA capabilities over standard TCP/IP networks.
4. InfiniBand: A high-speed interconnect technology that provides extremely low-latency and high-bandwidth communication between systems.

These protocols are typically used in high-performance computing (HPC) environments, where low-latency and high-bandwidth communication is critical for achieving maximum performance. They are also used in other applications that require high-speed data transfer, such as machine learning, data analytics, and high-performance storage systems. Some examples of adapter features include:

• Advanced offloading capabilities: High-performance adapters can offload CPU-intensive tasks such as packet processing, encryption/decryption, and compression/decompression, freeing up server resources for other tasks.
• Low latency: Many high-performance adapters are designed to minimize latency, which is especially important for applications that require fast response times, such as high-frequency trading, real-time analytics, and scientific computing.
• Scalability: Some adapters support features such as RDMA and SR-IOV, which allow multiple virtual machines to share a single adapter while maintaining high performance and low latency.
• Security: Many high-performance adapters have hardware-based security features such as secure boot, secure firmware updates, and hardware-based encryption/decryption, which can help protect against attacks and data breaches.
• Management and monitoring: High-performance adapters often come with tools for monitoring and managing network traffic, analyzing performance, and troubleshooting issues.

A network adapter, also known as a network interface card (NIC), is a hardware component that allows a computer or other device to connect to a network. It typically includes a connector for a cable or antenna, as well as the necessary electronics to transmit and receive data over the network. Network adapters can be internal, installed inside the computer or device, or external, connected via USB or other ports. They are used for wired or wireless connections and support different types of networks such as Ethernet, WiFi, Bluetooth, and cellular networks.

source

A host bus adapter (HBA) is a hardware component that connects a server or other device to a storage area network (SAN). It is responsible for managing the flow of data between the server and the storage devices on the SAN. HBAs typically include a connector for a Fibre Channel or iSCSI cable, as well as the necessary electronics to transmit and receive data over the SAN. They are used to connect servers to storage devices such as disk arrays, tape libraries, and other storage systems.

Common Network Protocols Used in Distributed Storage:

1. IB: used for the front-end storage network in the DPC scenario.
2. RoCE: used for the back-end storage network.
3. TCP/IP: used for service network.

Network adapters are used to connect a computer or device to a network, while host bus adapters are used to connect a computer or device to a storage area network. Network adapters are used for data transmission over networks, while host bus adapters are used for data transmission over storage area networks. There are several network adapters that are commonly used in servers, and the best option will depend on the specific needs of the server and the network it will be connecting to. Some popular options include:

1. Intel Ethernet Converged Network Adapter X520-DA2: This is a 10 Gigabit Ethernet adapter that is designed for use in data center environments. It supports both copper and fiber connections and is known for its high performance and reliability.
2. Mellanox ConnectX-4 Lx EN: This is another 10 Gigabit Ethernet adapter that is designed for use in data centers. It supports both copper and fiber connections and is known for its low latency and high throughput.
3. Broadcom BCM57416 NetXtreme-E: This is a 25 Gigabit Ethernet adapter that is designed for use in data centers. It supports both copper and fiber connections and is known for its high performance and reliability.
4. Emulex LPe1605A: This is a 16 Gbps Fibre Channel host bus adapter (HBA) that is designed for use in storage area networks (SANs). It supports both N_Port ID Virtualization (NPIV) and N_Port Virtualization (NPV) and is known for its high performance and reliability.

IBM produces a wide range of servers for various types of environments, here are a few examples of IBM servers:

1. IBM Power Systems: These servers are designed for high-performance computing and big data workloads, and are based on the Power architecture. They support IBM's AIX, IBM i, and Linux operating systems.
2. IBM System x: These servers are designed for general-purpose computing and are based on the x86 architecture. They support a wide range of operating systems, including Windows and Linux.
3. IBM System z: These servers are designed for mainframe computing and support IBM's z/OS and z/VM operating systems.
4. IBM BladeCenter: These servers are designed for blade server environments and support a wide range of operating systems, including Windows and Linux.
5. IBM Storage: These servers are designed for storage and data management workloads, and support a wide range of storage protocols and operating systems.
6. IBM Cloud servers: IBM Cloud servers are designed for cloud-based computing and are based on the x86 architecture. They support a wide range of operating systems, including Windows and Linux.

Emulex Corporation Device e228 is a network adapter produced by Emulex Corporation. It is an Ethernet controller, which means it is responsible for controlling the flow of data packets over an Ethernet network. The Emulex Corporation Device e228 is part of the Emulex OneConnect family of network adapters, which are designed for use in data center environments. These adapters are known for their high performance, low latency, and high throughput. They also provide advanced features such as virtualization support, Quality of Service (QoS) and offloads (TCP/IP, iSCSI, and FCoE) to improve network performance. It supports 10Gbps Ethernet and can be used in both copper and fiber connections. This adapter is typically used in servers and storage systems that require high-speed network connections and advanced features to support data-intensive applications. The "be2net" kernel driver is a Linux device driver that is used to control the Emulex Corporation Device e228 network adapter. A kernel driver is a type of low-level software that interfaces with the underlying hardware of a device, such as a network adapter. It provides an interface between the hardware and the operating system, allowing the operating system to communicate with and control the device. The "be2net" driver is specifically designed to work with the Emulex Corporation Device e228 network adapter, and is responsible for managing the flow of data packets between the device and the operating system. It provides the necessary functionality for the operating system to access the adapter's features and capabilities, such as configuring network settings, monitoring link status and performance, and offloading network processing tasks. The be2net driver is typically included with the Linux operating system and it's loaded automatically when the device is detected. It's also available as a separate package, that can be installed and configured manually.

The Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] is a network adapter produced by Mellanox Technologies. It is an Ethernet controller, which means it is responsible for controlling the flow of data packets over an Ethernet network. This adapter is part of the Mellanox ConnectX-5 Ex family of network adapters, which are designed for use in data center environments. These adapters are known for their high performance, low latency, and high throughput. They support 100 Gbps Ethernet, RoCE v2 and InfiniBand protocols and provide advanced features such as virtualization support, Quality of Service (QoS), and offloads to improve network performance. It's worth noting that the Mellanox ConnectX-5 Ex Virtual Function is a specific type of adapter that is designed to be used in virtualized environments. It allows multiple virtual machines to share a single physical adapter, thus providing better flexibility and resource utilization. This adapter is typically used in servers, storage systems, and other high-performance computing devices that require high-speed network connections and advanced features to support data-intensive applications such as big data analytics, machine learning, and high-performance computing.

The Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] and the Emulex Corporation Device e228 are both network adapters, but there are some key differences between them: Speed and protocol support: The Mellanox ConnectX-5 Ex supports 100 Gbps Ethernet, RoCE v2 and InfiniBand protocols, while the Emulex Device e228 supports 10 Gbps Ethernet. This means that the Mellanox adapter is capable of higher data transfer speeds and can support multiple protocols for different types of networks.

Advanced features: Both adapters offer advanced features such as virtualization support, Quality of Service (QoS), and offloads. However, the Mellanox ConnectX-5 Ex also supports features like hardware-based time stamping, hardware-based packet filtering and dynamic rate scaling.

Target market: The Mellanox ConnectX-5 Ex is designed for use in data center environments, while the Emulex Device e228 is also designed for data center environments. Mellanox ConnectX-5 Ex is more geared towards high-performance computing and big data analytics, while the Emulex Device e228 is more geared towards general data center use.

Virtualization: Mellanox ConnectX-5 Ex Virtual Function is a specific type of adapter that is designed to be used in virtualized environments, allowing multiple virtual machines to share a single physical adapter, thus providing better flexibility and resource utilization. Emulex Device e228 supports virtualization, but it does not have a specific virtual function version. In summary, the Mellanox ConnectX-5 Ex is a high-speed, high-performance network adapter that offers advanced features and support for multiple protocols, while the Emulex Device e228 is a lower-speed, general-purpose network adapter that is geared towards data center environments.

Mellanox Technologies produces networking equipment, including network adapters. Some of the Mellanox adapters that support CNA (Converged Network Adapter) are:

1. Mellanox ConnectX-5 CNA: This adapter supports both Ethernet and Fibre Channel over Ethernet (FCoE) on a single adapter, and provides high-performance, low-latency data transfer.
2. Mellanox ConnectX-6 CNA: This adapter supports 100 GbE and 200 GbE speeds and provides hardware offloads for RoCE, iWARP and TCP/IP, in addition to supporting FC and FCoE protocols.
3. Mellanox ConnectX-5 EN CNA: This adapter supports both Ethernet and InfiniBand protocols, providing high-performance, low-latency data transfer for data center and high-performance computing environments.
4. Mellanox ConnectX-6 Lx CNA: This adapter supports 25 GbE and 50 GbE speeds, and provides hardware offloads for RoCE, iWARP, and TCP/IP, in addition to supporting FC and FCoE protocols.

Slingshot is a high-performance network fabric developed by the company Cray, now owned by Hewlett Packard Enterprise. It is designed to provide low-latency and high-bandwidth communication between nodes in high-performance computing systems, such as supercomputers and data centers. It is based on a packet-switched network architecture, with each node connected to a network switch. It supports a range of network topologies, including fat-tree, hypercube, and dragonfly. The fabric is designed to be scalable, with support for thousands of nodes. It uses a range of advanced features to optimize performance, including adaptive routing, congestion control, and quality-of-service (QoS) mechanisms. It also includes support for features such as remote direct memory access (RDMA) and messaging passing interface (MPI) offload, which can further improve application performance. Overall, Slingshot is designed to provide high-performance, low-latency communication for demanding HPC workloads, making it a popular choice for large-scale scientific simulations, data analytics, and other compute-intensive applications.

RDMA Types As discussed before , there are three types of RDMA networks: Infiniband, RDMA over Converged Ethernet (RoCE), and iWARP.

source

The InfiniBand network is specially designed for RDMA to ensure reliable transmission at the hardware level. The technology is advanced, but the cost is high. RoCE and iWARP are both Ethernet-based RDMA technologies, which enable RDMA with high speed, ultra-low latency, and extremely low CPU usage to be deployed on the most widely used Ethernet.

source

The three RDMA networks have the following characteristics:

1. InfiniBand: RDMA is considered at the beginning of the design to ensure reliable transmission at the hardware level and provide higher bandwidth and lower latency. However, the cost is high because IB NICs and switches must be supported.
2. RoCE: RDMA based on Ethernet consumes less resources than iWARP and supports more features than iWARP. You can use common Ethernet switches that support RoCE NICs.
3. iWARP: TCP-based RDMA network, which uses TCP to achieve reliable transmission. Compared with RoCE, on a large-scale network, a large number of TCP connections of iWARP occupy a large number of memory resources. Therefore, iWARP has higher requirements on system specifications than RoCE. You can use common Ethernet switches that support iWARP NICs.

Comparison between RoCE and InfiniBand

Infiniband is a high-performance, low-latency interconnect technology used to connect servers, storage, and other data center equipment. It uses a switched fabric topology and supports both data and storage traffic. InfiniBand adapters are specialized network interface cards (NICs) that are designed to work with InfiniBand networks. Here are a few examples of InfiniBand adapters:

Mellanox ConnectX-4/5: These adapters support both 40 Gb/s and 100 Gb/s InfiniBand and provide high-performance, low-latency data transfer for data center and high-performance computing environments.

Mellanox ConnectX-6: This adapter supports 200 Gb/s InfiniBand, providing hardware offloads for RoCE, iWARP and TCP/IP, in addition to supporting FC and FCoE protocols.

Intel Omni-Path Architecture (OPA) 100 Series: This adapter supports 100 Gb/s InfiniBand and provides high-performance, low-latency data transfer for data center and high-performance computing environments.

Qlogic InfiniPath HTX: This adapter supports 10 Gb/s InfiniBand and provides high-performance, low-latency data transfer for data center and high-performance computing environments.

Mellanox ConnectX-4 Lx: This adapter supports 25 Gb/s and 50 Gb/s InfiniBand and provides hardware offloads for RoCE, iWARP, and TCP/IP, in addition to supporting FC and FCoE protocols.

RoCE (RDMA over Converged Ethernet) is a network protocol that allows for low-latency, high-throughput data transfer over Ethernet networks. It leverages Remote Direct Memory Access (RDMA) capabilities to accelerate communications between applications hosted on clusters of servers and storage arrays. It is based on the Remote Direct Memory Access (RDMA) protocol, which allows for direct memory access over a network without involving the CPU, resulting in low-latency and high-bandwidth data transfer. RoCE uses standard Ethernet networks and devices, so it is simpler to set up and manage than traditional RDMA over Infiniband. RoCE is designed for use in data center environments, and is particularly well-suited for use with high-performance computing and big data analytics applications, which require high-speed, low-latency data transfer. Some features of RoCE are:

• Low-latency: RoCE allows for very low-latency data transfer, which is critical for high-performance computing and big data analytics applications.
• High-throughput: RoCE allows for high-bandwidth data transfer, which is necessary for handling large amounts of data.
• RDMA support: RoCE is based on the RDMA protocol, which allows for direct memory access over a network, resulting in low-latency and high-bandwidth data transfer.
• Converged Ethernet: RoCE uses standard Ethernet networks and devices, making it simpler to set up and manage than traditional RDMA over Infiniband.
• Quality of Service (QoS) support: RoCE can provide Quality of Service (QoS) feature, which allows for guaranteed bandwidth and low-latency for critical applications.
• Virtualization support: RoCE can be used with virtualized environments, allowing multiple virtual machines to share a single physical adapter, thus providing better flexibility and resource utilization.

RoCE Overview RDMA over Converged Ethernet (RoCE) is a network protocol that leverages Remote Direct Memory Access (RDMA) capabilities to accelerate communications between applications hosted on clusters of servers and storage arrays. RoCE incorporates the IBTA RDMA semantics to allow devices to perform direct memory-to-memory transfers at the application level without involving the host CPU. Both the transport processing and the memory translation and placement are performed by the hardware which enables lower latency, higher throughput, and better performance compared to software-based protocols.

Infiniband RDMA to RoCE :

Both InfiniBand RDMA and RoCE can implement remote memory access network protocols. The two currently have their own advantages and disadvantages on the market, and both are used in HPC cluster architecture or large-scale data centers. Comparing the two, InfiniBand has better performance. But InfiniBand is a dedicated network technology. It cannot inherit the user's accumulation and platform of operation on the IP network, which causes the high cost in operation and maintenance. Therefore, carrying RDMA based on the traditional Ethernet network is also inevitable for the large-scale application of RDMA. To guarantee RDMA performance and network layer communication, many network switches use RoCEv2 to carry high-performance distributed applications.

CNA (Converged Network Adapter) is a type of network adapter that supports multiple protocols, such as Ethernet and Fibre Channel over Ethernet (FCoE), on a single adapter. A CNA typically includes both a NIC and a Host Bus Adapter (HBA) to support both data and storage traffic. When using a CNA with SRIOV (Single Root I/O Virtualization) and ROCE (RDMA over Converged Ethernet), multiple virtual functions (VFs) can be created on the CNA, each with its own MAC address, VLAN ID, and other network attributes. Each VF can be assigned to a different virtual machine (VM) or a container, and each VM or container can have its own network configuration and parameters. Each VF can be configured to support ROCE, allowing for low-latency, high-throughput data transfer over Ethernet networks. This can be particularly useful in high-performance computing and big data analytics environments, where low-latency and high-bandwidth data transfer is critical.

converged network adapter (CNA)

SRIOV with ROCE on a CNA can provide the following benefits: Improved resource utilization: By allowing multiple VMs or containers to share a single physical adapter, SRIOV with ROCE on a CNA can improve resource utilization and reduce costs.

Improved network performance: ROCE allows for low-latency, high-throughput data transfer over Ethernet networks, which can improve network performance in high-performance computing and big data analytics environments.

Fine-grained control of network resources: SRIOV with ROCE on a CNA allows for fine-grained control of network resources, allowing each VM or container to have its own network configuration.

I/O Virtualization Overview: CNA, SR-IOV

Differences between RoCE, Infiniband RDMA, and TCP/IP.

The fastest network adapter available today depends on the specific application and the network infrastructure. Generally, there are different types of network adapters that support different speeds and protocols, and each one is suitable for different use cases.

For example, for data center environments, 100 GbE (gigabit ethernet) adapters are currently considered as the fastest option, providing high-bandwidth and low-latency data transfer. These adapters use the latest technologies such as SFP28 and QSFP28 connectors and support both copper and fiber cabling. Mellanox ConnectX-6, Intel Omnipath and Marvell FastLinQ are some examples of 100 GbE adapters.

For High-Performance Computing (HPC) and Artificial Intelligence (AI) applications, Infiniband adapters are considered as the fastest option providing low-latency and high-bandwidth data transfer. Mellanox ConnectX-6 HDR and Intel OPA 100 series are examples of 200 Gb/s Infiniband adapters. For storage, Fibre Channel (FC) and Fibre Channel over Ethernet (FCoE) adapters are considered as the fastest option providing low-latency and high-bandwidth data transfer. Mellanox ConnectX-6, Emulex Gen 6 Fibre Channel and Qlogic Gen 6 Fibre Channel are examples of these adapters.

Supercomputer systems, like the Summit and Sierra, developed by Oak Ridge National Laboratory and Lawrence Livermore National Laboratory, respectively, use a high-performance interconnect technology called Infiniband for their internal communication. Mellanox Technologies is the company that provides the Infiniband adapters and host bus adapters (HBAs) for these supercomputers.

Summit and Sierra use Mellanox's Connect-IB adapter which supports 100 Gb/s InfiniBand and provides hardware offloads for RoCE, iWARP and TCP/IP. The Connect-IB adapters are designed to handle the high-bandwidth and low-latency requirements of large-scale supercomputing applications. The Host Bus Adapter (HBA) Mellanox ConnectX-4 Lx is used for these systems. ConnectX-4 Lx is a single-port 100 Gb/s InfiniBand adapter that supports both 25 Gb/s and 50 Gb/s speeds. The adapters provide hardware offloads for RoCE, iWARP, and TCP/IP, in addition to supporting FC and FCoE protocols.

Frontier is a planned supercomputer that is being developed by Oak Ridge National Laboratory and Cray Inc. It is the world's fastest supercomputer in 2021. According to the information available, Frontier uses high-performance interconnect technology called Slingshot, developed by Cray, for its internal communication. Slingshot is a next-generation interconnect technology that promises to provide low-latency, high-bandwidth, and high-message-rate data transfer.

In terms of network adapters and host bus adapters (HBAs), the information available is not specific, but it's known that Cray has collaborated with Mellanox Technologies to provide the network interconnect technology for Frontier. This suggests that Mellanox's InfiniBand and/or Slingshot adapters may be used in Frontier.

A host bus adapter (HBA) is a specialized type of network interface card (NIC) that connects a host computer to a storage area network (SAN). HBAs provide a bridge between the host computer and the storage devices, allowing the host to access and manage the storage devices as if they were locally attached.

Here are a few key things to know to get familiar with storage HBAs:

• Protocols: HBAs support different storage protocols such as Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), and iSCSI. FC and FCoE are commonly used in enterprise environments, while iSCSI is more commonly used in smaller, SMB environments. Speed: HBAs are available in different speeds, such as 8 Gb/s, 16 Gb/s, and 32 Gb/s. Higher speeds provide faster data transfer and improved performance.
• Multi-Path Support: HBAs often support multi-path I/O, which allows multiple paths to the storage devices to be used for failover and load balancing. Compatibility: HBAs are designed to work with specific operating systems, so it's important to check the compatibility of the HBA with the operating system you are using.
• Management and Monitoring: Many HBAs include management and monitoring software that allows administrators to view and configure the HBA's settings, such as Fibre Channel zoning, and to monitor the performance of the HBA and the storage devices it is connected to. Driver and Firmware: HBA's require driver and firmware to work properly, so it's important to ensure that the HBA has the latest driver and firmware updates installed.
• Vendor Support: It's important to consider the vendor support of the HBA, as well as the warranty and technical support options available, as these can be critical factors when choosing an HBA. Architecture: Some HBAs are based on ASIC (Application-Specific Integrated Circuit) while others on FPGA (Field-Programmable Gate Array) architecture, both have their own pros and cons.

Power10 is the latest generation of IBM's Power Architecture designed for high-performance computing and big data workloads, and is intended to deliver significant performance and efficiency improvements over its predecessor, Power9. Some of the key features of the Power10 architecture include:

• Higher core count: Power10 processors have a higher core count than Power9 processors, which allows for more parallel processing and improved performance.
• Improved memory bandwidth: Power10 processors have more memory bandwidth than Power9 processors, which allows for faster data transfer between the processor and memory. Enhanced security features: Power10 processors include enhanced security features, such as hardware-enforced memory encryption and real-time threat detection, to protect against cyber-attacks.
• Improved energy efficiency: Power10 processors are designed to be more energy efficient than Power9 processors, which can help to reduce power consumption and cooling costs. Optimized for AI workloads: Power10 processors are optimized for AI workloads and have better support for deep learning and other AI-related tasks.
• More flexible and open: Power10 architecture is more flexible and open. It supports more operating systems, and it has more open interfaces and more standard protocols to connect to other devices. Example: IBM Power Systems AC922.

AI workloads refer to tasks that involve the use of artificial intelligence and machine learning algorithms, such as:

• Natural Language Processing (NLP): This includes tasks such as speech recognition, text-to-speech, and machine translation.
• Computer Vision: This includes tasks such as image recognition, object detection, and facial recognition. Predictive analytics: This includes tasks such as forecasting, anomaly detection, and fraud detection.
• Robotics: This includes tasks such as navigation, object manipulation, and decision making. Recommender Systems: This includes tasks such as personalized product recommendations, content recommendations, and sentiment analysis.
• Generative Models: These include tasks such as image and video generation, text generation and music generation. Reinforcement learning: These include tasks such as game playing, decision making and control systems.
• Deep Learning: These include tasks such as Image and speech recognition, natural language processing and predictive analytics.

These are just a few examples of AI workloads, there are many more possible applications of AI in various industries such as healthcare, finance, transportation, and manufacturing. As AI technology continues to advance, new possibilities for AI workloads will continue to emerge.

OpenMPI and UCX are both middleware for high-performance computing, but they are not directly connected to adapter design. However, they can utilize hardware-specific features and optimizations of network adapters to improve performance.

MPI (Message Passing Interface) and AI (Artificial Intelligence) are interrelated because MPI can be used to distribute the computational workload of AI applications across multiple nodes in a distributed computing environment. Many AI algorithms, such as deep learning, machine learning, and neural networks, require a significant amount of computational resources, memory, and data storage. These algorithms can be parallelized and run in a distributed environment using MPI, which allows them to take advantage of the computing power of multiple nodes. MPI can be used to distribute the data and the workload of AI applications across multiple nodes, enabling parallel processing and reducing the time required to complete the computation. This can significantly improve the performance of AI applications and enable researchers to train and optimize more complex models. Moreover, MPI can be integrated with other libraries, such as OpenMP, CUDA, and UCX, to further improve the performance of AI applications. For example, CUDA is a parallel computing platform that enables programmers to use GPUs (Graphics Processing Units) for general-purpose processing, and MPI can be used to distribute the workload across multiple GPUs and nodes. In summary, MPI provides a scalable and efficient way to distribute the computational workload of AI applications across multiple nodes, enabling researchers and developers to build and run more complex models and achieve better performance. The choice of MPI communication method that is best suited for AI workloads depends on the specific characteristics of the workload and the system architecture. However, some general guidelines can help in selecting the appropriate MPI communication method for AI workloads. For AI workloads that involve large amounts of data, non-blocking point-to-point communication and collective communication methods are generally preferred. Non-blocking point-to-point communication methods, such as MPI_Isend and MPI_Irecv, allow the application to continue processing while the communication is in progress, which can help reduce the overall communication time. Collective communication methods, such as MPI_Allreduce and MPI_Allgather, can also be highly effective in AI workloads, as they enable efficient data sharing and synchronization among multiple nodes. These methods can be used to distribute the workload of an AI application across multiple nodes, enabling parallel processing and reducing the time required to complete the computation. Additionally, the choice of MPI communication method may also depend on the underlying system architecture. For example, on a system with a high-speed interconnect, such as InfiniBand, the use of MPI communication methods that take advantage of the RDMA (Remote Direct Memory Access) capabilities of the interconnect, such as UCX, can provide significant performance benefits. The best MPI communication method for AI workloads depends on the specific characteristics of the workload and the system architecture. However, non-blocking point-to-point communication and collective communication methods are generally preferred for AI workloads that involve large amounts of data, and the use of RDMA-enabled MPI communication methods can provide significant performance benefits on high-speed interconnects.

The mapping of adapters in supercomputers and network adapters is an important aspect of designing and building a supercomputer. In general, supercomputers use high-performance network adapters that can handle large amounts of data at high speeds, with low latency and high bandwidth. The choice of network adapter depends on the specific requirements of the supercomputer, such as the type and size of data being processed, the number of nodes in the system, and the desired performance characteristics. Some of the common network adapters used in supercomputers include InfiniBand adapters, Ethernet adapters, and Omni-Path adapters. These adapters are typically integrated with the server hardware, either as separate network interface cards (NICs) or as part of the motherboard design. These adapters provide low-latency, high-bandwidth interconnects between nodes in a cluster, enabling parallel computing and large-scale data processing. In addition to high-performance interconnects, HPC also relies on specialized hardware accelerators like GPUs, FPGAs, and ASICs to offload compute-intensive tasks from the CPU and improve overall system performance. These accelerators are often used in combination with high-performance network adapters to enable faster data transfer and processing in HPC environments.

IBM offloaded Watson Health assets to investment firm Francisco Partners

IBM Watson is a question-answering computer system capable of answering questions posed in natural language, developed in IBM's DeepQA project by a research team led by principal investigator David Ferrucci. Watson was named after IBM's founder and first CEO, industrialist Thomas J. Watson. IBM’s then- CEO Ginni Rometty called the project a “moon shot,” but her replacement was less enthused about the business. The computer system was initially developed to answer questions on the quiz show Jeopardy!.

IBM launched Watson Health in early 2015 and made a series of acquisitions that cost $4 billion. They included Merge Healthcare, Truven Health Analytics, Phytel, and Explorys. IBM sold Watson Health for $1B, which is 25% of what it paid to acquire four strong businesses. The assets involved include Health Insights, MarketScan, Clinical Development, Social Program Management, Micromedex, and imaging software products. IBM offloaded Watson Health this year because it doesn't have the requisite vertical expertise in the healthcare sector.

Talking at stock market analyst Bernstein's 38th Annual Strategic Decisions Conference, the big boss was asked to outline the context for selling the healthcare data and analytics assets of the business to private equity provider Francisco Partners for $1 billion in January.

 The Watson brand will be  carrier for AI.

It's a question of verticals versus horizontals. IBM believes that they are best positioned to take these technologies.They will always have an industry lens but through their consulting team. They want to work on technologies that are horizontal across all industries."

Verticals should belong to people who really have all of the domain expertise, they have credibility in that vertical. And healthcare companies, people in medical devices, they will have the credibility to carry out how AI is applied to health in depth i.e AI as applied to healthcare, to financial services, to compliance, in that case, regulatory compliance, is going to be a massive market.

To succeed in health, they need doctors and nurse practitioners to speak to the buyers of Watson Health. That's not the IBM go-to-market field force, so there's a misalignment. Ditto in Promontory, that needed  ex-regulators and accountants to go talk to people worrying about financial compliance. So, that's a little bit different than IBM. IBM still sells Watson solutions in financial services, advertising, business automation, and video streaming and hosting. As for AI in the enterprise where  inflation, labor costs and the world undergoing a "demographic shift" means that "there are fewer people with the skills" and so AI and automation will be "applied to more and more domains." Trend is going to reverse in the next few decades."

IBM’s Watson Health is being sold for parts. The technology giant has agreed to sell the division’s data and analytics assets to private equity firm Francisco Partners. Terms weren’t disclosed, although Bloomberg values the deal at more than $1 billion. Launched in 2015, Watson Health’s goal was to revolutionize medicine through AI. After years of pricey expansion — it spent more than $4 billion on acquisitions, per Axios — and reports of ineffectiveness, the unit scaled back its ambitions.

Once viewed as a flagship of AI in medcine and life science, IBM Watson Health couldn't live up to its ambitious promises to transform everything from drug discovery to cancer care. It would be interesting to see how the new firm who bought this giant from IBM will transform its data and analytic assets and realize their full potential.

BIG MPI

In order to describe a structured region of memory, the routines in the MPI standard use a (count, datatype) pair. The C specification for this convention uses an int type for the count. Since C int types are nearly always 32 bits large and signed, counting more than 2 power 31 elements poses a challenge. Instead of changing the existing MPI routines, and all consumers of those routines, the MPI Forum asserts that users can build up large datatypes from smaller types. To evaluate this hypothesis and to provide a user-friendly solution to the large-count issue, we have developed BigMPI, a library on top of MPI that maps large count MPI-like functions to MPI-3 standard features. BigMPI demonstrates a way to perform such a construction, reveals shortcomings of the MPI standard, and uncovers bugs in MPI implementations

References:

https://www.mcs.anl.gov/papers/P5210-1014.pdf

https://github.com/jeffhammond/BigMPI

MPI [ Message Passing Interfaces] - behind the scenes

Parallel computing is accomplished by splitting up large and complex tasks across multiple processors. In order to organize and orchestrate parallel processing, our program must consider automatically decomposing the problem at hand and allowing the processors to communicate with each other when necessary while performing their work. This introduces a new overhead, the synchronization and the communication itself.

Computing parallelism can be roughly classified as Distributed Memory (DM) or Shared Memory(SM) class. In Distributed Memory (DM), each processor has its own memory which are connected through a network that can exchange data, thus, limiting the DM performance and scalability. In Shared

Memory (SM), each processor can access all of the memory, resulting in automatic distribution of procedurally iterations over several processors - autoparallelization, explicit distribution of work over the processors by compiler directives or function calls to threading libraries. If this overhead is not accounted. It can create several issues like bottlenecks in the parallel computer design and load imbalances.

MPI is an API for message passing between entities with separated memory spaces - processes. The standard doesn't care where those processes run - it could be on networked computers (clusters), it could be on a big shared memory machine or it could be any other architecture that provide the same semantics (e.g. IBM Blue Gene)

OpenMPI is a widely used message passing interface (MPI) library for parallel computing. It provides an abstraction layer that allows application developers to write parallel code without worrying about the underlying hardware details. However, OpenMPI also provides support for hardware-specific optimizations, including those for network adapters. For example, it supports the use of high-speed interconnects such as InfiniBand and RoCE, and it can take advantage of hardware offload capabilities such as Remote Direct Memory Access (RDMA).

UCX (Unified Communication X) is another middleware library for communication in distributed systems. It is designed to be highly scalable and to support a wide range of hardware platforms, including network adapters. UCX provides a portable API that allows applications to take advantage of hardware-specific features of network adapters, such as RDMA and network offloading. UCX can also integrate with other system-level libraries such as OpenMPI and hwloc to optimize performance on specific hardware configurations.

Hwloc (Hardware Locality) is a library for topology discovery and affinity management in parallel computing. It provides a portable API for discovering the hierarchical structure of the underlying hardware, including network adapters, and it allows applications to optimize performance by binding threads and processes to specific hardware resources. Hwloc can be used in conjunction with OpenMPI and UCX to optimize communication and data movement on high-performance computing systems.

TCP/IP is a family of networking protocols. IP is the lower-level protocol that's responsible for getting packets of data from place to place across the Internet. TCP sits on top of IP and adds virtual circuit/connection semantics. With IP alone you can only send and receive independent packets of data that are not organized into a stream or connection. It's possible to use virtually any physical transport mechanism to move IP packets around. For local networks it's usually Ethernet, but you can use anything. There's even an RFC specifying a way to send IP packets by carrier pigeon.

Sockets is a semi-standard API for accessing the networking features of the operating system. Your program can call various functions named socket, bind, listen, connect, etc., to send/receive data, connect to other computers, and listen for connections from other computers. You can theoretically use any family of networking protocols through the sockets API--the protocol family is a parameter that you pass in--but these days you pretty much always specify TCP/IP. (The other option that's in common use is local Unix sockets.)

When you are interested  to  write a parallel programming application, you should probably not be looking at TCP/IP or sockets as those things are going to be much lower level than you want. You'll probably want to look at something like MPI or any of the PGAS languages like UPC, Co-array Fortran, Global Arrays, Chapel, etc. They're going to be far easier to use than essentially writing your own networking layer.

When you use one of these higher level libraries,  you get lots of nice abstractions like collective operations, remote memory access, and other features that make it easier to just write your parallel code instead of dealing with all of the OS stuff underneath. It also makes your code portable between different machines/architectures.

MPI is free to use any available communication path(s) for MPI messages in the new communicator; the socket is only used for the initial handshaking. 

A common problem is the one of two processes each opening connections to each other. The socket code assume that the sockets are bidirectional, thus only one socket is needed by each pair of connected processes, not one socket for each member of the pair. Then it should refactor the states and state machine into a clear set of VC connection states and connection states.

There are three related objects used during a connection event. They are the connection itself (a structure specific to the communication method, sockets in the case of this note), the virtual connection, and the process group to which the virtual connection belongs

If a socket connection between two processes is established, there are always two sides: The connecting side and the accepting side. The connecting side sends an active message to the accepting side. This sides first accepts the connection. However, if both processes try to connect to each other (head-to-head situation), the processes n have both, a connecting and an accepting connection. In this situation, one of the connections is refused/discarded - while the other connection is established. This is decided on the accepting side.

State machines for establishing a connection:

Connect side :

The connecting side tries to establish a connection by sending an active message to the remote side. If the connection is accepted, the pg_id is send to the remote side. Now, the process waits, until the connection is finally accepted or refused. For this decision, the remote side requires the gp_id . Based on the answer from the remote side (ack = yes or ack = no) the connection is connected or closed.

Accept side:

The accept side receives a connection request on the listening socket. In the first instance, it accepts the connection an allocates the required structures (conn, socket). Then, the connection waits for the pg_id of the remote side to assign the socket-connection to a VC. The decision, if a connection is accepted or refused, is based on the following steps:

1. The VC has to aktive connection (vc->conn == NULL) : The new connection is accepted
2. The VC has an aktive connection
3. If my_pg_id < remote_pg_id: accept and discard other connection
4. If my_pg_id > remote_pg_id: refuse  

The answer is send to the remote note.

Data Types Required by the MPI Standard:

MPI point-to-point communication sends messages between two different MPI processes. One process performs a send operation while the other performs a matching read

MPI collectives: MPI provides a set of routines for communication patterns that involve all the processes of a certain communicator, so-called collectives. The two main advantages of using collectives are:

1) Less programming effort. 

2) Performance optimization, as the implementations are usually efficient, especially if optimized for specific architectures

For collective communication, significant gains in performance can be achieved by implementing topology- and performance-aware collectives.

Three common blocking collectives are Barrier(), Bcast() and Reduce().

1. Allreduce(). Combination of reduction and a broadcast so that the output is available for all processes.
2. Scatter(). Split a block of data available in a root process and send different fragments to each process.
3. Gather(). Send data from different processes and aggregate it in a root process.
4. Allgather(). Similar to Gather() but the output is aggregated in buffers of all the processes.
5. Alltoall(). All processes scatter data to all processes.

Reference:

https://www.sciencedirect.com/topics/computer-science/point-to-point-communication

https://wiki.mpich.org/mpich/index.php/Establishing_Socket_Connections

https://aist-itri.github.io/gridmpi/publications/cluster04-slide.pdf