tag:blogger.com,1999:blog-83064005249456207962024-03-28T10:39:13.851+05:30 LINUX & HPC : Advanced Large Scale Computing at a Glance !Newbie's Guide for Spectrum LSF, Message Passing Interface(MPI), Kubernetes, Big Data applications ,Docker, Jenkins, Spark, Hadoop, Quantum Computing, Linux Operating system and features, git, DevOps...........! Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.comBlogger59125tag:blogger.com,1999:blog-8306400524945620796.post-84978419745226959772023-09-08T10:38:00.002+05:302023-09-08T11:41:07.721+05:30Multipath setup in Linux<div><div><div>A multipath setup in Linux is a configuration that allows multiple physical paths (usually represented by multiple physical storage devices or network connections) to be used to access a single logical device or storage target. The primary goal of multipath is to enhance redundancy and fault tolerance while providing load balancing and improved performance. Multipath is commonly used in storage area networks (SANs) and environments where high availability and reliability are critical.</div><div><br /></div><div>Here are the key components and concepts of a multipath setup in Linux:</div><div><br /></div><div>Multipath Devices (Multipath): In a multipath setup, there is a logical device known as a multipath device (often referred to as a multipath or mpath). This logical device represents a single storage target, such as a disk or LUN (Logical Unit Number), even though it is accessible through multiple physical paths.</div><div><br /></div><div>Physical Paths: Physical paths are the actual connections or channels through which the storage target is accessible. These paths can be physical SCSI buses, Fibre Channel links, iSCSI connections, or any other transport mechanism. Each path is associated with a unique identifier, typically called a World Wide Name (WWN), device name, or other similar identifiers.</div><div><br /></div><div>Path Management: The multipath software in Linux (such as multipathd and multipath-tools) manages the physical paths and ensures that they are utilized effectively. It monitors the status of the paths and makes decisions about which path to use for I/O operations. It can also detect and respond to path failures or changes in path availability.</div><div><br /></div><div>Load Balancing: Multipath configurations often include load balancing mechanisms that distribute I/O requests across the available paths. This helps improve performance by distributing the workload and preventing one path from becoming a bottleneck.</div><div><br /></div><div>Redundancy and Failover: Multipath setups provide redundancy and failover capabilities. If one path fails due to hardware or network issues, the system can automatically switch to an alternate path without interrupting I/O operations. This enhances system reliability and availability.</div><div><br /></div><div>Device Mapper (DM-Multipath): In Linux, the Device Mapper subsystem is commonly used to manage multipath devices. DM-Multipath is a kernel component that works with the multipath software to create and manage multipath devices. It presents a single device to the operating system, which is actually a combination of the multiple physical paths.</div><div><br /></div><div>Configuration Files: To set up multipath in Linux, administrators configure multipath settings using configuration files. The main configuration file is typically located at /etc/multipath.conf (or a similar location) and defines the behavior of the multipath devices.</div><div><br /></div><div>Multipath Tools: The multipath tools package (multipath-tools or similar) includes utilities such as multipath and multipathd that are used to manage and configure multipath devices. These tools help monitor path status, configure load balancing policies, and perform other administrative tasks related to multipathing.</div><div>-----------------------------------</div></div><div><br /></div><div>For Example : This system is using multipath and LVM for storage management to provide redundancy and flexibility in managing storage devices. Both sda and sdb are part of the multipath configuration, and their partitions are managed using LVM. This setup is commonly used in enterprise environments for high availability and fault tolerance</div><div><br /></div><div><br /></div><div>There are three partitions on the multipath device mpatha (which represents both sda and sdb due to multipathing). </div></div><div><br /></div><div><br /></div><div>The lsblk command is used to list block devices on this system, displaying information about disks, partitions, and their relationships. Let's break down the lsblk output:</div><div>----------------------------------------------------------------------------------------</div><div><br /></div><div>#lsblk</div><div>NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT</div><div>sda 8:0 0 80G 0 disk</div><div>└─mpatha 253:0 0 80G 0 mpath</div><div> ├─mpatha1 253:1 0 4M 0 part</div><div> ├─mpatha2 253:2 0 1G 0 part /boot</div><div> └─mpatha3 253:3 0 79G 0 part</div><div> ├─rhel_myhost-root 253:4 0 47.7G 0 lvm /</div><div> ├─rhel_myhost-swap 253:5 0 8G 0 lvm [SWAP]</div><div> └─rhel_myhost-home 253:6 0 23.3G 0 lvm /home</div><div>sdb 8:16 0 80G 0 disk</div><div>└─mpatha 253:0 0 80G 0 mpath</div><div> ├─mpatha1 253:1 0 4M 0 part</div><div> ├─mpatha2 253:2 0 1G 0 part /boot</div><div> └─mpatha3 253:3 0 79G 0 part</div><div> ├─rhel_myhost-root 253:4 0 47.7G 0 lvm /</div><div> ├─rhel_myhost-swap 253:5 0 8G 0 lvm [SWAP]</div><div> └─rhel_myhost-home 253:6 0 23.3G 0 lvm /home</div><div>#</div><div><br /></div><div>Here's a breakdown of the information in each column:</div><div><br /></div><div>NAME: This column shows the name of the block device.</div><div>MAJ:MIN: Major and minor device numbers that uniquely identify the device to the operating system.</div><div>RM: This indicates whether the device is removable (1 for yes, 0 for no).</div><div>SIZE: The size of the device, often in gigabytes (G) or megabytes (M).</div><div>RO: This indicates whether the device is read-only (1 for yes, 0 for no).</div><div>TYPE: The type of device, which can be "disk" for physical disks or "part" for partitions. In this case, you also see "mpath" and "lvm," which are related to storage management.</div><div>MOUNTPOINT: The mount point where the device is currently mounted. If it's not mounted, this field will be empty.</div><div><br /></div><div>Now, let's interpret the information based on the provided output:</div><div><br /></div><div>There are two disk devices: sda and sdb.</div><div>Both sda and sdb are part of a multipath configuration, as indicated by the "mpath" type.</div><div>Each disk (sda and sdb) has three partitions (mpatha1, mpatha2, and mpatha3), and each of these partitions is used in an LVM (Logical Volume Management) setup.</div><div>The /boot partition (mpatha2) is mounted on both sda and sdb, and it contains the boot files.</div><div>The root (/) partition (rhel_myhost-root) is mounted on both sda and sdb, and it is the root filesystem.</div><div>The swap partition (rhel_myhost-swap) is also mounted on both sda and sdb and is used for swap space.</div><div>The /home partition (rhel_myhost-home) is mounted on both sda and sdb and is used for user home directories.</div><div><br /></div><div>Here's what each of these partitions is typically used for:</div><div><br /></div><div>mpatha1: This partition appears to be very small (only 4MB), and it is often used for storing bootloader-related files. Specifically, it might contain the GRUB bootloader's core files or other boot-related data. It's a common practice to allocate a small partition for bootloader files to ensure that they are easily accessible and less likely to be affected by changes or issues in the rest of the filesystem. A small partition like this is often sufficient for storing the essential boot files.</div><div><br /></div><div>mpatha2: This partition is mounted as /boot, and it contains the kernel and initial ramdisk files needed for booting the system. /boot typically holds the Linux kernel, GRUB configuration files, and other boot-related data. Having a separate /boot partition is a common practice, especially in systems that use LVM or other complex storage configurations. It ensures that essential boot files are easily accessible and are less prone to issues that might affect other partitions.</div><div><br /></div><div>mpatha3: This partition appears to be the largest and is not directly mounted as part of the root filesystem (/). Instead, it seems to be part of an LVM (Logical Volume Management) setup. It is divided into multiple logical volumes (rhel_myhost-root, rhel_myhost-swap, and rhel_myhost-home) that are used for various purposes:</div><div><br /></div><div>rhel_myhost-root: This is the root filesystem (/) where most of the operating system and software are installed.</div><div><br /></div><div>rhel_myhost-swap: This logical volume is used as swap space, which is used for virtual memory and can help improve system performance.</div><div><br /></div><div>rhel_myhost-home: This logical volume is typically used for user home directories. User data and files are stored in the /home directory, which is often mounted on a separate filesystem to isolate user data from the root filesystem.</div><div><br /></div><div>So, mpatha1 is likely used for bootloader files, mpatha2 is the /boot partition containing boot-related files, and mpatha3 represents an LVM setup with separate logical volumes for the root filesystem, swap space, and user home directories. This kind of partitioning and storage management allows for flexibility, scalability, and better system maintenance</div><div><br /></div><div><div>You can map the UUID specified in the /etc/fstab file to the corresponding device name, such as /dev/sda1, /dev/sdb1, or /dev/mpatha1. The UUID (Universally Unique Identifier) is a unique identifier assigned to each filesystem or partition and is a more reliable way to identify devices than device names, which can change if hardware configurations are altered.</div><div><br /></div><div>To map a UUID to the corresponding device name, you can use the blkid command. </div></div><div><br /></div><div><br /></div><div><div>[root@myhost ~]# blkid</div><div>/dev/mapper/rhel_myhost-root: UUID="XXXXXXXXXXXXXXX" BLOCK_SIZE="512" TYPE="xfs"</div><div>/dev/mapper/mpatha3: UUID="XXXXXXXXX" TYPE="LVM2_member" PARTUUID="XXXXXXXX"</div><div>/dev/sda: PTUUID="XXXXXXXX" PTTYPE="dos"</div><div>/dev/mapper/mpatha: PTUUID="abc" PTTYPE="dos"</div><div>/dev/sdb: PTUUID="XXXXXXX" PTTYPE="dos"</div><div>/dev/mapper/mpatha1: PARTUUID="abc-01"</div><div>/dev/mapper/mpatha2: UUID="XXXXXXXXXXXXXXXXXXXX" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="XXXXX"</div><div>/dev/mapper/rhel_myhost-swap: UUID="abc123c" TYPE="swap"</div><div>/dev/mapper/rhel_myhost-home: UUID="xyz123" BLOCK_SIZE="512" TYPE="xfs"</div><div>[root@myhost ~]#</div><div>-----------------</div><div><br /></div><div>[root@myhost ~]# cat /boot/grub2/device.map</div><div># this device map was generated by anaconda</div><div>(hd0) /dev/mapper/mpatha</div><div>[root@myhost ~]#</div><div><br /></div><div><br /></div><div>--------------</div><div><br /></div><div><br /></div><div>[root@myhost ~]# cat /etc/fstab</div><div><br /></div><div>#</div><div># /etc/fstab</div><div>/dev/mapper/rhel_myhost-root / xfs defaults 0 0</div><div>UUID=XXXXXXXXXXXXXXXXXXXXXXX /boot xfs defaults 0 0</div><div>/dev/mapper/rhel_myhost-home /home xfs defaults 0 0</div><div>/dev/mapper/rhel_myhost-swap none swap defaults 0 0</div><div>[root@myhost ~]#</div></div><div><br /></div><div>A multipath setup in Linux provides redundancy, load balancing, and fault tolerance for storage devices, ensuring that data remains accessible even if one path or connection fails. This technology is crucial in enterprise environments where continuous access to data is essential for operations.</div><div><br /></div><div>----------------------------</div><div><br /></div><div><div>The grub2-install command is a utility used in Linux to install the GRUB (Grand Unified Bootloader) bootloader onto a device, typically a hard disk or a partition.</div><div><br /></div><div>The primary purpose of the grub2-install command is to install the GRUB bootloader on a specific device. You specify the target device as an argument to the command. </div><div><br /></div><div>For example : grub2-install /dev/sda</div><div><br /></div><div>In this example, the GRUB bootloader is installed on the MBR (Master Boot Record) of /dev/sda, which is typically the primary boot device.</div><div><br /></div><div><br /></div><div>Boot Device Configuration:</div><div>When GRUB is installed on a device, it configures the bootloader to locate and load the kernel and initial ramdisk (initrd) from the designated boot device or partition. It also stores configuration information, such as the location of the kernel and the root filesystem.</div><div><br /></div><div>Device Map Configuration:</div><div>GRUB maintains a device map that associates BIOS drive numbers (e.g., (hd0), (hd1)) with actual device names (e.g., /dev/sda, /dev/sdb). The grub2-install command updates or creates this device map, ensuring that GRUB can correctly identify the boot device.</div><div><br /></div><div>Bootloader Configuration File:</div><div>GRUB bootloader configurations are specified in the /boot/grub2/grub.cfg (or similar) file. This configuration file is automatically generated by GRUB utilities and scripts based on the system's configuration, such as the kernel and initrd locations, boot options, and menu entries.</div><div><br /></div><div>Boot Menu:</div><div>GRUB provides a boot menu during system startup, allowing users to select from available kernels and boot options. The grub2-install command ensures that the necessary components for this boot menu are installed and configured correctly.</div><div><br /></div><div>Updating GRUB Configuration:</div><div>In addition to installing GRUB, the grub2-install command also updates the bootloader's configuration to reflect changes in the system's disk layout or partitioning scheme. This includes updating device names and paths if necessary.</div><div><br /></div><div>EFI and UEFI Support:</div><div>The behavior of grub2-install can differ depending on whether the system uses BIOS or UEFI (Unified Extensible Firmware Interface) for booting. For UEFI systems, the grub2-install command installs the UEFI version of GRUB and configures it accordingly.</div><div><br /></div><div>Additional Options:</div><div>The grub2-install command supports various options to specify installation details, such as the target architecture, firmware type (BIOS or UEFI), and more. You can use the --target, --boot-directory, and other options to customize the installation.</div></div><div><br /></div><div>----------------------------------------------------------</div><div><br /></div><div><div>The PReP boot partition is a specialized partition used on PowerPC systems that follow the PReP boot standard to store firmware-specific bootloader and boot-related files. On the other hand, the /boot partition is a common convention on Linux systems, including PowerPC systems, to store kernel, initramfs, and bootloader configuration files, but it is not tied to any specific firmware standard and is used across various hardware architectures.</div><div><br /></div><div>PReP Boot Partition:</div><div>The PReP boot partition is a specific partition type used in the context of the PReP boot standard, which is a firmware standard for booting PowerPC-based systems.</div><div>Its primary purpose is to store the bootloader and boot-related information required to initiate the boot process on PowerPC systems adhering to the PReP standard.</div><div>It typically contains essential firmware boot files, such as Open Firmware or IEEE 1275-compliant firmware, which are necessary to start the system.</div><div><br /></div><div>/boot Partition:</div><div>The /boot partition is a common convention used on various Linux distributions, including those running on PowerPC systems.</div><div>Its purpose is to store the kernel, initramfs (initial RAM disk), bootloader configuration files, and other files required for the early stages of the boot process.</div><div>The /boot partition is part of the Linux filesystem structure and is used by the Linux bootloader (e.g., GRUB) to locate and load the kernel and initramfs during the boot process.</div><div>Firmware Dependency:</div><div><br /></div><div>PReP Boot Partition:</div><div>The PReP boot partition's usage is closely tied to the firmware standard it follows, such as Open Firmware or IEEE 1275-compliant firmware. The firmware is responsible for loading the bootloader from this partition.</div><div>It may also contain firmware-specific files and configurations.</div><div><br /></div><div>/boot Partition:</div><div>The /boot partition is not firmware-dependent and is part of the Linux filesystem. It is managed by the Linux bootloader (e.g., GRUB) and the operating system itself.</div><div>The bootloader reads the kernel and initramfs from the /boot partition during the boot process, and this partition is independent of the system's firmware.</div><div>Common Usage:</div><div><br /></div><div>PReP Boot Partition:</div><div>Commonly used on older PowerPC-based systems that adhere to the PReP standard.</div><div>It's specific to the boot process defined by the firmware standard used on these systems</div><div><br /></div><div>/boot Partition:</div><div>Widely used on various Linux distributions, including those on PowerPC systems.</div><div>It's part of the standard Linux filesystem structure and is used on many different hardware platforms.</div></div><div><br /></div><div><br /></div>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-6925086525774503612023-09-04T17:51:00.007+05:302023-09-04T17:54:06.787+05:30Openstack Framework and components <p><span style="text-align: justify;">OpenStack is an open-source cloud computing platform that provides a set of software tools and components for building and managing public and private clouds. It enables organizations to create and manage cloud infrastructure services, including compute, storage, networking, and more. OpenStack is designed to be highly flexible, scalable, and customizable, making it a popular choice for building cloud solutions.</span></p><p style="text-align: justify;">OpenStack is an open-source cloud computing platform that was initially launched in July 2010 as a joint project by Rackspace Hosting and NASA. Since then, it has grown into a vibrant open-source community with contributions from a wide range of organizations and individuals. Here's a brief history of OpenStack and an overview of its main components:</p><p style="text-align: justify;"><b>OpenStack History:</b></p><p style="text-align: justify;">Launch (2010): OpenStack was publicly launched in July 2010 with the release of the first two core projects, Nova (compute) and Swift (object storage). It was created to address the need for an open and flexible cloud computing platform.</p><p style="text-align: justify;">Expanding Community (2011-2012): The OpenStack community quickly expanded, with numerous companies joining the project. The community released new versions of OpenStack, including Diablo, Essex, and Folsom, each with additional core and supporting projects.</p><p style="text-align: justify;">Foundation Establishment (2012): In September 2012, the OpenStack Foundation was established to oversee the project's development and ensure its long-term governance as an open-source project.</p><p style="text-align: justify;">Maturing Ecosystem (2013-2015): OpenStack continued to evolve, with new releases like Grizzly, Havana, Icehouse, and Juno. During this period, more projects were added to the ecosystem, covering areas such as networking (Neutron), block storage (Cinder), and identity (Keystone).</p><p style="text-align: justify;">Enterprise Adoption (2016-2017): OpenStack gained significant traction among enterprises and service providers. Projects like Heat (orchestration) and Magnum (containers) were introduced to support cloud automation and container orchestration.</p><p style="text-align: justify;">Continued Growth (2018-Present): OpenStack has continued to grow and evolve, with new projects and features being added regularly. The community releases new versions of OpenStack every six months, with each version introducing enhancements and improvements.</p><p style="text-align: justify;"><b>Openstack Releases: </b>Currently running Openstack is release is "xena". Austin was the 1st Openstack release and it obsolete now. For more details check the links below:</p><p style="text-align: justify;"><a href="https://docs.openstack.org/puppet-openstack-guide/latest/install/releases.html">https://docs.openstack.org/puppet-openstack-guide/latest/install/releases.html</a></p><p style="text-align: justify;"><a href="https://releases.openstack.org/">https://releases.openstack.org/</a></p><p style="text-align: justify;"><span style="font-size: x-small;"><span>Austin (2010): The first official release of OpenStack, code-named "Austin."</span><br />Bexar (2011): The second release, code-named "Bexar."<br />Cactus (2011): The third release, code-named "Cactus."<br />Diablo (2011): The fourth release, code-named "Diablo."<br />Essex (2012): The fifth release, code-named "Essex."<br />Folsom (2012): The sixth release, code-named "Folsom."<br />Grizzly (2013): The seventh release, code-named "Grizzly."<br />Havana (2013): The eighth release, code-named "Havana."<br />Icehouse (2014): The ninth release, code-named "Icehouse."<br />Juno (2014): The tenth release, code-named "Juno."<br />Kilo (2015): The eleventh release, code-named "Kilo."<br />Liberty (2015): The twelfth release, code-named "Liberty."<br />Mitaka (2016): The thirteenth release, code-named "Mitaka."<br />Newton (2016): The fourteenth release, code-named "Newton."<br />Ocata (2017): The fifteenth release, code-named "Ocata."<br />Pike (2017): The sixteenth release, code-named "Pike."<br />Queens (2018): The seventeenth release, code-named "Queens."<br />Rocky (2018): The eighteenth release, code-named "Rocky."<br />Stein (2019): The nineteenth release, code-named "Stein."<br />Train (2019): The twentieth release, code-named "Train."<br />Ussuri (2020): The twenty-first release, code-named "Ussuri."<br />Victoria (2020): The twenty-second release, code-named "Victoria."<br />Wallaby (2021): The twenty-third release, code-named "Wallaby."<br />Xena (2021): The twenty-fourth release, code-named "Xena."<br />Yoga (2022): The twenty-fifth release, code-named "Yoga."<br />Zuul (2022): The twenty-sixth release, code-named "Zuul."</span></p><p style="text-align: justify;">OpenStack's modular architecture allows organizations to choose the components that best fit their cloud computing needs, making it a versatile and customizable platform for building private, public, and hybrid clouds. OpenStack is built using a modular architecture, where each component provides a specific cloud service. These components can be combined to create a custom cloud infrastructure tailored to the organization's needs. OpenStack is composed of multiple projects, each providing a specific cloud service. </p><p></p><ol style="text-align: left;"><li>Multi-Tenancy: OpenStack supports multi-tenancy, allowing organizations to create isolated environments within the cloud infrastructure. This means that multiple users or projects can share the same cloud while maintaining security and resource separation.</li><li>Open Source: OpenStack is released under an open-source license, making it freely available for anyone to use, modify, and contribute to. This open nature has led to a vibrant community of developers and users collaborating on its development.</li><li>Integration and Compatibility: OpenStack is designed to integrate with various virtualization technologies, hardware vendors, and third-party tools. It can be used with different hypervisors, storage systems, and networking solutions.</li><li>Private and Public Clouds: Organizations can use OpenStack to create private clouds within their data centers or deploy public cloud services to offer cloud resources to external customers or users.</li><li>Hybrid Clouds: OpenStack can be part of a hybrid cloud strategy, where organizations combine private and public cloud resources to achieve flexibility and scalability</li></ol><p></p><p>Here are some of the main components:</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSP2kqLqxtyPWASJz6MNn1-GQUUNGjAXWK_d1EsQ2nLzOpxy-obt36zDoz1b5gCnUKxa4eKHJToV5agTasciQ4K3_OlckiLjt6cclz-b7moTT6Xe_xws5_BCkPAUAHpM0BGHyQ_RKEUdZPxSZODxnYbjhi91fApf_GCn2X5gd-lO89j4VH4dngkDav5s_m/s1920/openstack3.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1080" data-original-width="1920" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSP2kqLqxtyPWASJz6MNn1-GQUUNGjAXWK_d1EsQ2nLzOpxy-obt36zDoz1b5gCnUKxa4eKHJToV5agTasciQ4K3_OlckiLjt6cclz-b7moTT6Xe_xws5_BCkPAUAHpM0BGHyQ_RKEUdZPxSZODxnYbjhi91fApf_GCn2X5gd-lO89j4VH4dngkDav5s_m/w640-h360/openstack3.jpg" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.wikipedia.org/wiki/OpenStack#/media/File:Openstack-map-v20221001.jpg" target="_blank">source</a></td></tr></tbody></table><p></p><ol style="text-align: left;"><li>Nova (Compute): Manages and orchestrates virtual machines (instances) on hypervisors. It provides features for creating, scheduling, and managing VMs.</li><li>Swift (Object Storage): Offers scalable and durable object storage services for storing and retrieving data, including large files and unstructured data.</li><li>Cinder (Block Storage): Manages block storage volumes that can be attached to instances. It provides persistent storage for VMs.</li><li>Neutron (Networking): Handles networking services, including the creation and management of networks, subnets, routers, and security groups.</li><li>Keystone (Identity): Manages identity and authentication services, including user management, role-based access control (RBAC), and token authentication.</li><li>Glance (Image Service): Stores and manages virtual machine images (VM snapshots) that can be used to create instances.</li><li>Horizon (Dashboard): A web-based user interface that provides a graphical way to manage and monitor OpenStack resources.</li><li>Heat (Orchestration): Provides orchestration and automation services for defining and managing cloud application stacks.</li><li>Ceilometer (Telemetry): Collects telemetry data, including usage and performance statistics, for billing, monitoring, and auditing.</li><li>Trove (Database-as-a-Service): Manages database instances as a service, making it easier to provision and manage databases.</li><li>Ironic (Bare Metal): Manages bare-metal servers as a service, allowing users to provision physical machines in the same way as virtual machines.</li><li>Zaqar (Messaging and Queuing): Provides messaging and queuing services for distributed applications.</li><li>Magnum (Container Orchestration): Orchestrates container platforms like Kubernetes to manage containerized applications.</li></ol><p></p><p><br /></p><p style="text-align: justify;">Postman provides a user-friendly interface for building and sending API requests, inspecting responses, and automating API testing. Internally, Postman is a comprehensive software tool that facilitates the process of sending HTTP requests to APIs, receiving responses, and performing various tasks related to API testing, monitoring, and development. It operates through a combination of user interactions and underlying components. Postman simplifies the process of sending HTTP requests to APIs by providing a user-friendly interface, generating HTTP requests based on user input, and enabling users to work with API responses. It also supports more advanced features such as scripting, automation, and test execution for comprehensive API testing and monitoring. It's widely used by developers to</p><p></p><ol style="text-align: left;"><li>Test APIs: Developers can use Postman to send requests to APIs and receive responses, making it easy to test how the API functions.</li><li>Automate Tests: Postman allows you to create and automate test scripts to ensure that your APIs are working as expected. You can set up tests to validate the response data, headers, and more.</li><li>Document APIs: You can use Postman to generate API documentation, which is useful for sharing information about how to use an API with others.</li><li>Monitor APIs: Postman can be used to monitor APIs and receive alerts when issues or errors occur.</li><li>Mock Servers: Postman provides the ability to create mock servers, which can simulate an API's behavior without the actual backend being implemented yet.</li></ol><p></p><p> Here's how Postman is involved and invoked internally when working with the examples provided:</p><p>1) User Interface (UI): Postman provides a user-friendly graphical interface where users can create, manage, and send API requests. Users interact with this UI to input API details, such as request URLs, headers, parameters, and request bodies.</p><p>2) Request Configuration: When you create a request in Postman, you configure various aspects of the request, including the request method (e.g., GET, POST, PUT, DELETE), request URL, headers, query parameters, request body (if applicable), and authentication settings.</p><p>3) HTTP Request Generation: Postman internally generates the corresponding HTTP request based on the user's configuration. For example, if you configure a GET request to retrieve user data, Postman generates an HTTP GET request to the specified URL with the provided headers and parameters.</p><p>4) Request Sending: When you click the "Send" button within Postman, it sends the generated HTTP request to the target API endpoint using the configured settings (e.g., URL, headers, body). This request is sent via the HTTP protocol to the specified API server.</p><p>5) API Server Interaction: The HTTP request sent by Postman is received by the API server. The server processes the request based on the HTTP method, URL, and other request details. For example, in a RESTful API, a GET request may retrieve data, while a POST request may create new data.</p><p>6) Response Reception: After the API server processes the request, it sends an HTTP response back to Postman. This response includes data (e.g., JSON or XML) and metadata (e.g., status code, headers) generated by the server.</p><p>7) Response Handling: Postman receives the HTTP response and presents it to the user within its UI. The user can inspect the response content, status code, headers, and other details. Postman also provides tools for handling response data, such as extracting values or running tests.</p><p>8)Test Execution: Users can define tests and assertions within Postman using scripts (e.g., JavaScript). When a test script is defined, Postman internally executes the script and checks the results against the specified assertions.</p><p>9) Results Reporting: Postman provides feedback to the user about the outcome of the API request and any tests that were run. Users can view whether the request was successful, the response met the expected criteria, and any potential errors or issues.</p><p>10)Automation: Postman can be integrated into automated testing pipelines, continuous integration (CI) workflows, and monitoring systems. It can be invoked programmatically to run collections of requests, automate tests, and monitor APIs at specified intervals.</p><p><b>Examples:</b> make sure you have access to a RESTful API that you want to test. Replace the URL, endpoints, and parameters with the appropriate values for your specific API.</p><p>1) GET Request to Retrieve Data . To retrieve data from an API using a GET request:</p><p></p><ul style="text-align: left;"><li> GET https://api.example.com/users</li></ul><p></p><p>2) GET Request with Query Parameters.To retrieve data with query parameters:</p><p></p><ul style="text-align: left;"><li>GET https://api.example.com/users?id=123&name=John</li></ul><p></p><p>3) POST Request to Create Data.To create data using a POST request with a JSON body:</p><p></p><ul style="text-align: left;"><li>POST https://api.example.com/users</li></ul><span style="font-size: xx-small;">Headers:</span><br /><span style="font-size: x-small;">Content-Type: application/json</span><br /><span style="font-size: xx-small;"><br /></span><span style="font-size: x-small;">Body (JSON):</span><br /><span style="font-size: x-small;">{</span><br /><span style="font-size: x-small;"> "name": "Alice",</span><br /><span style="font-size: x-small;"> "email": "alice@example.com"</span><br /><span style="font-size: x-small;">}</span><p></p><p>4) PUT Request to Update Data.To update data using a PUT request with a JSON body:</p><p></p><ul style="text-align: left;"><li>PUT https://api.example.com/users/123</li></ul><p></p><p style="text-align: left;"><span style="font-size: xx-small;">Headers:</span><br /><span style="font-size: x-small;">Content-Type: application/json</span><br /><span style="font-size: xx-small;"><br /></span><span style="font-size: x-small;">Body (JSON):</span><br /><span style="font-size: x-small;">{</span><br /><span style="font-size: x-small;"> "name": "Updated Name",</span><br /><span style="font-size: x-small;"> "email": "updated@example.com"</span><br /><span style="font-size: x-small;">}</span></p><p><br /></p><p>5) DELETE Request to Remove Data. To delete data using a DELETE request:</p><p></p><ul style="text-align: left;"><li> DELETE https://api.example.com/users/123</li></ul><p></p><p>6) Headers and Authentication. You can add headers, such as authorization headers, to your requests. For example, to send an API key in the headers</p><p></p><ul style="text-align: left;"><li>GET https://api.example.com/resource</li></ul><span style="font-size: xx-small;">Headers:<br />Authorization: Bearer YOUR_API_KEY</span><p></p><p>7) Handling Response Data:After sending a request, you can inspect the response data. For example, to extract a specific value from the response, you can use JavaScript-like syntax in Postman's Tests tab:</p><p style="text-align: left;"><span style="font-size: xx-small;">// Extract the value of the "name" field from the JSON response</span><br /><span style="font-size: x-small;">var jsonData = pm.response.json();</span><br /><span style="font-size: x-small;">pm.environment.set("username", jsonData.name);</span></p><p>These are just some basic examples of how to use Postman to interact with RESTful APIs. You can create collections of requests, use variables, and write more complex tests to thoroughly test and validate your APIs.</p><p>Python code example that demonstrates how to make an HTTP GET request to a RESTful API using the popular requests library. In this example, we'll use the JSONPlaceholder API, which provides dummy data for testing and learning purposes:</p><p style="text-align: left;"><span style="font-size: xx-small;">import requests<br /># Define the API endpoint URL<br />api_url = "https://jsonplaceholder.typicode.com/posts/1"<br />try:<br /> # Send an HTTP GET request to the API endpoint<br /> response = requests.get(api_url)<br /> # Check if the request was successful (status code 200)<br /> if response.status_code == 200:<br /> # Parse the JSON response<br /> data = response.json()<br /> # Print the response data<br /> print("Title:", data["title"])<br /> print("Body:", data["body"])<br /> else:<br /> print("HTTP Request Failed with Status Code:", response.status_code)<br />except requests.exceptions.RequestException as e:<br /> # Handle any exceptions that may occur during the request<br /> print("An error occurred:", e)</span></p><p style="text-align: justify;"><b>NOTE:</b> We define the API endpoint URL (api_url) that we want to retrieve data from. In this example, we're fetching data for a specific post using its ID.</p><p style="text-align: justify;">and use a try block to send an HTTP GET request to the API endpoint using requests.get(api_url).</p><p style="text-align: justify;">We check the HTTP response status code. If it's 200, the request was successful, and we proceed to parse the JSON response.If the request was successful, we parse the JSON response using response.json() and print specific fields from the response (in this case, the post's title and body). If the request fails or encounters an exception, we handle it and print an error message.</p><p style="text-align: justify;">OpenStack provides a set of RESTful APIs for managing cloud infrastructure resources. These APIs are used to create, manage, and interact with virtualized resources such as instances (virtual machines), volumes, networks, and more. Here are some common API endpoint examples with respect to OpenStack:</p><p><span style="font-size: x-small;">1) Identity (Keystone) API:</span></p><p><span style="font-size: x-small;">Authentication and token management.</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:5000/v3/</span></p><p><span style="font-size: x-small;">Compute (Nova) API:</span></p><p><span style="font-size: x-small;">2) Management of virtual machines (instances).</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:8774/v2.1/</span></p><p><span style="font-size: x-small;">Block Storage (Cinder) API:</span></p><p><span style="font-size: x-small;">3) Management of block storage volumes.</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:8776/v2/</span></p><p><span style="font-size: x-small;">Object Storage (Swift) API:</span></p><p><span style="font-size: x-small;">4) Storage and retrieval of objects (files and data).</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:8080/v1/</span></p><p><span style="font-size: x-small;">Image (Glance) API:</span></p><p><span style="font-size: x-small;">5) Management of virtual machine images (VM snapshots).</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:9292/v2/</span></p><p><span style="font-size: x-small;">Network (Neutron) API:</span></p><p><span style="font-size: x-small;">6) Management of network resources, including routers, subnets, and security groups.</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:9696/v2.0/</span></p><p><span style="font-size: x-small;">Orchestration (Heat) API:</span></p><p><span style="font-size: x-small;">7) Orchestration of cloud resources through templates.</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:8004/v1/</span></p><p><span style="font-size: x-small;">Telemetry (Ceilometer) API:</span></p><p><span style="font-size: x-small;">8) Collection of usage and performance data.</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:8777/v2/</span></p><p><span style="font-size: x-small;">Dashboard (Horizon) API:</span></p><p><span style="font-size: x-small;">9) Web-based user interface for OpenStack services.</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>/dashboard/</span></p><p><span style="font-size: x-small;">Placement (Placement) API:</span></p><p><span style="font-size: x-small;">10) Management of resource placement and allocation.</span></p><p><span style="font-size: x-small;">Example: http://<OpenStack-IP>:8778/</span></p><p><span style="font-size: x-small;">These are just some examples of the core OpenStack APIs and their respective endpoint URLs.</span></p><p>--------</p><p style="text-align: justify;">To check if a user exists in your OpenStack environment, you can use the Identity (Keystone) API, which manages authentication and user-related operations. Specifically, you can make a request to the Keystone API to list users and then check if the desired user is in the list. Here are the general steps to do this:</p><p style="text-align: justify;">Step 1 :Authenticate with Keystone:</p><p style="text-align: justify;">Before making any requests to the Keystone API, you need to authenticate. Typically, this involves sending a POST request with your credentials to the Keystone authentication endpoint. You'll receive a token in response, which you can use to make subsequent API requests.</p><p>List Users:</p><p>Step 2 : Make a GET request to the Keystone API's user listing endpoint to retrieve a list of all users in the OpenStack environment.</p><p>Example API endpoint for listing users: http://<OpenStack-IP>:5000/v3/users</p><p>Include the authentication token in the request headers.</p><p>Check User Existence:</p><p>Step 3 : After receiving the list of users, you can iterate through the user data and check if the desired user exists by comparing usernames, IDs, or other unique identifiers.</p><p>Here's a Python example using the requests library to check if a user exists in Keystone:</p><p style="text-align: left;"><span style="font-size: xx-small;">import requests</span><br /><span style="font-size: x-small;"># Keystone authentication endpoint</span><br /><span style="font-size: x-small;">auth_url = "http://<OpenStack-IP>:5000/v3/auth/tokens"</span><br /><span style="font-size: x-small;"># Keystone user listing endpoint</span><br /><span style="font-size: x-small;">users_url = "http://<OpenStack-IP>:5000/v3/users"</span><br /><span style="font-size: x-small;"># Replace with your OpenStack credentials</span><br /><span style="font-size: x-small;">auth_data = {</span><br /><span style="font-size: x-small;"> "auth": {</span><br /><span style="font-size: x-small;"> "identity": {</span><br /><span style="font-size: x-small;"> "methods": ["password"],</span><br /><span style="font-size: x-small;"> "password": {</span><br /><span style="font-size: x-small;"> "user": {</span><br /><span style="font-size: x-small;"> "name": "your_username",</span><br /><span style="font-size: x-small;"> "domain": {"name": "your_domain"},</span><br /><span style="font-size: x-small;"> "password": "your_password"</span><br /><span style="font-size: x-small;"> }</span><br /><span style="font-size: x-small;"> }</span><br /><span style="font-size: x-small;"> }</span><br /><span style="font-size: x-small;"> }</span><br /><span style="font-size: x-small;">}</span><br /><span style="font-size: xx-small;"><br /></span><span style="font-size: x-small;"># Authenticate and get a token</span><br /><span style="font-size: x-small;">response = requests.post(auth_url, json=auth_data)</span><br /><span style="font-size: x-small;">if response.status_code == 201:</span><br /><span style="font-size: x-small;"> token = response.headers["X-Subject-Token"]</span><br /><span style="font-size: x-small;"> </span><br /><span style="font-size: x-small;"> # List all users</span><br /><span style="font-size: x-small;"> headers = {"X-Auth-Token": token}</span><br /><span style="font-size: x-small;"> response = requests.get(users_url, headers=headers)</span><br /><span style="font-size: xx-small;"><br /></span><span style="font-size: x-small;"> if response.status_code == 200:</span><br /><span style="font-size: x-small;"> users = response.json()["users"]</span><br /><span style="font-size: xx-small;"><br /></span><span style="font-size: x-small;"> # Check if the user exists</span><br /><span style="font-size: x-small;"> target_user = "desired_username"</span><br /><span style="font-size: x-small;"> user_exists = any(user["name"] == target_user for user in users)</span><br /><span style="font-size: x-small;"> if user_exists:</span><br /><span style="font-size: x-small;"> print(f"User {target_user} exists.")</span><br /><span style="font-size: x-small;"> else:</span><br /><span style="font-size: x-small;"> print(f"User {target_user} does not exist.")</span><br /><span style="font-size: x-small;"> else:</span><br /><span style="font-size: x-small;"> print("Failed to list users.")</span><br /><span style="font-size: x-small;">else:</span><br /><span style="font-size: x-small;"> print("Authentication failed.")</span></p><p>This example demonstrates how to authenticate with Keystone, list users, and check if a specific user exists by comparing usernames. Replace placeholders with your OpenStack-specific values and adjust the code as needed for your environment</p><p>-----------------------</p><p><b>OpenStack service overview: </b></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNJMD_tpOHiYXGyNKQlooOJehoUT1K1P1BZ9NRwgpA2nIh9L6F7dJkTx9KBESo848O1-W68TbCGOmVvs5GyLkk6IyffytuJm4pmpU35Ku4jlXTJjmkEuyVqPi9uFyIR-NgzTtHTruwoG7stMhB-VvfOb0kiw7w2YTi5B2TZSTBhlqM2dzRCw2MwGd4xqDz/s1051/openstack2.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="337" data-original-width="1051" height="206" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNJMD_tpOHiYXGyNKQlooOJehoUT1K1P1BZ9NRwgpA2nIh9L6F7dJkTx9KBESo848O1-W68TbCGOmVvs5GyLkk6IyffytuJm4pmpU35Ku4jlXTJjmkEuyVqPi9uFyIR-NgzTtHTruwoG7stMhB-VvfOb0kiw7w2YTi5B2TZSTBhlqM2dzRCw2MwGd4xqDz/w640-h206/openstack2.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://docs.openstack.org/security-guide/introduction/introduction-to-openstack.html">source</a></td></tr></tbody></table><p style="text-align: justify;">Nova , Cinder, Swift and Neutron -these OpenStack services together provide a comprehensive cloud computing platform. Nova manages compute resources, Cinder offers block storage, Swift provides object storage, and Neutron handles networking, enabling organizations to build and manage private and public clouds tailored to their specific needs.</p><p style="text-align: justify;">Nova (OpenStack Compute): Nova is the core compute service in OpenStack. It manages the creation, scheduling, and management of virtual machines (VMs) in a cloud environment. Nova is hypervisor-agnostic, supporting various virtualization technologies, and it provides features for live migration, scaling, and resource management.</p><p style="text-align: justify;">Cinder (OpenStack Block Storage): Cinder is the block storage service in OpenStack. It offers block-level storage volumes that can be attached to VMs. Users can create, manage, and snapshot these volumes, making it suitable for data persistence in applications like databases.</p><p style="text-align: justify;">Swift (OpenStack Object Storage): Swift is the object storage service in OpenStack. It is designed for the storage of large amounts of unstructured data, such as images, videos, and backups. Swift provides scalable, redundant, and highly available storage with easy-to-use APIs.</p><p style="text-align: justify;">Neutron (OpenStack Networking): Neutron is the networking service in OpenStack. It enables users to create and manage networks, subnets, routers, and security groups for VMs. Neutron supports various network configurations, including flat networks, VLANs, and overlay networks, allowing for flexibility in network design.</p><p style="text-align: justify;">--------</p><p style="text-align: justify;">Key Differences between <b>Cinder and swift </b>: The object storage and block storage serve different purposes and have distinct access methods. Object storage is well-suited for handling unstructured data and large-scale content distribution, while block storage is preferred for applications requiring direct control over data blocks and high performance. Organizations often choose between these storage types based on their specific use cases and storage needs.</p><p style="text-align: justify;">Access Level: Object storage uses a higher-level access method, where data is accessed and managed as whole objects using unique identifiers. Block storage provides lower-level access, treating data as raw blocks.</p><p style="text-align: justify;">Use Cases: Object storage is ideal for storing large amounts of unstructured data and content distribution, while block storage is suited for applications requiring direct control over storage blocks.</p><p style="text-align: justify;">Scalability: Object storage is known for its horizontal scalability and ease of expansion, whereas block storage scalability may require more planning and management.</p><p style="text-align: justify;">Data Management: Object storage systems often manage data redundancy and durability internally, while block storage may rely on external solutions or the application to manage data redundancy.</p><p style="text-align: justify;">Data Retrieval: Object storage is optimized for read-heavy workloads and large-scale data distribution, while block storage is designed for high performance and low-latency access.</p><p style="text-align: justify;">------------</p><p style="text-align: justify;"><b>Ceph:</b></p><p style="text-align: justify;">Ceph is an open-source, distributed storage system designed for both object and block storage. It is known for its flexibility, scalability, and ability to provide a unified storage platform. Ceph is often used in cloud computing environments, data centers, and high-performance computing (HPC) clusters.</p><p style="text-align: justify;">Key components and features of Ceph include:</p><p style="text-align: justify;">Object Storage (RADOS Gateway): Ceph provides object storage capabilities through its RADOS (Reliable Autonomic Distributed Object Store) Gateway. This allows users to store and retrieve objects using a RESTful API compatible with Amazon S3 and Swift.</p><p style="text-align: justify;">Block Storage (RBD): Ceph's RADOS Block Device (RBD) allows users to create block storage volumes that can be attached to virtual machines or used as raw block storage. RBD is often integrated with virtualization platforms like KVM.</p><p style="text-align: justify;">Scalability: Ceph scales seamlessly from a few nodes to thousands of nodes by distributing data across OSDs (Object Storage Daemons) and MONs (Monitor Daemons). This scalability makes it suitable for large-scale storage deployments.</p><p style="text-align: justify;">Data Redundancy: Ceph replicates data across multiple OSDs to ensure redundancy and high availability. It uses a CRUSH algorithm to distribute data efficiently.</p><p style="text-align: justify;">Self-Healing: Ceph can automatically detect and recover from hardware failures or data inconsistencies. It continuously monitors data integrity.</p><p style="text-align: justify;">Unified Storage: Ceph provides a unified storage platform that combines object, block, and file storage, allowing users to access data in various ways, depending on their requirements.</p><p style="text-align: justify;">Community and Ecosystem: Ceph has a vibrant open-source community and a wide ecosystem of tools and projects that integrate with it. This includes interfaces for OpenStack integration.</p><p style="text-align: justify;">-------------------------</p><p style="text-align: justify;"><b>Neutron,</b> the networking component of OpenStack, plays a crucial role in creating and managing networking resources within a cloud infrastructure. </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2EVZWgg8uDMxQKehbPl92fT3CKp5c2oEvX5Lf0iAldHNl-nKtbR-UDeuch9J5A7rol8dS4o3hD4b5ZiIuq8BZ7nNnnm4xSO0LEbU7eb9LqWKvCTfv7bbwayboGhGRrjTQUHCgM5GQvOKzpyK5tkhUZtsDaNktxKd85NNUi_0hUzm02-jA9dPvnDnnGHve/s723/OpenStack-HA-neutron.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="468" data-original-width="723" height="414" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2EVZWgg8uDMxQKehbPl92fT3CKp5c2oEvX5Lf0iAldHNl-nKtbR-UDeuch9J5A7rol8dS4o3hD4b5ZiIuq8BZ7nNnnm4xSO0LEbU7eb9LqWKvCTfv7bbwayboGhGRrjTQUHCgM5GQvOKzpyK5tkhUZtsDaNktxKd85NNUi_0hUzm02-jA9dPvnDnnGHve/w640-h414/OpenStack-HA-neutron.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.researchgate.net/figure/OpenStack-HA-Architecture_fig1_308188705" target="_blank"><span style="font-size: xx-small;">source</span></a></td></tr></tbody></table><p style="text-align: justify;">Here are some interesting factors and capabilities related to Neutron:</p><p style="text-align: justify;">Network Abstraction: Neutron abstracts network resources, allowing users to create and manage virtual networks, subnets, routers, and security groups through APIs or the dashboard. This abstraction simplifies complex networking tasks and provides a consistent interface.</p><p style="text-align: justify;">Multi-Tenancy: Neutron supports multi-tenancy, enabling the isolation of network resources between different projects or tenants. This ensures that one tenant's network activities do not impact another's.</p><p style="text-align: justify;">Pluggable Architecture: Neutron follows a pluggable architecture, allowing users to integrate with various networking technologies and solutions. This includes support for different plugins and drivers, enabling compatibility with a wide range of network devices and services.</p><p style="text-align: justify;">Software-Defined Networking (SDN): Neutron can be used in conjunction with SDN controllers and solutions to provide advanced network automation, programmability, and flexibility. SDN allows for the dynamic configuration of network services and policies.</p><p style="text-align: justify;">Networking Interfaces: Neutron allows the creation of various types of networking interfaces for virtual machines, including:</p><p style="text-align: justify;">Port: Neutron manages ports, which represent virtual interfaces connected to a network. VMs attach to ports to access the network.</p><p style="text-align: justify;">Router: Routers connect different subnets and provide inter-subnet routing. Neutron manages router interfaces and routing rules.</p><p style="text-align: justify;">Floating IPs: Floating IPs provide external network access to VMs. Neutron can assign floating IPs dynamically or statically.</p><p style="text-align: justify;">Bonding and Teaming: Neutron can manage bonded network interfaces (NIC bonding) for redundancy and increased network bandwidth. This is especially useful for ensuring high availability and load balancing of VMs.</p><p style="text-align: justify;">Security Groups: Neutron's security groups feature allows users to define firewall rules and policies to control incoming and outgoing traffic to VMs. It enhances network security within the cloud environment.</p><p style="text-align: justify;">L3 and L2 Services: Neutron supports Layer 3 (routing) and Layer 2 (bridging) services. This flexibility enables complex network topologies and scenarios.</p><p style="text-align: justify;">Interoperability: Neutron integrates with various network technologies, including VLANs, VXLANs, GRE tunnels, and more. It provides interoperability with physical network infrastructure and external networks.</p><p style="text-align: justify;">Communication Between VMs: Neutron ensures that VMs can communicate with each other within the same network or across networks using routing. It manages the routing tables and connectivity.</p><p style="text-align: justify;">Load Balancing as a Service (LBaaS): Neutron offers LBaaS, allowing users to create and manage load balancers to distribute traffic among multiple VMs or instances.</p><p style="text-align: justify;">High Availability (HA): Neutron can be configured for high availability, ensuring network services remain operational even in the event of network node failures.</p><p style="text-align: justify;">---------------------------------------------------------</p><p style="text-align: justify;">Containerization in OpenStack involves deploying and managing containers within an OpenStack cloud environment. This allows users to run containerized applications and microservices alongside traditional virtual machines (VMs).</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgu07jT57k8S7Ohe2Gw_2ah5-Z91yi33wVGDzGs0PGBOmjdmdCqpskN-MlEhoX_IGvoHMwpv3QoakeyGFjrdybbV_ntBdDYt1v6eXiEmZLxOYbfi1uiqsoftYGD6LNpsRrf_OgTQTTdlBuq4VChrXtgGNqcxisG9H6DM505TB8TP6l5fq9d4CvV-trFKbtL/s1137/openstack6.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="635" data-original-width="1137" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgu07jT57k8S7Ohe2Gw_2ah5-Z91yi33wVGDzGs0PGBOmjdmdCqpskN-MlEhoX_IGvoHMwpv3QoakeyGFjrdybbV_ntBdDYt1v6eXiEmZLxOYbfi1uiqsoftYGD6LNpsRrf_OgTQTTdlBuq4VChrXtgGNqcxisG9H6DM505TB8TP6l5fq9d4CvV-trFKbtL/w640-h358/openstack6.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.openstack.org/use-cases/containers/leveraging-containers-and-openstack/" target="_blank"><span style="font-size: x-small;">source</span></a></td></tr></tbody></table><p style="text-align: justify;">Here's a step-by-step explanation of the design and components involved in containerization within OpenStack:</p><p style="text-align: justify;">1. Container Orchestration Framework: OpenStack supports various container orchestration frameworks, with Kubernetes being one of the most popular choices. Kubernetes helps manage the deployment, scaling, and operation of application containers. It serves as the foundation for container orchestration in an OpenStack environment.</p><p style="text-align: justify;">2. Container Runtime: Containers are run using a container runtime, such as Docker or containerd. This runtime manages the execution of containerized applications and provides isolation between containers. In an OpenStack-based containerization setup, a container runtime is installed on each compute node in the OpenStack cluster.</p><p style="text-align: justify;">3. OpenStack Components:</p><p style="text-align: justify;"></p><ul><li>Nova (Compute Service): Nova is responsible for managing compute resources, including VMs and, in a containerized environment, bare metal servers. It can provision servers specifically for running containers alongside traditional VMs.</li><li>Neutron (Networking Service): Neutron handles networking and connectivity for containers. It ensures that containers can communicate with each other, VMs, and external networks.</li><li>Cinder (Block Storage Service): Cinder provides block-level storage for containers when persistent storage is required. Containers can use Cinder volumes for data storage.</li></ul><p></p><p style="text-align: justify;">4. Magnum (Container Orchestration Service): OpenStack Magnum is a dedicated service for managing container orchestration clusters, such as Kubernetes, within the OpenStack cloud. It simplifies the deployment and management of container orchestration platforms.</p><p style="text-align: justify;">5. Heat (Orchestration Service): Heat is an orchestration service in OpenStack that enables the automated deployment and scaling of infrastructure resources, including containers. It allows users to define templates describing the desired container infrastructure and then deploys and manages the resources accordingly.</p><p style="text-align: justify;">6. Glance (Image Service): Glance is responsible for storing and managing container images. Containers are typically built from base images, and Glance helps manage these images within the OpenStack environment.</p><p style="text-align: justify;">7. Keystone (Identity Service): Keystone provides authentication and authorization services for containerized applications and services. It ensures that only authorized users and services can access containers and container orchestration platforms.</p><p style="text-align: justify;">8. Container Networking and Storage Plugins: In an OpenStack-based containerization environment, specialized networking and storage plugins are often used to integrate container networking and storage with OpenStack services. These plugins enable efficient communication and data storage for containers.</p><p style="text-align: justify;">9. User Interface: Users interact with the containerization platform through the OpenStack dashboard (Horizon) or through the command-line interface (CLI). They can deploy and manage containers, container orchestration clusters, and associated resources.</p><p style="text-align: justify;">10. Monitoring and Logging: Containerized applications generate logs and require monitoring for performance and resource usage. OpenStack can be integrated with monitoring and logging solutions like Prometheus, Grafana, and ELK (Elasticsearch, Logstash, and Kibana) to provide insights into containerized workloads.</p><p style="text-align: justify;">11. External Services Integration: Containers often need to interact with external services and APIs. OpenStack allows for integration with external services through the use of network configurations, load balancers, and other relevant components.</p><p style="text-align: justify;">In summary, containerization in OpenStack involves a combination of OpenStack services, container orchestration frameworks like Kubernetes, container runtimes, and specialized plugins to provide a seamless environment for deploying and managing containerized applications alongside traditional VMs within an OpenStack cloud infrastructure. This setup offers flexibility, scalability, and isolation for running containerized workloads in a cloud environment.</p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-6010611881101772742023-07-30T23:22:00.013+05:302023-08-07T16:31:02.498+05:30Watsonx AI and data platform with Foundation Models<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIPMtF7y-I30M2v_3Irk3JSD8oQ4h3ls8fThdofYDf-bzO47eR_2Wk23LjrkGbXgeP7CTARS-6J2uk4fIWO8x2l3aA08wd8FSnZB738hMNX2Qzy_yRUznVoLvjLqFSLQLqtW-Prw9LpWdzh1DDsNQd6GgMKI1gmBHdfSgxvxp36qUnSINrxWMQN8l4-sZu/s942/1.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="490" data-original-width="942" height="119" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIPMtF7y-I30M2v_3Irk3JSD8oQ4h3ls8fThdofYDf-bzO47eR_2Wk23LjrkGbXgeP7CTARS-6J2uk4fIWO8x2l3aA08wd8FSnZB738hMNX2Qzy_yRUznVoLvjLqFSLQLqtW-Prw9LpWdzh1DDsNQd6GgMKI1gmBHdfSgxvxp36qUnSINrxWMQN8l4-sZu/w244-h119/1.png" width="244" /></a></div><div style="text-align: justify;">We are witnessing a fundamental shift in AI driven by self-supervision and by the ability to create foundation models that power generative AI. Several exciting new Foundation Model capabilities have been announced at IBM Think 2023. Watsonx is a new platform for foundation models and generative AI, offering a studio, data store, and governance toolkit. Let’s take a look what this platform intends to provide.</div><p></p><p style="text-align: justify;">Why can't we build and reuse AI models? More data, more problems? Learn how AI foundation models change the game for training AI/ML from IBM Research AI VP Sriram Raghavan and Darío Gil, SVP and Director of IBM Research as they demystifies the technology and shares a set of principles to guide your generative AI business strategy. Experience watsonx, IBM’s new data and AI platform for generative AI and learn about the breakthroughs that IBM Research is bringing to this platform and to the world of computing. and to explore foundation models, an emerging approach to machine learning and data representation. Even in the age of big data when AI/ML is more prevalent, training the next generation of AI tools like NLP requires enormous data, and using AI models to new or different domains may be tricky. A foundation model can consolidate data from several sources so that one model may then be used for various activities. But how will foundation models be used for things beyond natural language processing? Don't miss this episode to explore how foundation models are a paradigm shift in how AI gets done. </p><p style="text-align: justify;">You can bring your own data and AI models to watsonx or choose from a library of tools and technologies. You can train or influence training (if you want), then you can tune, that way you can have transparency and control over governing data and AI models. You can prompt it too. Instead of only one model, you can have family of models. The foundation models trained with your own data will become more valuable asset. Watsonx is a new integrated data platform to become a value creator. It consists of 3 primary parts, first watsonx.data is massive curated data repository that is ready to be tapped to train and fine-tune models with data management system. Watsonx.ai is an enterprise studio to train, validate, tune and deploy traditional ML and foundation models that provide generative AI capabilities. Watson.governance is a powerful set of tools to ensure your AI is executing responsibly. They work together seemlessly throughout the entire lifecycle of foundation models. Watsonx built on top of RedHat Openshift. The lifecycle consists of </p><p style="text-align: justify;">STEP 1: preparing our data [Acquire, filter and pre-process, version & tag]. Each data set after being filtered , processed , it receives a data card. Data card has name and version of pile, specifies its content and filters that have been applied to it. We can have multiple data piles . They co-exists in .data and access different versions of data maintained for different purpose is managed seamlessly.</p><p style="text-align: justify;">STEP2 : using it to train the model, validate the model, Tune the model and deploying applications and solutions. So we moved from .data to .AI and start picking a model architecture from the five families that IBM provides. These are bedrocks of models and they range from encoder only, encoder-decoder, decoder only and other novel architectures.</p><p style="text-align: justify;">What Are Foundation Models? . Foundation models are AI neural networks trained on massive unlabeled datasets to handle a wide variety of jobs from translating text to analyzing medical images.<span style="text-align: left;"> We're witnessing a transition in AI. Systems that execute specific tasks in a single domain are giving way to broad AI that learns more generally and works across domains and problems. Foundation models, trained on large, unlabeled datasets and fine-tuned for an array of applications, are driving this shift. </span>The models are pre-trained to support a range of natural language processing (NLP) type tasks including question answering, content generation and summarization, text classification and extraction. Future releases will provide access to a greater variety of IBM-trained proprietary foundation models for efficient domain and task specialization.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSoK4Rvkr8GH_BnRuygccde1wWTuy_gzvvoGTTilFUdVf3GopeQCD4bScbj0tHcuolT_8zsBjzFfaajInWKf4bxWY2cyeEyT-16LUnPenWbgPku264dTEB4Ijhf5Du-IomGUnAKyMcHh_c3CMRkp-myZvAbyAbBX0vrusgtASjnLz4tW0yHzG8c7E7nrDv/s1580/Data_tasks.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1080" data-original-width="1580" height="438" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSoK4Rvkr8GH_BnRuygccde1wWTuy_gzvvoGTTilFUdVf3GopeQCD4bScbj0tHcuolT_8zsBjzFfaajInWKf4bxWY2cyeEyT-16LUnPenWbgPku264dTEB4Ijhf5Du-IomGUnAKyMcHh_c3CMRkp-myZvAbyAbBX0vrusgtASjnLz4tW0yHzG8c7E7nrDv/w640-h438/Data_tasks.jpg" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://blogs.nvidia.com/blog/2023/03/13/what-are-foundation-models/"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><p style="text-align: justify;">Foundation models are trained with massive amounts of data that allow for generative AI capabilities with a broad set of raw data that can be applied to different tasks, such as natural language processing. Instead of one model built solely for one task, foundation models can be adapted across a wide variety of different scenarios, summarizing documents, generating stories, answering questions, writing code, solving math problems, synthesizing audio. A year after the group defined foundation models, other tech watchers coined a related term — generative AI. It’s an umbrella term for transformers, large language models, diffusion models and other neural networks capturing people’s imaginations because they can create text, images, music, software and more.</p><p>IBM has planned to offer a suite of foundation models, for example smaller encoder based models, but also encoder-decoder or just decoder based models. </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsJCBAIyXMSMy594XZLBD8yrRoQ0YdQ21LZnCAHQtOg2SGoWMEaZq-rAPmMPf4Ug77zlYng2P4YWWmNm5XBTrz-__XNoDAu7aFCaZQh2rKrI__31PknvaK4czrg7L5fFWZL2igj-oe6OyhqiuIFzdk0pxBpJgl0qiFLyT9xNWy4-6DraUgUeUVJAavAtlI/s1718/IBM-foundation_models.png" style="margin-left: auto; margin-right: auto;"><span style="color: black;"><img border="0" data-original-height="924" data-original-width="1718" height="344" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsJCBAIyXMSMy594XZLBD8yrRoQ0YdQ21LZnCAHQtOg2SGoWMEaZq-rAPmMPf4Ug77zlYng2P4YWWmNm5XBTrz-__XNoDAu7aFCaZQh2rKrI__31PknvaK4czrg7L5fFWZL2igj-oe6OyhqiuIFzdk0pxBpJgl0qiFLyT9xNWy4-6DraUgUeUVJAavAtlI/w640-h344/IBM-foundation_models.png" title="source" width="640" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.youtube.com/watch?v=FrDnPTPgEmk"><span style="color: black;">source</span></a><br /><br /></td></tr></tbody></table><div style="text-align: justify;">Recognizing that one size doesn’t fit all, we’re building a family of language and code foundation models of different sizes and architectures. Each model family has a geology-themed code name —<b>Granite, Sandstone, Obsidian, and Slate</b> — which brings together cutting-edge innovations from IBM Research and the open research community. Each model can be customized for a range of enterprise tasks. While Foundation Models are in general good in performing multiple tasks, they have been trained with generic data. To optimize them, fine tuning with domain specific or proprietary data can be done.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><p>Watsonx is our enterprise-ready AI and data platform designed to multiply the impact of AI across your business. The platform comprises three powerful products: the watsonx.ai studio for new foundation models, generative AI and machine learning; the watsonx.data fit-for-purpose data store, built on an open lakehouse architecture; and the watsonx.governance toolkit, to accelerate AI workflows that are built with responsibility, transparency and explainability. It consists of Watsonx.data, Watsonx.ai and Watsonx.governance</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgohHDLYFk30mLNh8s-_Pl3mswfON0cvTZoN_zKVXHnQQDcO-T1f3LLxSA06vA3qVcixTd3TrKlwrnQxUK2d8k8bhSJ4X5Jo1reiWeKhiBbC1vgCOGGLcuKqzezaCpceoAu9Y4G5nWAngcrIwtl0hdU3cXSvHPA5QipxoEDfV9kk0dQbJmUIg-2s1SV6KAj/s1183/WatsonX-3-parts.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="885" data-original-width="1183" height="478" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgohHDLYFk30mLNh8s-_Pl3mswfON0cvTZoN_zKVXHnQQDcO-T1f3LLxSA06vA3qVcixTd3TrKlwrnQxUK2d8k8bhSJ4X5Jo1reiWeKhiBbC1vgCOGGLcuKqzezaCpceoAu9Y4G5nWAngcrIwtl0hdU3cXSvHPA5QipxoEDfV9kk0dQbJmUIg-2s1SV6KAj/w640-h478/WatsonX-3-parts.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.youtube.com/watch?v=FrDnPTPgEmk"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><br /><div><br /></div><div><div><b>Watsonx.data : An open, hybrid and governed data store</b></div><div>It makes it possible for enterprises to scale analytics and AI with a fit-for-purpose data store, built on an open lakehouse architecture, supported by querying, governance and open data formats to access and share data. With watsonx.data, you can connect to data in minutes, quickly get trusted insights and reduce your data warehouse costs. Now available as a service on IBM Cloud and AWS and as containerized software.</div><div><br /></div></div><div><p><b>Watsonx.ai Studio: </b>is an AI studio that combines the capabilities of IBM Watson Studio with the latest generative AI capabilities that leverage the power of foundation models. It provides access to high-quality, pre-trained, and proprietary IBM foundation models built with a rigorous focus on data acquisition, provenance, and quality. watsonx.ai is user-friendly. It’s not just for data scientists & developers, but also for business users. It provides a simple, natural language interface for different tasks. Watsonx.ai Studio with the new playground including easy to use Prompt Tuning. With watsonx.xi, you can train, validate, tune and deploy AI models.</p><p><b>WatsonX.governance</b> : IBM has described watsonX.governance as a tool for building responsible, transparent and explainable AI workflows. According to IBM, watsonx.governance will also enable customers to direct, manage and monitor AI activities, map with regulatory requirements, and address ethical issues. The more AI is embedded into daily workflows, the more you need proactive governance to drive responsible, ethical decisions across the business. Watsonx.governance allows you to direct, manage, and monitor your organization’s AI activities, and employs software automation to strengthen your ability to mitigate risk, manage regulatory requirements and address ethical concerns without the excessive costs of switching your data science platform—even for models developed using third-party tools.</p></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWEgdvsCHpgW33AGfor_mfK3C85yDmXJqlVNkv3zYLhV8sgauytlg0VdgfCkzxigNU_osiNK28JACl3UVuuyJjzcoPXBLGKuFxRELoXmkIprStgdt2NpFOVilD83zkrmqyzP0xiE3_udnyZ4qsKn380cRRxBEXrEJHi3qbKxNjDRCQI7GH-dX5rfETIlnb/s1365/watsonX-openshift-clouds.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="857" data-original-width="1365" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWEgdvsCHpgW33AGfor_mfK3C85yDmXJqlVNkv3zYLhV8sgauytlg0VdgfCkzxigNU_osiNK28JACl3UVuuyJjzcoPXBLGKuFxRELoXmkIprStgdt2NpFOVilD83zkrmqyzP0xiE3_udnyZ4qsKn380cRRxBEXrEJHi3qbKxNjDRCQI7GH-dX5rfETIlnb/w640-h402/watsonX-openshift-clouds.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.youtube.com/watch?v=FrDnPTPgEmk"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><p><br /></p></div><div style="text-align: justify;">IBM plans to provide Foundation Models as a Service with the capabilities of IBM’s first AI-optimized, cloud-native supercomputer Vela as a Service. The stack utilizes Red Hat OpenShift, so that it could also be run on multiple clouds or on-premises. It is based on popular open source frameworks and communities like PyTorch, Ray and Hugging Face. </div><div style="text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYUhutgsdcPCMzRNeXQSddq71qJRMnnym8PCjurkIKi7CqTVwVx0FPDUo0euYCRp7rxz-Sei-iDGrTANB9zz8Expl0Am9PQNNvLfCAdDC44x_lQkZqqoWOSsWskGgi9oBat2sPefpRRNVQfWrLkUan5jDr2cSC2Y2Ur6Uo5EEUu9_4OqUDtZwvLiu-CN-1/s762/IBM-vela.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="762" data-original-width="732" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYUhutgsdcPCMzRNeXQSddq71qJRMnnym8PCjurkIKi7CqTVwVx0FPDUo0euYCRp7rxz-Sei-iDGrTANB9zz8Expl0Am9PQNNvLfCAdDC44x_lQkZqqoWOSsWskGgi9oBat2sPefpRRNVQfWrLkUan5jDr2cSC2Y2Ur6Uo5EEUu9_4OqUDtZwvLiu-CN-1/w614-h640/IBM-vela.png" width="614" /></a><span style="text-align: justify;"> </span></div><div><p>Why we built an AI supercomputer in the cloud?. Introducing Vela, IBM’s first AI-optimized, cloud-native supercomputer.</p><p style="text-align: justify;">IBM built Vela supercomputer designed specifically for training so-called “foundation” AI models such as GPT-3. According to IBM, this new supercomputer should become the basis for all its own research and development activities for these types of AI models.IBM’s Vela supercomputer uses x86-based standard hardware. In the Vela system, each node’s hardware consists of a pair of “regular” Intel Xeon Scalable processors. To this are added eight 80GB Nvidia A100 GPUs per node. Furthermore, each node within the supercomputer is connected to several 100 Gbps Ethernet network interfaces. Each Vela node also has 1.5TB of DRAM internal memory and four 3.2TB NVMe drives for storage.In addition, IBM has also built a new workload-scheduling system for the Vela, the MultiCluster App Dispatcher (MCAD) system. This should handle cloud-based job scheduling for training foundation AI models.</p><p>Multi-Cluster Application Dispatcher:</p><p style="text-align: justify;">The multi-cluster-app-dispatcher is a Kubernetes controller providing mechanisms for applications to manage batch jobs in a single or mult-cluster environment. The multi-cluster-app-dispatcher (MCAD) controller is capable of (i) providing an abstraction for wrapping all resources of the job/application and treating them holistically, (ii) queuing job/application creation requests and applying different queuing policies, e.g., First In First Out, Priority, (iii) dispatching the job to one of multiple clusters, where a MCAD queuing agent runs, using configurable dispatch policies, and (iv) auto-scaling pod sets, balancing job demands and cluster availability.</p><p>What is prompt-tuning?</p><p style="text-align: justify;">Prompt-tuning is an efficient, low-cost way of adapting an AI foundation model to new downstream tasks without retraining the model and updating its weights. Redeploying an AI model without retraining it can cut computing and energy use by at least 1,000 times, saving thousands of dollars. With prompt-tuning, you can rapidly spin up a powerful model for your particular needs. It also lets you move faster and experiment.</p><p style="text-align: justify;">In prompt-tuning, the best cues, or front-end prompts, are fed to your AI model to give it task-specific context. The prompts can be extra words introduced by a human, or AI-generated numbers introduced into the model's embedding layer. Like crossword puzzle clues, both prompt types guide the model toward a desired decision or prediction. Prompt-tuning allows a company with limited data to tailor a massive model to a narrow task. It also eliminates the need to update the model’s billions (or trillions) of weights, or parameters. Prompt-tuning originated with large language models but has since expanded to other foundation models, like transformers that handle other sequential data types, including audio and video. Prompts may be snippets of text, streams of speech, or blocks of pixels in a still image or video. We don’t touch the model. It’s frozen. </p><p style="text-align: justify;"><a href="https://blog.hootsuite.com/ai-art-prompts/">For Example: How do AI art generators work?</a></p><p style="text-align: justify;"> AI art generators don’t know what an owl looks like in the wild. They don’t know what a sunset looks like in a physical sense. They can only understand details about features, patterns, and relationships within the datasets they’ve been trained on. Prompting for a “beautiful face” is not very helpful. It is more effective to prompt for specific features such as symmetry, big lips, and green eyes. Even if the bot doesn’t understand beauty, it can recognize the features you describe as beautiful and generate something relatively accurate. To get the best results from your AI art generator prompt, you’ll need to give clear and detailed instructions. An effective AI art prompt should include specific descriptions, shapes, colors, textures, patterns, and artistic styles. This allows the neural networks used by the generator to create the best possible visuals.</p><p style="text-align: justify;">T5 (Text to test transfer transformer ) is an encoder decoder model pre trained on a multi-task mixture of unsupervised and supervised tasks. We have complete transformer. T5 provides simple way to train a single model on a wide variety of text tasks. FLAN is Fine-Tuning LANguage Model. FLAN already been fine tuned by google and you try your multiple tasks on already pre tuned Model by Google. If you fine tune, then you may destroy that fine tuned model by overwriting it. Flan-UL2 is an encoder decoder model based on the T5 architecture. It uses the same configuration as the UL2 model released earlier last year. It was fine tuned using the “Flan” prompt tuning and dataset collection. With its impressive 20 billion parameters, Flan-UL2 is a remarkable encoder-decoder model with exceptional performance. UL2 20B: An Open Source Unified Language Learner.In “Unifying Language Learning Paradigms”, we present a novel language pre-training paradigm called Unified Language Learner (UL2) that improves the performance of language models universally across datasets and setups. UL2 frames different objective functions for training language models as denoising tasks, where the model has to recover missing sub-sequences of a given input. During pre-training it uses a novel mixture-of-denoisers that samples from a varied set of such objectives, each with different configurations. We demonstrate that models trained using the UL2 framework perform well in a variety of language domains, including prompt-based few-shot learning and models fine-tuned for down-stream tasks. Additionally, we show that UL2 excels in generation, language understanding, retrieval, long-text understanding and question answering tasks.</p><p style="text-align: justify;"><b>Retrieval Augmented Generation (RAG): </b></p><p style="text-align: justify;">Foundation models are usually trained offline, making the model agnostic to any data that is created after the model was trained. Additionally, foundation models are trained on very general domain corpora, making them less effective for domain-specific tasks. You can use Retrieval Augmented Generation (RAG) to retrieve data from outside a foundation model and augment your prompts by adding the relevant retrieved data in context. For more information about RAG model architectures, see Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.</p><p style="text-align: justify;">With RAG, the external data used to augment your prompts can come from multiple data sources, such as a document repositories, databases, or APIs. The first step is to convert your documents and any user queries into a compatible format to perform relevancy search. To make the formats compatible, a document collection, or knowledge library, and user-submitted queries are converted to numerical representations using embedding language models. Embedding is the process by which text is given numerical representation in a vector space. RAG model architectures compare the embeddings of user queries within the vector of the knowledge library. The original user prompt is then appended with relevant context from similar documents within the knowledge library. This augmented prompt is then sent to the foundation model. You can update knowledge libraries and their relevant embeddings asynchronously.</p><p style="text-align: justify;"><span style="font-size: x-small;">Pre-requisites: data sets </span></p><p style="text-align: justify;"><span style="font-size: x-small;">1) Training data set ( contains question and answer )</span></p><p style="text-align: justify;"><span style="font-size: x-small;">2) Test data set </span></p><p style="text-align: justify;"><span style="font-size: x-small;">embedding generation -----> storing it in vector data base ---> giving a user question ----> convert it into embedding --->sending it to vector database ---> getting an answer ---> Finally creating a prompt ---->sending it to foundation model Flan-UL2 (encoder-decoder Model) ---> getting an answer </span> </p></div><div>Be a value creator. You can build foundation models using watsonx Platform on your data and that will be under your control. It will become your most valuable asset. Don’t outsource that and don’t reduce your AI strategy to an API call. One model will not rule them all. Build responsibly, transparently and put governance into the heart of your AI lifecycle.</div><div><br /></div><div><span style="font-size: xx-small;">Reference:</span></div><div><span style="font-size: xx-small;">https://www.youtube.com/watch?v=FrDnPTPgEmk</span></div><div><span style="font-size: xx-small;">https://www.ibm.com/products/watsonx-ai</span></div><div><span style="font-size: xx-small;">https://www.ibm.com/products/watsonx-governance</span></div><div><span style="font-size: xx-small;">https://www.ibm.com/products/watsonx-data</span></div><div><span style="font-size: xx-small;"><br /></span></div>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-85000183758699832562023-04-19T17:50:00.011+05:302023-04-21T14:13:00.033+05:30Kubernetes - decommissioning a node from the cluster<p style="text-align: justify;"> Kubernetes cluster is a group of nodes that are used to run containerized applications and services. The cluster consists of a control plane, which manages the overall state of the cluster, and worker nodes, which run the containerized applications.</p><p style="text-align: justify;">The control plane is responsible for managing the configuration and deployment of applications on the cluster, as well as monitoring and scaling the cluster as needed. It includes components such as the Kubernetes API server, the etcd datastore, the kube-scheduler, and the kube-controller-manager.</p><p style="text-align: justify;">The worker nodes are responsible for running the containerized applications and services. Each node typically runs a container runtime, such as Docker or containerd, as well as a kubelet process that communicates with the control plane to manage the containers running on the node.</p><p style="text-align: justify;">In a Kubernetes cluster, applications are deployed as pods, which are the smallest deployable units in Kubernetes. Pods contain one or more containers, and each pod runs on a single node in the cluster. Kubernetes manages the deployment and scaling of the pods across the cluster, ensuring that the workload is evenly distributed and resources are utilized efficiently.</p><p style="text-align: justify;">In Kubernetes, the native scheduler is a built-in component responsible for scheduling pods onto worker nodes in the cluster. When a new pod is created, the scheduler evaluates the resource requirements of the pod, along with any constraints or preferences specified in the pod's definition, and selects a node in the cluster where the pod can be scheduled. The native scheduler uses a combination of heuristics and policies to determine the best node for each pod. It considers factors such as the available resources on each node, the affinity and anti-affinity requirements of the pod, any node selectors or taints on the nodes, and the current state of the cluster. The native scheduler in Kubernetes is highly configurable and can be customized to meet the specific needs of different workloads. For example, you can configure the scheduler to prioritize certain nodes in the cluster over others, or to balance the workload evenly across all available nodes.</p><p style="text-align: justify;"><span style="background-color: #cccccc;">[sachinpb@remotehostn18 ~]$ kubectl get pods -n kube-system | grep kube-scheduler<br />kube-scheduler-remotehost18 1/1 Running 11 398d</span></p><p><b>kubectl cordon</b> is a command in Kubernetes that is used to mark a node as unschedulable. This means that Kubernetes will no longer schedule any new pods on the node, but will continue to run any existing pods on the node.</p><p style="text-align: justify;">The kubectl cordon command is useful when you need to take a node offline for maintenance or other reasons, but you want to ensure that the existing pods on the node continue to run until they can be safely moved to other nodes in the cluster. By marking the node as unschedulable, you can prevent Kubernetes from scheduling any new pods on the node, which helps to ensure that the overall health and stability of the cluster is maintained.</p><span style="background-color: #cccccc;">[sachinpb@remotenode18 ~]$ kubectl get nodes<br />NAME STATUS ROLES AGE VERSION<br />remotenode01 Ready worker 270d v1.23.4<br />remotenode02 Ready worker 270d v1.23.4<br />remotenode03 Ready worker 270d v1.23.4<br />remotenode04 Ready worker 81d v1.23.4<br />remotenode07 Ready worker 389d v1.23.4<br />remotenode08 Ready worker 389d v1.23.4<br />remotenode09 Ready worker 389d v1.23.4<br />remotenode14 Ready worker 396d v1.23.4<br />remotenode15 Ready worker 81d v1.23.4<br />remotenode16 Ready worker 396d v1.23.4<br />remotenode17 Ready worker 396d v1.23.4<br />remotenode18 Ready control-plane,master 398d v1.23.4</span><div><span style="background-color: #cccccc;"><br />[sachinpb@remotenode18 ~]$ kubectl cordon remotenode16<br />node/remotenode16 cordoned<br />[sachinpb@remotenode18 ~]$ kubectl uncordon remotenode16<br />node/remotenode16 uncordoned</span></div><div><span style="background-color: #cccccc;"><br />[sachinpb@remotenode18 ~]$ kubectl cordon remotenode16<br />node/remotenode16 cordoned<br />[sachinpb@remotenode18 ~]$ kubectl get nodes<br />NAME STATUS ROLES AGE VERSION<br />remotenode01 Ready worker 270d v1.23.4<br />remotenode02 Ready worker 270d v1.23.4<br />remotenode03 Ready worker 270d v1.23.4<br />remotenode04 Ready worker<br />remotenode07 Ready worker 389d v1.23.4<br />remotenode08 Ready worker 389d v1.23.4<br />remotenode09 Ready worker 389d v1.23.4<br />remotenode14 Ready worker 396d v1.23.4<br />remotenode15 Ready worker 81d v1.23.4<br />remotenode16 Ready,</span><span style="background-color: #cccccc;">SchedulingDisabled</span><span style="background-color: #cccccc;"> worker 396d v1.23.4</span></div><div><span style="background-color: #cccccc;">remotenode17 Ready worker 396d v1.23.4<br />remotenode18 Ready control-plane,master 398d v1.23.4<br /></span></div><p><span style="background-color: #cccccc;">[sachinpb@remotenode18 ~]$</span> </p><p>After the node has been cordoned off, you can use the kubectl drain command to safely and gracefully terminate any running pods on the node and reschedule them onto other available nodes in the cluster. Once all the pods have been moved, the node can then be safely removed from the cluster.</p><p>kubectl drain is a command in Kubernetes that is used to gracefully remove a node from a cluster. This is typically used when performing maintenance on a node, such as upgrading or replacing hardware, or when decommissioning a node from the cluster.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhesegxzoUM7HGome1UZFUxTdl_cD4fDZw6LeQcz-Ncyfdy62xSKdFGufaZgmoq9ssZbGH5flrSf-mwrKnJKAFpmVyAJDTz5xgHcToVFKyAF5DObZYE2bXnM6666abT28AxTeqwZfPwBzxcGwxtLxYIlhBlXGUUPSB6hyrfGw-3t5fTvVQ-JN3bep_-Bg/s917/Kubectl_drain_flowchart.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="757" data-original-width="917" height="528" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhesegxzoUM7HGome1UZFUxTdl_cD4fDZw6LeQcz-Ncyfdy62xSKdFGufaZgmoq9ssZbGH5flrSf-mwrKnJKAFpmVyAJDTz5xgHcToVFKyAF5DObZYE2bXnM6666abT28AxTeqwZfPwBzxcGwxtLxYIlhBlXGUUPSB6hyrfGw-3t5fTvVQ-JN3bep_-Bg/w640-h528/Kubectl_drain_flowchart.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://kubernetes.io/images/docs/kubectl_drain.svg" target="_blank"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><br /><p style="text-align: left;"><span style="background-color: #cccccc;">[sachinpb@remotenode18 ~]$ kubectl drain --ignore-daemonsets remote16<br />node/remote16 already cordoned<br />WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-j749l, kube-system/fuse-device-plugin-daemonset-59lrp, kube-system/kube-proxy-v26k2, kube-system/nvidia-device-plugin-daemonset-w2k57, kube-system/rdma-shared-dp-ds-zdpfw, sys-monitor/prometheus-op-prometheus-node-exporter-rh4db<br />node/remote16 drained<br />[sachinpb@remotenode18 ~]$</span></p><p>By default kubectl drain is non-destructive, you have to override to change that behaviour. It runs with the following defaults:</p><p style="text-align: left;"><span style="background-color: #cccccc;"> --delete-local-data=false</span><br /><span style="background-color: #cccccc;"> --force=false</span><br /><span style="background-color: #cccccc;"> --grace-period=-1 <span style="font-size: xx-small;">(</span></span><span style="font-size: xx-small;">Period of time in seconds given to each pod to terminate gracefully. If negative, the default value specified in the pod will be used.<span style="background-color: #cccccc;">)</span></span><br /><span style="background-color: #cccccc;"> --ignore-daemonsets=false</span><br /><span style="background-color: #cccccc;"> --timeout=0s</span></p><p style="text-align: justify;">Each of these safeguard deals with a different category of potential destruction (local data, bare pods, graceful termination, daemonsets). It also respects pod disruption budgets to adhere to workload availability. Any non-bare pod will be recreated on a new node by its respective controller (e.g. daemonset controller, replication controller). It's up to you whether you want to override that behaviour (for example you might have a bare pod if running jenkins job. If you override by setting --force=true it will delete that pod and it won't be recreated). If you don't override it, the node will be in drain mode indefinitely (--timeout=0s))</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGVb84NGCKCnhxczjWTJUgYZvx7s5oi6tykIj-SG1AueJr0OtUeD_S7e3Db7zw3A4ESCrqbUob_LgJBoSQQH2yvST5GyQxoq_bJhL92RTRckGXv1hjcGIoTqHplKjQyJLA74h_hBt2PLw1RhJ4GPZFP0NOhui58V60HnAom6p98JOhMwGjBOPAzixPJw/s851/Example-drainb.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="513" data-original-width="851" height="386" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGVb84NGCKCnhxczjWTJUgYZvx7s5oi6tykIj-SG1AueJr0OtUeD_S7e3Db7zw3A4ESCrqbUob_LgJBoSQQH2yvST5GyQxoq_bJhL92RTRckGXv1hjcGIoTqHplKjQyJLA74h_hBt2PLw1RhJ4GPZFP0NOhui58V60HnAom6p98JOhMwGjBOPAzixPJw/w640-h386/Example-drainb.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://man.archlinux.org/man/community/kubectl/kubectl-drain.1.en" target="_blank"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><p><span style="background-color: #cccccc;"></span></p><p style="text-align: justify;">When a node is drained, Kubernetes will automatically reschedule any running pods onto other available nodes in the cluster, ensuring that the workload is not interrupted. The kubectl drain command ensures that the node is cordoned off, meaning no new pods will be scheduled on it, and then gracefully terminates any running pods on the node. This helps to ensure that the pods are shut down cleanly, allowing them to complete any in-progress tasks and save any data before they are terminated.</p><p style="text-align: justify;">After the pods have been rescheduled, the node can then be safely removed from the cluster. This helps to ensure that the overall health and stability of the cluster is maintained, even when individual nodes need to be taken offline for maintenance or other reasons</p><p style="text-align: justify;">When kubectl drain returns successfully, that indicates that all of the pods have been safely evicted. It is then safe to bring down the node. After maintenance work we can use kubectl uncordon to tell Kubernetes that it can resume scheduling new pods onto the node.</p><p style="text-align: justify;"><span style="background-color: #cccccc; text-align: left;">[sachinpb@remotenode18 ~]$ kubectl uncordon remotenode16</span><br style="text-align: left;" /><span style="background-color: #cccccc; text-align: left;">node/remotenode16 uncordoned</span></p><p style="text-align: left;"><span style="background-color: white;">Let's try all the above steps and see :</span></p><p>1) Retrieve information from a Kubernetes cluster</p><p style="text-align: left;"><span style="background-color: #eeeeee;">[sachinpb@remotenode18 ~]$ kubectl get nodes<br />NAME STATUS ROLES AGE VERSION<br />remotenode01 Ready worker 270d v1.23.4<br />remotenode02 Ready worker 270d v1.23.4<br />remotenode03 Ready worker 270d v1.23.4<br />remotenode04 Ready worker 81d v1.23.4<br />remotenode07 Ready worker 389d v1.23.4<br />remotenode08 Ready worker 389d v1.23.4<br />remotenode09 Ready worker 389d v1.23.4<br />remotenode14 Ready worker 396d v1.23.4<br />remotenode15 Ready worker 81d v1.23.4<br />remotenode16 Ready worker 396d v1.23.4<br />remotenode17 Ready worker 396d v1.23.4<br />remotenode18 Ready control-plane,master 398d v1.23.4</span></p><p>--------------------------------</p><p>2) Kubernetes cordon is an operation that marks or taints a node in your existing node pool as unschedulable.</p><p style="text-align: left;"><span style="background-color: #eeeeee;">[sachinpb@remotenode18 ~]$ kubectl cordon remotenode16<br />node/remotenode16 cordoned<br />[sachinpb@remotenode18 ~]$</span></p><p style="text-align: left;"><span style="background-color: #cccccc;">[sachinpb@remotenode18 ~]$ kubectl get nodes<br />NAME STATUS ROLES AGE VERSION<br />remotenode01 Ready worker 270d v1.23.4<br />remotenode02 Ready worker 270d v1.23.4<br />remotenode03 Ready worker 270d v1.23.4<br />remotenode04 Ready worker 81d v1.23.4<br />remotenode07 Ready worker 389d v1.23.4<br />remotenode08 Ready worker 389d v1.23.4<br />remotenode09 Ready worker 389d v1.23.4<br />remotenode14 Ready worker 396d v1.23.4<br />remotenode15 Ready worker 81d v1.23.4<br />remotenode16 Ready,SchedulingDisabled worker 396d v1.23.4<br />remotenode17 Ready worker 396d v1.23.4<br />remotenode18 Ready control-plane,master 398d v1.23.4</span></p><p>3) Drain node in preparation for maintenance. The given node will be marked unschedulable to prevent new pods from arriving. Then drain deletes all pods</p><p><br /></p><p style="text-align: left;"><span style="background-color: #cccccc;">[sachinpb@remotenode18 ~]$ kubectl drain remotenode16 --grace-period=2400<br />node/remotenode16 already cordoned<br />error: unable to drain node "remotenode16" due to error:cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/calico-node-j749l, kube-system/fuse-device-plugin-daemonset-59lrp, kube-system/kube-proxy-v26k2, kube-system/nvidia-device-plugin-daemonset-w2k57, kube-system/rdma-shared-dp-ds-zdpfw, sys-monitor/prometheus-op-prometheus-node-exporter-rh4db, continuing command...<br />There are pending nodes to be drained:<br /> remotenode16<br />cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/calico-node-j749l, kube-system/fuse-device-plugin-daemonset-59lrp, kube-system/kube-proxy-v26k2, kube-system/nvidia-device-plugin-daemonset-w2k57, kube-system/rdma-shared-dp-ds-zdpfw, sys-monitor/prometheus-op-prometheus-node-exporter-rh4db<br />[sachinpb@remotenode18 ~]$</span></p><p>NOTE:</p><p style="text-align: justify;">The given node will be marked unschedulable to prevent new pods from arriving. Then drain deletes all pods except mirror pods (which cannot be deleted through the API server). If there are DaemonSet-managed pods, drain will not proceed without –ignore-daemonsets, and regardless it will not delete any DaemonSet-managed pods, because those pods would be immediately replaced by the DaemonSet controller, which ignores unschedulable markings. If there are any pods that are neither mirror pods nor managed–by ReplicationController, DaemonSet or Job–, then drain will not delete any pods unless you use –force.</p><p>----------------------------</p><p>4) Drain node with --ignore-daemonsets </p><p style="text-align: left;"><span style="background-color: #cccccc;">[sachinpb@remotenode18 ~]$ kubectl drain --ignore-daemonsets remotenode16 --grace-period=2400<br />node/remotenode16 cordoned<br />WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-j749l, kube-system/fuse-device-plugin-daemonset-59lrp, kube-system/kube-proxy-v26k2, kube-system/nvidia-device-plugin-daemonset-w2k57, kube-system/rdma-shared-dp-ds-zdpfw, sys-monitor/prometheus-op-prometheus-node-exporter-rh4db<br />node/remotenode16 drained</span></p><p>----------------------</p><p>5) Uncordon will mark the node as schedulable.</p><p style="text-align: left;"><span style="background-color: #cccccc;">[sachinpb@remotenode18 ~]$ kubectl uncordon remotenode16<br />node/remotenode16 uncordoned<br />[sachinpb@remotenode18 ~]$</span></p><p>-----------------</p><p>6) Retrieve information from a Kubernetes cluster</p><p style="text-align: left;"><span style="background-color: #cccccc;">[sachinpb@remotenode18 ~]$ kubectl get nodes<br />NAME STATUS ROLES AGE VERSION<br />remotenode01 Ready worker 270d v1.23.4<br />remotenode02 Ready worker 270d v1.23.4<br />remotenode03 Ready worker 270d v1.23.4<br />remotenode04 Ready worker 81d v1.23.4<br />remotenode07 Ready worker 389d v1.23.4<br />remotenode08 Ready worker 389d v1.23.4<br />remotenode09 Ready worker 389d v1.23.4<br />remotenode14 Ready worker 396d v1.23.4<br />remotenode15 Ready worker 81d v1.23.4<br />remotenode16 Ready worker 396d v1.23.4<br />remotenode17 Ready worker 396d v1.23.4<br />remotenode18 Ready control-plane,master 398d v1.23.4</span></p><p style="text-align: left;"><span><b style="background-color: white;">How to automate above process creating Jenkins pipeline job to cordon ,drain and uncordon the nodes with the help of groovy script</b><span style="background-color: white;">:</span></span></p><p style="text-align: left;"><span style="background-color: white;">-------------------------<span style="font-size: medium;">Sample groovy script</span>--------------------------------</span></p><p style="text-align: left;"></p><blockquote><blockquote><span style="font-size: x-small;"><b>node("Kubernetes-master-node") {<br /> stage("1") {<br /> sh 'hostname'<br /> sh 'cat $SACHIN_HOME/manual//hostfile'<br /> k8s_cordon_drain()<br /> k8s_uncordon() <br /> } <br />}<br /><br />/*<br />* CI -Kubernetes cluster : This function will cordon/drain the worker nodes in hostfile <br /><br />*/<br />def k8s_cordon_drain() {<br /> def maxTries = 3 // the maximum number of times to retry the kubectl commands<br /> def sleepTime = 5 * 1000 // the amount of time to wait between retries (in milliseconds)<br /> def filename = '$SACHIN_HOME/manual/hostfile'<br /> def content = readFile(filename)<br /> def hosts = content.readLines().collect { it.split()[0] }<br /> println "List of Hostnames to be cordoned from K8s cluster: ${hosts}"<br /> hosts.each { host -><br /> def command1 = "kubectl cordon $host"<br /> def command2 = "kubectl drain --ignore-daemonsets --grace-period=2400 $host"<br /> def tries = 0<br /> def result1 = null<br /> def result2 = null<br /> while (tries < maxTries) {<br /> result1 = sh(script: command1, returnStatus: true)<br /> if (result1 == 0) {<br /> println "Successfully cordoned $host"<br /> break<br /> } else {<br /> tries++<br /> println "Failed to cordoned $host (attempt $tries/$maxTries), retrying in ${sleepTime/1000} seconds..."<br /> sleep(sleepTime)<br /> }<br /> }<br /> if (result1 == 0) {<br /> tries = 0<br /> while (tries < maxTries) {<br /> result2 = sh(script: command2, returnStatus: true)<br /> if (result2 == 0) {<br /> println "Successfully drained $host"<br /> break<br /> } else {<br /> tries++<br /> println "Failed to drain $host (attempt $tries/$maxTries), retrying in ${sleepTime/1000} seconds..."<br /> sleep(sleepTime)<br /> }<br /> }<br /> }<br /><br /> if (result2 != 0) {<br /> println "Failed to drain $host after $maxTries attempts"<br /> }<br /> }<br />}<br /><br />/*<br />* CI - Kubernetes cluster : This function will uncordon the worker nodes in hostfile <br /><br />*/<br />def k8s_uncordon() {<br /> def maxTries = 3 // the maximum number of times to retry the kubectl commands<br /> def sleepTime = 5 * 1000 // the amount of time to wait between retries (in milliseconds)<br /> def filename = '$SACHIN_HOME/manual/hostfile'<br /> def content = readFile(filename)<br /> def hosts = content.readLines().collect { it.split()[0] }<br /> println "List of Hostnames to be uncordoned from K8s cluster: ${hosts}"<br /> hosts.each { host -><br /> def command1 = "kubectl uncordon $host"<br /> def tries = 0<br /> def result1 = null<br /> while (tries < maxTries) {<br /> result1 = sh(script: command1, returnStatus: true)<br /> if (result1 == 0) {<br /> println "Successfully cordoned $host"<br /> break<br /> } else {<br /> tries++<br /> println "Failed to uncordon $host (attempt $tries/$maxTries), retrying in ${sleepTime/1000} seconds..."<br /> sleep(sleepTime)<br /> }<br /> }<br /> if (result1 != 0) {<br /> println "Failed to uncordon $host after $maxTries attempts"<br /> }<br /> }<br />}</b></span></blockquote></blockquote><p></p><p><span style="background-color: white;"></span></p><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><ol style="text-align: left;"></ol><p></p><p><span style="background-color: white;"></span></p><p></p><p style="text-align: left;">------------------Jenkins Console output for pipeline job -----------------</p><p style="text-align: left;"><b style="font-size: x-small;">Started by user jenkins-admin</b><br /><b style="font-size: x-small;">[Pipeline] Start of Pipeline</b><br /><b style="font-size: x-small;">[Pipeline] node</b><br /><b style="font-size: x-small;">Running on Kubernetes-master-node in $SACHIN_HOME/workspace/test_sample4_cordon_drain</b><br /><b style="font-size: x-small;">[Pipeline] {</b><br /><b style="font-size: x-small;">[Pipeline] stage</b><br /><b style="font-size: x-small;">[Pipeline] { (1)</b><br /><b style="font-size: x-small;">[Pipeline] sh</b><br /><b style="font-size: x-small;">+ hostname</b><br /><span style="font-size: xx-small;"><b>kubernetes-master-node</b></span><br /><b style="font-size: x-small;">[Pipeline] sh</b><br /><b style="font-size: x-small;">+ cat $SACHIN_HOME/manual//hostfile</b><br /><b style="font-size: x-small;">Remotenode16 slots=4</b><br /><b style="font-size: x-small;">Remotenode17 slots=4</b><br /><b style="font-size: x-small;">[Pipeline] readFile</b><br /><b style="font-size: x-small;">[Pipeline] echo</b><br /><b style="font-size: x-small;">List of Hostnames to be cordoned from K8s cluster: [Remotenode16, Remotenode17]</b><br /><b style="font-size: x-small;">[Pipeline] sh</b><br /><b style="font-size: x-small;">+ kubectl cordon Remotenode16</b><br /><b style="font-size: x-small;">node/Remotenode16 cordoned</b><br /><b style="font-size: x-small;">[Pipeline] echo</b><br /><b style="font-size: x-small;">Successfully cordoned Remotenode16</b><br /><b style="font-size: x-small;">[Pipeline] sh</b><br /><b style="font-size: x-small;">+ kubectl drain --ignore-daemonsets --grace-period=2400 Remotenode16</b><br /><b style="font-size: x-small;">node/Remotenode16 already cordoned</b><br /><b style="font-size: x-small;">WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-j749l, kube-system/fuse-device-plugin-daemonset-59lrp, kube-system/kube-proxy-v26k2, kube-system/nvidia-device-plugin-daemonset-w2k57, kube-system/rdma-shared-dp-ds-zdpfw, sys-monitor/prometheus-op-prometheus-node-exporter-rh4db</b><br /><b style="font-size: x-small;">node/Remotenode16 drained</b><br /><b style="font-size: x-small;">[Pipeline] echo</b><br /><b style="font-size: x-small;">Successfully drained Remotenode16</b><br /><b style="font-size: x-small;">[Pipeline] sh</b><br /><b style="font-size: x-small;">+ kubectl cordon Remotenode17</b><br /><b style="font-size: x-small;">node/Remotenode17 cordoned</b><br /><b style="font-size: x-small;">[Pipeline] echo</b><br /><b style="font-size: x-small;">Successfully cordoned Remotenode17</b><br /><b style="font-size: x-small;">[Pipeline] sh</b><br /><b style="font-size: x-small;">+ kubectl drain --ignore-daemonsets --grace-period=2400 Remotenode17</b><br /><b style="font-size: x-small;">node/Remotenode17 already cordoned</b><br /><b style="font-size: x-small;">WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-hz5zh, kube-system/fuse-device-plugin-daemonset-dj72m, kube-system/kube-proxy-g87dc, kube-system/nvidia-device-plugin-daemonset-tk5x8, kube-system/rdma-shared-dp-ds-n4g5w, sys-monitor/prometheus-op-prometheus-node-exporter-gczmz</b><br /><b style="font-size: x-small;">node/Remotenode17 drained</b><br /><b style="font-size: x-small;">[Pipeline] echo</b><br /><b style="font-size: x-small;">Successfully drained Remotenode17</b><br /><b style="font-size: x-small;">[Pipeline] readFile</b><br /><b style="font-size: x-small;">[Pipeline] echo</b><br /><b style="font-size: x-small;">List of Hostnames to be uncordoned from K8s cluster: [Remotenode16, Remotenode17]</b><br /><b style="font-size: x-small;">[Pipeline] sh</b><br /><b style="font-size: x-small;">+ kubectl uncordon Remotenode16</b><br /><b style="font-size: x-small;">node/Remotenode16 uncordoned</b><br /><b style="font-size: x-small;">[Pipeline] echo</b><br /><b style="font-size: x-small;">Successfully cordoned Remotenode16</b><br /><b style="font-size: x-small;">[Pipeline] sh</b><br /><b style="font-size: x-small;">+ kubectl uncordon Remotenode17</b><br /><b style="font-size: x-small;">node/Remotenode17 uncordoned</b><br /><b style="font-size: x-small;">[Pipeline] echo</b><br /><b style="font-size: x-small;">Successfully cordoned Remotenode17</b><br /><b style="font-size: x-small;">[Pipeline] }</b><br /><b style="font-size: x-small;">[Pipeline] // stage</b><br /><b style="font-size: x-small;">[Pipeline] }</b><br /><b style="font-size: x-small;">[Pipeline] // node</b><br /><b style="font-size: x-small;">[Pipeline] End of Pipeline</b><br /><b style="font-size: x-small;">Finished: SUCCESS</b></p><p><span style="background-color: white;"></span></p><p><span style="background-color: white;"></span></p><p><b><span style="font-size: x-small;">-----------------------------------------------------------------</span></b></p><p><b><span style="font-size: x-small;">Reference:</span></b></p><p><a href="https://kubernetes.io/docs/home/"><span style="font-size: xx-small;">https://kubernetes.io/docs/home/</span></a></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-21654828773004152792023-04-13T17:14:00.012+05:302023-04-29T23:30:15.195+05:30IBM Spectrum Symphony and LSF with Apache Hadoop<p style="text-align: justify;">IBM Spectrum Symphony (formerly known as IBM Platform Symphony) is a high-performance computing (HPC) and grid computing software platform that enables organizations to process large amounts of data and run compute-intensive applications at scale. It provides a distributed computing infrastructure that can be used for a wide range of data-intensive workloads, such as scientific simulations, financial modeling, and big data analytics. IBM Spectrum Symphony is a parallel services middleware and cluster manager. It is widely used in banks for risk analytics, data analytics in a shared, multi-user, multi-application, multi-job environment. IBM Spectrum Symphony also works with IBM Spectrum LSF (for batch workloads) in the same cluster to allow both batch and parallel services workloads to share the same cluster.</p><p style="text-align: justify;">Some of the key features of IBM Spectrum Symphony include:</p><p style="text-align: justify;"></p><ol><li>Distributed computing: The platform allows organizations to distribute computing workloads across a large number of nodes, which can be located in different data centers or cloud environments.</li><li>Resource management: IBM Spectrum Symphony provides a resource management framework that allows organizations to allocate and manage compute, storage, and network resources more efficiently.</li><li>High availability: The platform is designed to provide high availability and fault tolerance, ensuring that applications can continue to run even if individual nodes or components fail.</li><li>Performance optimization: IBM Spectrum Symphony includes a range of performance optimization features, such as load balancing and data caching, which can help organizations to achieve faster processing times and better overall performance.</li><li>Support for multiple programming languages: The platform supports a wide range of programming languages, including Java, Python, and C++, which makes it easy for developers to build and deploy applications on the platform.</li></ol><p></p><div style="text-align: justify;">IBM Spectrum LSF (Load Sharing Facility) is another software platform that is often used in conjunction with IBM Spectrum Symphony to manage and optimize workloads in a distributed computing environment. LSF provides a range of features for resource management, workload scheduling, and job prioritization, which can help organizations to improve performance and efficiency.</div><div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">When used together, IBM Spectrum Symphony and IBM Spectrum LSF can provide a comprehensive solution for managing and optimizing large-scale distributed computing environments. IBM Spectrum Symphony provides the distributed computing infrastructure and application management capabilities, while IBM Spectrum LSF provides the workload management and optimization features.</div></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div>Some of the key features of LSF that complement IBM Spectrum Symphony include:</div><div><ol><li>Advanced job scheduling: LSF provides sophisticated job scheduling capabilities, allowing organizations to prioritize and schedule jobs based on a wide range of criteria, such as resource availability, job dependencies, and user priorities.</li><li>Resource allocation: LSF can manage the allocation of resources, ensuring that jobs are run on the most appropriate nodes and that resources are used efficiently.</li><li>Job monitoring: LSF provides real-time monitoring of job progress and resource usage, allowing organizations to quickly identify and resolve issues that may impact performance.</li><li>Integration with other tools: LSF can be integrated with a wide range of other HPC tools and applications, including IBM Spectrum Symphony, providing a seamless workflow for managing complex computing workloads.</li></ol><div><div>Integrating LSF with Hadoop can help organizations to optimize the use of their resources and achieve better performance when running Hadoop workloads. <div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGlODtHlSDBHNPT0XUXKUnp8PBx4hm-tVFOzNhS6hedXcFq-0TZLaHEpFjKloPU9RkyFifUVuadd0fH0Z1sooLXB2u8BtkJhKM9PdBcQjeLbpllYOcON6yaWiUqH9apGI4HE0GLo4OWKpBpfadIUibOZGkw-cK-U1E8nHDm5_hvhSjIYnq84wUkwKjsg/s567/spectrum_LSF.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="167" data-original-width="567" height="94" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGlODtHlSDBHNPT0XUXKUnp8PBx4hm-tVFOzNhS6hedXcFq-0TZLaHEpFjKloPU9RkyFifUVuadd0fH0Z1sooLXB2u8BtkJhKM9PdBcQjeLbpllYOcON6yaWiUqH9apGI4HE0GLo4OWKpBpfadIUibOZGkw-cK-U1E8nHDm5_hvhSjIYnq84wUkwKjsg/w318-h94/spectrum_LSF.png" width="318" /></a><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5cWphiGmWydCMzfJtIylMUfljpYlLTLiMqOuGoWccTNL0XZIFTIsoI25RqTodRqC53BnRlPMI_Rw8oWaapaeQqfkgYlD_FKImJbU1ppzvTTaMqd6zrYLfNTFpMgWT4sEf9RMVx0z98xJ3HhqLPm1fxsVTY84pCDkI5Xmvg7qIZojAxHRbD2PNAHRihw/s2000/hadoopelephant_rgb.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1473" data-original-width="2000" height="236" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5cWphiGmWydCMzfJtIylMUfljpYlLTLiMqOuGoWccTNL0XZIFTIsoI25RqTodRqC53BnRlPMI_Rw8oWaapaeQqfkgYlD_FKImJbU1ppzvTTaMqd6zrYLfNTFpMgWT4sEf9RMVx0z98xJ3HhqLPm1fxsVTY84pCDkI5Xmvg7qIZojAxHRbD2PNAHRihw/w325-h236/hadoopelephant_rgb.png" width="325" /></a></div></div><div><div>Apache Hadoop ("Hadoop") is a framework for large-scale distributed data storage and processing on computer clusters that uses the Hadoop Distributed File System ("HDFS") for the data storage and MapReduce programming model for the data processing. Since MapReduce workloads might only represent a small fraction of overall workload, but typically requires their own standalone environment, MapReduce is difficult to support within traditional HPC clusters. However, HPC clusters typically use parallel file systems that are sufficient for initial MapReduce workloads, so you can run MapReduce workloads as regular parallel jobs running in an HPC cluster environment. Use the IBM Spectrum LSF integration with Apache Hadoop to submit Hadoop MapReduce workloads as regular LSF parallel jobs.</div><div>To run your Hadoop application through LSF, submit it as an LSF job. Once the LSF job starts to run, the Hadoop connector script (lsfhadoop.sh) automatically provisions an open source Hadoop cluster within LSF allocated resources, then submits actual MapReduce workloads into this Hadoop cluster. Since each LSF Hadoop job has its own resource (cluster), the integration provides a multi-tenancy environment to allow multiple users to share the common pool of HPC cluster resources. LSF is able to collect resource usage of MapReduce workloads as normal LSF parallel jobs and has full control of the job life cycle. After the job is complete, LSF shuts down the Hadoop cluster.</div><div><br /></div><div>By default, the Apache Hadoop integration configures the Hadoop cluster with direct access to shared file systems and does not require HDFS. This allows you to use existing file systems in your HPC cluster without having to immediately invest in a new file system. Through the existing shared file system, data can be stored in common share locations, which avoids the typical data stage-in and stage-out steps with HDFS.</div></div><div><br /></div></div><div>The general steps to integrate LSF with Hadoop:</div><div><ol><li>Install and configure LSF: The first step is to install and configure LSF on the Hadoop cluster. This involves setting up LSF daemons on the cluster nodes and configuring LSF to work with the Hadoop Distributed File System (HDFS).</li><li>Configure Hadoop for LSF: Hadoop needs to be configured to use LSF as its resource manager. This involves setting the yarn.resourcemanager.scheduler.class property in the Hadoop configuration file to com.ibm.platform.lsf.yarn.LSFYarnScheduler.</li><li>Configure LSF for Hadoop: LSF needs to be configured to work with Hadoop by setting up the necessary environment variables and resource limits. This includes setting the LSF_SERVERDIR and LSF_LIBDIR environment variables to the LSF installation directory and configuring LSF resource limits to ensure that Hadoop jobs have access to the necessary resources.</li><li>Submit Hadoop jobs to LSF: Hadoop jobs can be submitted to LSF using the yarn command-line tool with the -Dmapreduce.job.submithostname and -Dmapreduce.job.queuename options set to the LSF submit host and queue, respectively.</li><li>Monitor Hadoop jobs in LSF: LSF provides a web-based user interface and command-line tools for monitoring and managing Hadoop jobs running on the cluster. This allows users to monitor job progress, resource usage, and other metrics, and to take corrective action if necessary.</li></ol></div></div></div><div><div><br /></div></div><div><div><b>LSF can be used as a standalone workload management software for Hadoop clusters, without the need for IBM Spectrum Symphony. </b>LSF provides advanced job scheduling and resource management capabilities, which can be used to manage and optimize Hadoop workloads running on large HPC clusters. By integrating LSF with Hadoop, organizations can ensure that Hadoop jobs have access to the necessary resources and are scheduled and managed efficiently, improving overall performance and resource utilization.</div><div><br /></div><div>In addition, IBM Spectrum Symphony provides additional capabilities beyond workload management, such as distributed computing infrastructure, data movement, and integration with other data center software. If an organization requires these additional capabilities, they may choose to use IBM Spectrum Symphony alongside LSF for even greater benefits. But LSF can be used independently as a workload manager for Hadoop clusters.</div></div><div><div><br /></div><div>Submitting LSF jobs to a Hadoop cluster involves creating an LSF job script that launches the Hadoop job and then submitting the job to LSF using the bsub command. . LSF will then schedule the job to run on the cluster. To submit LSF jobs to a Hadoop cluster, you need to follow these general steps:</div></div><div><div><ol><li>Write the Hadoop job: First, you need to write the Hadoop job that you want to run on the cluster. This can be done using any of the Hadoop APIs, such as MapReduce, Spark, or Hive.</li><li>Create the LSF job script: Next, you need to create an LSF job script that will launch the Hadoop job on the cluster. This script will typically include the Hadoop command to run the job, along with any necessary environment variables, resource requirements, and other LSF-specific settings.</li><li>Submit the LSF job: Once the job script is ready, you can submit it to LSF using the bsub command. This will add the job to the LSF queue and wait for available resources to run the job.</li><li>Monitor the job: LSF provides several tools for monitoring and managing jobs running on the cluster, such as the bjobs command and the LSF web interface. You can use </li></ol></div></div><div><div><b>Example 1: bsub command that can be used to submit a Hadoop job to an LSF-managed Hadoop cluster:</b></div><div><br /></div><div><span style="background-color: #cccccc;">bsub -J my_hadoop_job -oo my_hadoop_job.out -eo my_hadoop_job.err -R "rusage[mem=4096]" -q hadoop_queue hadoop jar my_hadoop_job.jar input_dir output_dir</span></div><div><br /></div><div>where:</div><div><span style="font-size: x-small;">-J: Specifies a name for the job. In this case, we're using "my_hadoop_job" as the job name.</span></div><div><span style="font-size: x-small;"><br /></span></div><div><span style="font-size: x-small;">-oo: Redirects the standard output of the job to a file. In this case, we're using "my_hadoop_job.out" as the output file.</span></div><div><span style="font-size: x-small;"><br /></span></div><div><span style="font-size: x-small;">-eo: Redirects the standard error of the job to a file. In this case, we're using "my_hadoop_job.err" as the error file.</span></div><div><span style="font-size: x-small;"><br /></span></div><div><span style="font-size: x-small;">-R: Specifies resource requirements for the job. In this case, we're requesting 4 GB of memory (mem=4096) for the job.</span></div><div><span style="font-size: x-small;"><br /></span></div><div><span style="font-size: x-small;">-q: Specifies the LSF queue to submit the job to. In this case, we're using the "hadoop_queue" LSF queue.</span></div><div><br /></div><div>After the bsub command options, we specify the Hadoop command to run the job (hadoop jar my_hadoop_job.jar) and the input and output directories for the job (input_dir and output_dir). This will submit the Hadoop job to LSF, which will then schedule and manage the job on the Hadoop cluster. For more details please refer these links.</div></div><div><br /></div><div><div><b>Example 2: How to submit a Hadoop job using bsub command with LSF? </b></div><div><br /></div><div><span style="background-color: #eeeeee;">bsub -q hadoop -J "Hadoop Job" -n 10 -o hadoop.log -hadoop /path/to/hadoop/bin/hadoop jar /path/to/hadoop/examples.jar pi 10 1000</span></div><div><br /></div><div>This command will submit a Hadoop job to the LSF scheduler and allocate resources as necessary based on the job's requirements.</div><div><br /></div><div>where:</div><div><span style="font-size: x-small;">-q hadoop specifies that the job should be submitted to the Hadoop queue.<br />-J "Hadoop Job" specifies a name for the job.<br />-n 10 specifies the number of cores to use for the job.<br />-o hadoop.log specifies the name of the output log file.<br />-hadoop specifies that the command that follows should be executed on a Hadoop cluster.<br />/path/to/hadoop/bin/hadoop specifies the path to the Hadoop executable.<br />jar /path/to/hadoop/examples.jar pi 10 1000 specifies the command to run the Hadoop job, which in this case is the pi example program with 10 mappers and 1000 samples.</span></div></div><div><br /></div><div><b>Example 3: How to submit a wordcount MapReduce job using bsub with LSF ?</b></div><div><br /></div><div><span style="background-color: #cccccc;">bsub -q hadoop -J "MapReduce Job" -n 10 -o mapreduce.log -hadoop /path/to/hadoop/bin/hadoop jar /path/to/hadoop/examples.jar wordcount /input/data /output/data</span></div><div><br /></div><div>where:</div><div><span style="font-size: x-small;">-q hadoop specifies that the job should be submitted to the Hadoop queue.</span></div><div><span style="font-size: x-small;">-J "MapReduce Job" specifies a name for the job.</span></div><div><span style="font-size: x-small;">-n 10 specifies the number of cores to use for the job.</span></div><div><span style="font-size: x-small;">-o mapreduce.log specifies the name of the output log file.</span></div><div><span style="font-size: x-small;">-hadoop specifies that the command that follows should be executed on a Hadoop cluster.</span></div><div><span style="font-size: x-small;">/path/to/hadoop/bin/hadoop specifies the path to the Hadoop executable.</span></div><div><span style="font-size: x-small;">jar /path/to/hadoop/examples.jar wordcount /input/data /output/data specifies the command to run the MapReduce job, which in this case is the wordcount example program with input data in /input/data and output data in /output/data.</span></div><div><br /></div><div><div><div><b>Example 4: How to submit a terasort MapReduce job using bsub with LSF?</b></div><div><br /></div><div><span style="background-color: #cccccc;">bsub -q hadoop -J "MapReduce Job" -n 20 -o mapreduce.log -hadoop /path/to/hadoop/bin/hadoop jar /path/to/hadoop/examples.jar terasort -Dmapred.map.tasks=100 -Dmapred.reduce.tasks=50 /input/data /output/data</span></div><div><span style="font-size: x-small;">where:</span></div><div><span style="font-size: x-small;">-q hadoop specifies that the job should be submitted to the Hadoop queue.</span></div><div><span style="font-size: x-small;">-J "MapReduce Job" specifies a name for the job.</span></div><div><span style="font-size: x-small;">-n 20 specifies the number of cores to use for the job.</span></div><div><span style="font-size: x-small;">-o mapreduce.log specifies the name of the output log file.</span></div><div><span style="font-size: x-small;">-hadoop specifies that the command that follows should be executed on a Hadoop cluster.</span></div><div><span style="font-size: x-small;">/path/to/hadoop/bin/hadoop specifies the path to the Hadoop executable.</span></div><div><span style="font-size: x-small;">jar /path/to/hadoop/examples.jar terasort -Dmapred.map.tasks=100 -Dmapred.reduce.tasks=50 /input/data /output/data specifies the command to run the MapReduce job, which in this case is the terasort example program with input data in /input/data and output data in /output/data, and specific configuration parameters to control the number of map and reduce tasks.</span></div></div><div><br /></div><div><div><b>Example 5: How to submit a grep MapReduce job using bsub with LSF?</b></div><div><br /></div><div><span style="background-color: #cccccc;">bsub -q hadoop -J "MapReduce Job" -n 30 -o mapreduce.log -hadoop /path/to/hadoop/bin/hadoop jar /path/to/hadoop/examples.jar grep -input /input/data -output /output/data -regex "example.*"</span></div><div><div><span style="font-size: x-small;">where:</span></div><div><span style="font-size: x-small;">-q hadoop specifies that the job should be submitted to the Hadoop queue.</span></div><div><span style="font-size: x-small;">-J "MapReduce Job" specifies a name for the job.</span></div><div><span style="font-size: x-small;">-n 30 specifies the number of cores to use for the job.</span></div><div><span style="font-size: x-small;">-o mapreduce.log specifies the name of the output log file.</span></div><div><span style="font-size: x-small;">-hadoop specifies that the command that follows should be executed on a Hadoop cluster.</span></div><div><span style="font-size: x-small;">/path/to/hadoop/bin/hadoop specifies the path to the Hadoop executable.</span></div><div><span style="font-size: x-small;">jar /path/to/hadoop/examples.jar grep -input /input/data -output /output/data -regex "example.*" specifies the command to run the MapReduce job, which in this case is the grep example program with input data in /input/data, output data in /output/data, and a regular expression pattern to search for.</span></div></div></div><div><br /></div><div><div><b>Example 6: How to submit a non MapReduce hadoop job using bsub with LSF?</b></div><div><b style="background-color: #cccccc;"><br /></b></div><div><span style="background-color: #cccccc;">bsub -q hadoop -J "Hadoop Job" -n 10 -o hadoopjob.log -hadoop /path/to/hadoop/bin/hadoop fs -rm -r /path/to/hdfs/directory</span></div><div><br /></div><div><span style="font-size: x-small;">where:</span></div><div><span style="font-size: x-small;">-q hadoop specifies that the job should be submitted to the Hadoop queue.</span></div><div><span style="font-size: x-small;">-J "Hadoop Job" specifies a name for the job.</span></div><div><span style="font-size: x-small;">-n 10 specifies the number of cores to use for the job.</span></div><div><span style="font-size: x-small;">-o hadoopjob.log specifies the name of the output log file.</span></div><div><span style="font-size: x-small;">-hadoop specifies that the command that follows should be executed on a Hadoop cluster.</span></div><div><span style="font-size: x-small;">/path/to/hadoop/bin/hadoop fs -rm -r /path/to/hdfs/directory specifies the command to run the Hadoop job, which in this case is to remove a directory in HDFS at /path/to/hdfs/directory.</span></div><div><span style="font-size: x-small;">This command will submit a non-MapReduce Hadoop job to the LSF scheduler and allocate resources as necessary based on the job's requirement</span></div></div><div><br /></div><div><div><br /></div><div><b>Example 7: If you have a Hadoop cluster with YARN and Spark installed, you can submit Spark jobs to the cluster using bsub as shown in the example.</b></div><div><br /></div><div><span style="background-color: #cccccc;">bsub -q normal -J "Spark Job" -n 20 -o sparkjob.log /path/to/spark/bin/spark-submit --class com.example.MyApp --master yarn --deploy-mode cluster /path/to/my/app.jar arg1 arg2</span></div><div><span style="font-size: x-small;">where:</span></div><div><span style="font-size: x-small;">-q normal specifies that the job should be submitted to the normal queue.</span></div><div><span style="font-size: x-small;">-J "Spark Job" specifies a name for the job.</span></div><div><span style="font-size: x-small;">-n 20 specifies the number of cores to use for the job.</span></div><div><span style="font-size: x-small;">-o sparkjob.log specifies the name of the output log file.</span></div><div><span style="font-size: x-small;">/path/to/spark/bin/spark-submit specifies the path to the spark-submit script.</span></div><div><span style="font-size: x-small;">--class com.example.MyApp specifies the main class of the Spark application.</span></div><div><span style="font-size: x-small;">--master yarn --deploy-mode cluster specifies the mode to run the application in.</span></div><div><span style="font-size: x-small;">/path/to/my/app.jar arg1 arg2 specifies the path to the application jar file and its arguments.</span></div><div><br /></div><div>The above example does not explicitly require Hadoop to be installed or used. However, it assumes that the Spark cluster is running in YARN mode, which is typically used in a Hadoop cluster. In general, Spark can be run in various modes, including standalone, YARN, and Mesos. There are various other parameters and configurations that can be specified. Some examples include:</div></div><div><div><span style="font-size: x-small;">--num-executors: Specifies the number of executor processes to use for the job.</span></div><div><span style="font-size: x-small;">--executor-cores: Specifies the number of cores to allocate per executor.</span></div><div><span style="font-size: x-small;">--executor-memory: Specifies the amount of memory to allocate per executor.</span></div><div><span style="font-size: x-small;">--driver-memory: Specifies the amount of memory to allocate for the driver process.</span></div><div><span style="font-size: x-small;">--queue: Specifies the YARN queue to submit the job to.</span></div><div><span style="font-size: x-small;">--files: Specifies a comma-separated list of files to be distributed with the job.</span></div><div><span style="font-size: x-small;">--archives: Specifies a comma-separated list of archives to be distributed with the job.</span></div><div><span style="font-size: x-small;"><br /></span></div><div>These parameters can be used to fine-tune the resource allocation and performance of Spark jobs in a Hadoop cluster. Additionally, there are other options that can be used to configure the behavior of the Spark application itself, such as --conf to specify Spark configuration options and --jars to specify external JAR files to be used by the application</div></div><div><br /></div><div><div>Here is an example LSF configuration file (lsf.conf) that includes settings for running Spark applications:</div><div><span style="background-color: #eeeeee; font-size: x-small;"># LSF Configuration File</span></div><div><span style="background-color: #eeeeee; font-size: x-small;"># Spark settings</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">LSB_JOB_REPORT_MAIL=N</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">LSB_DEFAULTGROUP=spark</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">LSB_DEFAULTJOBGROUP=spark</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">LSB_JOB_ACCOUNTING_INTERVAL=60</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">LSB_SUB_LOGLEVEL=3</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">LSB_JOB_PROLOGUE="/opt/spark/current/bin/load-spark-env.sh"</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">LSB_JOB_WRAPPER="mpirun -n 1 $LSF_BINDIR/lsb.wrapper $LSB_BINARY_NAME"</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">LSB_HOSTS_TASK_MODEL=cpu</span></div><div><br /></div><div><br /></div><div>An example Spark configuration file (spark-defaults.conf) that includes settings for running Spark applications using LSF:</div><div># Spark Configuration File</div><div># LSF settings</div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.master=yarn</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.submit.deployMode=cluster</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.yarn.queue=default</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.executor.instances=2</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.executor.memory=2g</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.executor.cores=2</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.driver.memory=1g</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.driver.cores=1</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.yarn.am.memory=1g</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.yarn.am.cores=1</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.yarn.maxAppAttempts=2</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.eventLog.enabled=true</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.eventLog.dir=hdfs://namenode:8020/spark-event-logs</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.history.fs.logDirectory=hdfs://namenode:8020/spark-event-logs</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.scheduler.mode=FAIR</span></div><div><span style="background-color: #eeeeee; font-size: x-small;">spark.serializer=org.apache.spark.serializer.KryoSerializer</span></div><div><br /></div><div>This configuration file sets several parameters for running Spark applications on a YARN cluster managed by LSF, including specifying the number of executor instances, executor memory, and executor cores, as well as setting the queue and memory allocation for the Spark ApplicationMaster.</div></div><div><br /></div><div><a href="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=hadoop-configure-apache-integration" target="_blank">Configure the Apache Hadoop integration</a></div><div><br /></div><div><a href="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=hadoop-run-application-lsf" target="_blank">Run a Hadoop application on LSF</a></div></div><div><br /></div><div><div>Using LSF as the scheduler for Hadoop can provide better resource utilization, job scheduling, queuing, integration with other workloads, and monitoring and management capabilities than the built-in YARN scheduler. This can help improve the performance, scalability, and efficiency of Hadoop clusters, especially in large, complex environments.</div><div><ol><li>Better resource utilization: LSF has advanced resource allocation and scheduling algorithms that can improve resource utilization in Hadoop clusters. This can lead to better performance and reduced infrastructure costs.</li><li>Better job scheduling: LSF has more advanced job scheduling features than YARN, such as support for job dependencies, job preemption, and priority-based job scheduling. This can help optimize job execution and reduce waiting times.</li><li>Advanced queuing: LSF allows for more flexible and advanced queuing mechanisms, including job prioritization and preemption, multiple queues with different priorities, and customizable scheduling policies.</li><li>Integration with other workloads: LSF is a general-purpose job scheduler that can be used to manage a wide range of workloads, including Hadoop, MPI, and other distributed computing frameworks. This allows for better integration and coordination of workloads on the same infrastructure.</li><li>Advanced monitoring and management: LSF provides more advanced monitoring and management tools than YARN, including web-based interfaces, command-line tools, and APIs for job management, resource monitoring, and performance analysis.</li></ol><div><div>LSF is a versatile job scheduler that can be used for a wide range of workloads, including batch and real-time scheduling. While LSF is often used for batch scheduling workloads, it can also be used for real-time scheduling workloads like Apache Kafka, thanks to its advanced scheduling capabilities and integration capabilities with other distributed computing frameworks.</div><div><br /></div><div>LSF has advanced scheduling capabilities that can help optimize the allocation of resources for real-time workloads, including support for job prioritization, preemption, and multiple queues with different priorities. This can help ensure that real-time workloads are allocated the necessary resources in a timely and efficient manner.</div><div><br /></div><div>Furthermore, LSF has integration capabilities with other distributed computing frameworks like Apache Kafka. For example, LSF can be used to manage the resource allocation and scheduling of Kafka brokers, consumers, and producers. This can help optimize the performance and scalability of Kafka clusters.</div></div></div></div><div><br /></div><div><div>Examples for applications with real time scheduling:</div><div><ol><li>A major financial services company uses Hadoop and LSF to process real-time financial data. LSF is used to manage the allocation of compute resources for Hadoop, including managing the cluster's memory, CPU, and disk resources. This setup enables the company to process real-time financial data with low latency and high throughput.</li><li>A large e-commerce company uses Hadoop and LSF to process large volumes of customer data in real-time. LSF is used to schedule and manage jobs across multiple Hadoop clusters, optimizing the allocation of resources to ensure that real-time processing is prioritized. This setup enables the company to personalize customer experiences and deliver targeted marketing campaigns in real-time.</li><li>A global telecommunications company uses Hadoop and LSF to process real-time data from its network infrastructure. LSF is used to manage job scheduling and resource allocation, ensuring that data is processed quickly and efficiently. This setup enables the company to monitor and optimize network performance in real-time, providing a better customer experience.</li></ol></div><div><br /></div><div>Overall, the combination of Hadoop and LSF can provide a powerful and flexible platform for processing both historical as well as real-time data in production environments. By leveraging the advanced resource management and scheduling capabilities of LSF, organizations can optimize performance, reduce latency, and improve the overall efficiency of their Hadoop clusters.</div></div><div><br /></div><div><span style="font-size: x-small;"><b>Reference:</b></span></div><div><div><ul><li><span style="font-size: xx-small;">https://www.sachinpbuzz.com/2019/08/spectum-lsf-multicluster-job-forwarding.html</span></li><li><span style="font-size: xx-small;">https://www.sachinpbuzz.com/2021/08/spectrum-lsf-101-installation-and.html</span></li><li><span style="font-size: xx-small;">https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=hadoop-about-lsf-apache</span></li><li><span style="font-size: xx-small;">https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=hadoop-run-application-lsf</span></li><li><span style="font-size: xx-small;"><a href="https://hadoop.apache.org/">https://hadoop.apache.org/</a></span></li><li><span style="font-size: xx-small;"><a href="https://www.edureka.co/blog/videos/hadoop-architecture/">https://www.edureka.co/blog/videos/hadoop-architecture/</a></span></li></ul></div></div></div>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-77286413895798121442023-04-04T18:31:00.025+05:302023-08-31T10:02:39.023+05:30Linux Test Harness : avocado and op-test framework<p style="text-align: justify;">A Test Harness, also known as a testing framework or testing tool, is a software tool or library that provides a set of functions, APIs, or interfaces for writing, organizing, and executing tests. Test harnesses provide a structured way to write tests and automate the testing process. <br /></p><p style="text-align: justify;">Linux avocado test framework and Linux op-test framework are both open-source testing frameworks designed for testing and validating Linux-based systems. Both frameworks are widely used in the Linux community and have a strong user base. The choice between the two depends on the specific testing needs and requirements of the user.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivk3LOMZyjAOBj_xmF9JHAlhl4jZ6XI60m4g7mR1do_OfveY44GoRWCnF0LaF8HWAvsFIxzi17d0AWXLWA7ZRsv_LlVG3jzmORXyl3NUJXkuNsJ4u5aCrftpOM1dwrjcMvxjTTiX39p4rl-90AbW_ndufiMw3u7n5XBbHHn_oWdyvNBgNHRqdTy1fUcw/s200/avocado.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="200" data-original-width="200" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivk3LOMZyjAOBj_xmF9JHAlhl4jZ6XI60m4g7mR1do_OfveY44GoRWCnF0LaF8HWAvsFIxzi17d0AWXLWA7ZRsv_LlVG3jzmORXyl3NUJXkuNsJ4u5aCrftpOM1dwrjcMvxjTTiX39p4rl-90AbW_ndufiMw3u7n5XBbHHn_oWdyvNBgNHRqdTy1fUcw/s1600/avocado.png" width="200" /></a></div><p></p><p style="text-align: justify;">The Linux avocado test framework is a modular and extensible testing framework that allows users to write and run tests for different levels of the Linux stack, including the kernel, user space, and applications. It provides a wide range of plugins and tools for testing, including functional, performance, and integration testing. The framework is easy to install and use and supports multiple test runners and reporting formats.</p><p style="text-align: justify;">On the other hand, the Linux op-test framework is a set of Python libraries and utilities that automate the testing of hardware and firmware components in Linux-based systems. It provides a high-level Python API for interacting with hardware and firmware interfaces, as well as a set of pre-built tests for validating various hardware components such as CPU, memory, and storage. The framework is highly flexible and customizable, allowing users to create their own tests and integrate with other testing tools and frameworks.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: justify;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdIGq0WLjWvhjSV60WyVCgdl9C4iaJHURw_dlXZnZoiaB2uTVu0eR6j9d-oRW5Toe0XJW6SpNxCKdchsuq0a6h7cZqZpRZME1yYQXpRFH_RvfLGfa8pIeeZ7clyNINvH9Rl3nBmzIQ3sIGGRSOLJXLb9zu0cgdwksRpzsVJBmN0YftdhYoPhzgyWMnCQ/s279/openpower.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="180" data-original-width="279" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdIGq0WLjWvhjSV60WyVCgdl9C4iaJHURw_dlXZnZoiaB2uTVu0eR6j9d-oRW5Toe0XJW6SpNxCKdchsuq0a6h7cZqZpRZME1yYQXpRFH_RvfLGfa8pIeeZ7clyNINvH9Rl3nBmzIQ3sIGGRSOLJXLb9zu0cgdwksRpzsVJBmN0YftdhYoPhzgyWMnCQ/s1600/openpower.png" width="279" /></a></div><p></p></blockquote></blockquote><p style="text-align: justify;">While both frameworks are designed for testing Linux-based systems, the Linux avocado test framework provides a broad range of testing capabilities across different levels of the Linux stack, while the Linux op-test framework focuses specifically on automating hardware and firmware testing. The choice between the two depends on the specific testing needs and requirements of the user.</p><p style="text-align: justify;">The Linux avocado test framework provides a plugin called "avocado-vt" which can be used to run tests that require a reboot between different test stages. This plugin enables the framework to run destructive tests, like kernel crash dump (kdump) testing, that require the system to be rebooted multiple times.</p><p style="text-align: justify;">Similarly, the Linux op-test framework also provides support for testing scenarios that require system reboot. The framework includes a "reboot" library that allows users to reboot the system under test and wait for it to come back up before continuing with the test. This library can be used to test scenarios like kdump and fadump that require system reboot.</p><p style="text-align: justify;"><b>The community maintained avocado tests repository</b>:</p><p style="text-align: justify;">Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.</p><p style="text-align: justify;">This repository contains a collection of miscellaneous tests and plugins for the Linux Avocado test framework that cover a wide range of functional, performance, and integration testing scenarios. The tests are designed to be modular and easy to use, and can be integrated with the Avocado test framework to extend its capabilities.</p><p style="text-align: justify;"><a href="https://github.com/avocado-framework-tests/avocado-misc-tests">https://github.com/avocado-framework-tests/avocado-misc-tests</a></p><p style="text-align: justify;"><b>How to run avocado misc tests :</b></p><p style="text-align: justify;">To run the Avocado Misc Tests, you first need to install the Linux Avocado test framework on your system. Once you have installed the framework, you can clone the Avocado Misc Tests repository from GitHub by running the following command in a terminal:</p><p style="text-align: justify;">git clone https://github.com/avocado-framework-tests/avocado-misc-tests.git </p><p style="text-align: justify;">git clone git@github.com:avocado-framework-tests/avocado-misc-tests.git</p><p style="text-align: justify;"><span style="background-color: #cccccc; font-size: x-small;"># git clone git@github.com:avocado-framework-tests/avocado-misc-tests.git<br />Cloning into 'avocado-misc-tests'...<br />remote: Enumerating objects: 18087, done.<br />remote: Counting objects: 100% (451/451), done.<br />remote: Compressing objects: 100% (239/239), done.<br />remote: Total 18087 (delta 242), reused 368 (delta 208), pack-reused 17636<br />Receiving objects: 100% (18087/18087), 6.15 MiB | 16.67 MiB/s, done.<br />Resolving deltas: 100% (11833/11833), done.<br />#</span></p><p style="text-align: justify;">This repository is dedicated to host any tests written using the Avocado API. It is being initially populated with tests ported from autotest client tests repository, but it's not limited by that.</p><p style="text-align: justify;">After cloning the repository, you can navigate to the avocado-misc-tests directory and run the tests using the avocado run command. For example, to run all the tests in the network category, you can run the following command:</p><p style="text-align: justify;"><span style="background-color: #cccccc;">cd avocado-misc-tests<br />avocado run network/</span></p><p style="text-align: justify;">This will run all the tests in the network category. You can also run individual tests by specifying the path to the test file, like this:</p><p style="text-align: justify;"><span style="background-color: #cccccc;">avocado run network/test_network_ping.py</span></p><p style="text-align: justify;">This will run the test_network_ping.py test in the network category.</p><p style="text-align: justify;">Before running the tests, you may need to configure the Avocado framework to use the appropriate test runner, test environment, and plugins for your system. You can find more information on how to configure and use the Avocado framework in the official documentation: </p><p style="text-align: justify;">https://avocado-framework.readthedocs.io/en/latest/</p><p style="text-align: justify;"><span style="background-color: #cccccc; font-size: x-small;">$ avocado run avocado-misc-tests/generic/stress.py<br />JOB ID : 0018adbc07c5d90d242dd6b341c87972b8f77a0b<br />JOB LOG : $HOME/avocado/job-results/job-2016-01-18T15.32-0018adb/job.log<br />TESTS : 1<br /> (1/1) avocado-misc-tests/generic/stress.py:Stress.test: PASS (62.67 s)<br />RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0<br />JOB HTML : $HOME/avocado/job-results/job-2016-01-18T15.32-0018adb/html/results.html<br />TIME : 62.67 s</span></p><p style="text-align: justify;">There are a few more interesting things about the Avocado test framework and its usability and use cases:</p><p style="text-align: justify;"></p><p style="text-align: justify;"><span style="background-color: #cccccc; font-size: x-small;"></span></p><p></p><p></p><ol><li>Flexible test design: The Avocado test framework is designed to be flexible and adaptable to a wide range of testing scenarios. It supports various test types, including functional, integration, performance, and stress tests, and can be used to test software at different levels of abstraction, from system-level to individual components. Avocado also provides a wide range of plugins and interfaces for integrating with other tools and frameworks, making it easy to customize and extend its capabilities.</li><li>Easy to use: Avocado is designed to be easy to use, even for users who are new to testing or have limited programming experience. It uses a simple YAML-based syntax for defining tests and test plans, and provides a user-friendly command-line interface for running tests and viewing results. Avocado also includes detailed documentation and tutorials to help users get started quickly.</li><li>Scalability and distributed testing: Avocado supports distributed testing across multiple systems, making it easy to scale up testing to handle large workloads. It includes a built-in job scheduler for managing test execution across multiple systems, and can be integrated with various cloud-based services for running tests in the cloud.</li><li>Community support: Avocado is an open-source project maintained by a vibrant community of developers and testers. The community provides regular updates and bug fixes, and is actively involved in improving the usability and functionality of the framework. The Avocado community also provides support through various channels, including GitHub, mailing lists, and IRC.</li><li>Use cases: Avocado is used by various organizations and companies for testing different types of software, including operating systems, virtualization platforms, container platforms, and cloud services. It is particularly well-suited for testing complex, distributed systems that require a high degree of automation and scalability. Some of the organizations that use Avocado include Red Hat, IBM, Intel, and Huawei.</li></ol><p style="text-align: justify;"><span style="font-size: x-small;"><b>License</b></span></p><p style="text-align: justify;"><span style="font-size: x-small;">Except where otherwise indicated in a given source file, all original contributions to Avocado are licensed under the GNU General Public License version 2 (GPLv2) or any later version. </span><span style="font-size: small;">By contributing you agree that these contributions are your own (or approved by your employer) and you grant a full, complete, irrevocable copyright license to all users and developers of the Avocado project, present and future, pursuant to the license of the project.</span></p><p style="text-align: justify;">================</p><p style="text-align: justify;">The community maintained, op-tests repository:</p><p style="text-align: justify;"><a href="https://github.com/open-power/op-test">https://github.com/open-power/op-test</a></p><p style="text-align: justify;">git clone git@github.com:open-power/op-test.git</p><p style="text-align: justify;"><span style="background-color: #cccccc;"># git clone git@github.com:open-power/op-test.git<br />Cloning into 'op-test'...<br />remote: Enumerating objects: 8716, done.<br />remote: Counting objects: 100% (623/623), done.<br />remote: Compressing objects: 100% (275/275), done.<br />remote: Total 8716 (delta 416), reused 480 (delta 347), pack-reused 8093<br />Receiving objects: 100% (8716/8716), 23.89 MiB | 23.39 MiB/s, done.<br />Resolving deltas: 100% (6488/6488), done.<br />#</span></p><div style="text-align: justify;"><b><span style="font-size: x-small;">Pre-requisites for op-tests: </span></b></div><div style="text-align: justify;"><span style="font-size: x-small;">1) yum install sshpass </span></div><div style="text-align: justify;"><span style="font-size: x-small;">2) pip3 install pexpect</span></div><div style="text-align: justify;"><span style="font-size: x-small;">3) echo "set enable-bracketed-paste off" > .inputrc ; export INPUTRC=$PWD/.inputrc</span></div><div style="text-align: justify;"><span style="font-size: x-small;"><div>bind 'set enable-bracketed-paste off'</div><div><br /></div><div>How to run testcase:</div></span></div><ol></ol><ol></ol><ol></ol><ol></ol><ol></ol><ol></ol><ol></ol><p></p><p style="text-align: justify;"><b> ./op-test -c machine.conf --run testcases.RunHostTest --host-cmd ls</b></p><p style="text-align: justify;"><span style="font-size: x-small;"><b>Testcase: </b>https://github.com/open-power/op-test/blob/master/testcases/RunHostTest.py</span></p><p style="text-align: justify;"><span style="font-size: x-small;"><b>where machine.conf :</b></span></p><p style="text-align: justify;"><span style="background-color: #cccccc;"><span style="font-size: x-small;">[op-test]</span><br /><span style="font-size: x-small;">bmc_type=OpenBMC /</span><span style="font-size: small;">EBMC_PHYP/</span><span style="font-size: x-small;">FSP_PHYP</span><br /><span style="font-size: small;">bmc_username=abc</span><br /><span style="font-size: small;">bmc_ip=w39</span><br /><span style="font-size: small;">bmc_username=root</span><br /><span style="font-size: small;">bmc_password=0penBmc</span><br /><span style="font-size: small;">hmc_ip=a.b.c.d</span><br /><span style="font-size: small;">hmc_username=hmcuser</span><br /><span style="font-size: small;">hmc_password=hmcpasswd123</span><br /><span style="font-size: small;">host_ip=x.y.x.k</span><br /><span style="font-size: small;">host_user=hostuser</span><br /><span style="font-size: small;">host_password=hostpasswd123</span><br /><span style="font-size: small;">system_name=power10</span><br /><span style="font-size: small;">lpar_name=lpar_name_1</span><br /><span style="font-size: small;">lpar_prof=default_profile</span></span></p><p style="text-align: justify;"><span><span style="background-color: white; font-size: small;">CASE2:</span></span></p><p style="text-align: justify;"><span><span style="background-color: white; font-size: small;">./op-test -c machine_ltcever7x0.config --run testcases.RunHostTest --host-cmd-file cmd.conf</span></span></p><p style="text-align: justify;"><span><span style="background-color: white; font-size: small;">where : </span></span></p><p style="text-align: justify;"><span style="font-size: x-small;"># cat cmd.conf<br />echo "welcome SACHIN P B"<br />hostname<br />uptime<br />date<br />#</span></p><p style="text-align: justify;"><span style="font-size: x-small;">OUTPUT:</span></p><p style="text-align: justify;"><span style="font-size: x-small;">----------------------------</span></p><p style="text-align: justify;"><span style="font-size: x-small;"># ./op-test -c machine_ltcever7x0.config --run testcases.RunHostTest --host-cmd-file cmd.conf</span><br /><span style="font-size: small;">Logs in: /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648</span><br /><span style="font-size: small;">2023-08-29 09:56:48,758:op-test:setUpLoggerFile:INFO:Preparing to set location of Log File to /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758035.main.log</span><br /><span style="font-size: small;">2023-08-29 09:56:48,758:op-test:setUpLoggerFile:INFO:Log file: /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758035.main.log</span><br /><span style="font-size: small;">2023-08-29 09:56:48,758:op-test:setUpLoggerDebugFile:INFO:Preparing to set location of Debug Log File to /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758291.debug.log</span><br /><span style="font-size: small;">[console-expect]#which whoami && whoami</span><br /><span style="font-size: small;">/usr/bin/whoami</span><br /><span style="font-size: small;">root</span><br /><span style="font-size: small;">[console-expect]#echo $?</span><br /><span style="font-size: small;">echo $?</span><br /><span style="font-size: small;">0</span><br /><span style="font-size: small;">[console-expect]#echo "welcome SACHIN P B"</span><br /><span style="font-size: small;">echo "welcome SACHIN P B"</span><br /><span style="font-size: small;">welcome SACHIN P B</span><br /><span style="font-size: small;">[console-expect]#echo $?</span><br /><span style="font-size: small;">echo $?</span><br /><span style="font-size: small;">0</span><br /><span style="font-size: small;">[console-expect]#hostname</span><br /><span style="font-size: small;">hostname</span><br /><span style="font-size: small;">myhost.com</span><br /><span style="font-size: small;">[console-expect]#echo $?</span><br /><span style="font-size: small;">echo $?</span><br /><span style="font-size: small;">0</span><br /><span style="font-size: small;">[console-expect]#uptime</span><br /><span style="font-size: small;">uptime</span><br /><span style="font-size: small;"> 09:58:15 up 7:50, 2 users, load average: 0.08, 0.02, 0.01</span><br /><span style="font-size: small;">[console-expect]#echo $?</span><br /><span style="font-size: small;">echo $?</span><br /><span style="font-size: small;">0</span><br /><span style="font-size: small;">[console-expect]#date</span><br /><span style="font-size: small;">date</span><br /><span style="font-size: small;">Tue Aug 29 09:58:15 CDT 2023</span><br /><span style="font-size: small;">[console-expect]#echo $?</span><br /><span style="font-size: small;">echo $?</span><br /><span style="font-size: small;">0</span><br /><span style="font-size: small;">ok</span><br /><span style="font-size: small;">Ran 1 test in 7.510s</span><br /><span style="font-size: small;">OK</span><br /><span style="font-size: small;">2023-08-29 09:58:17,787:op-test:<module>:INFO:Exit with Result errors="0" and failures="0"</span></p><p style="text-align: justify;"><span style="font-size: x-small;">------------------------------------------------------------------------------------------------------------------</span></p><p style="text-align: justify;"><span style="font-size: x-small;">Example 2: </span></p><p style="text-align: justify;"><span style="font-size: x-small;">python3 op-test -c machine.conf --run testcases.PowerNVDump.KernelCrash_disable_radix</span></p><p style="text-align: justify;"><span style="font-size: small;">python3 op-test -c machine.conf --run testcases.PowerNVDump.KernelCrash_XIVE_off</span></p><p style="text-align: justify;"><span style="font-size: x-small;">python3 op-test --run-suite osdump-suite -c CR-machine.conf</span></p><p style="text-align: justify;"><span style="font-size: x-small;">python3 op-test --run testcases.RunHostTest -c CR-Machine.conf --host-cmd-file CR-Machine_command.conf --host-cmd-timeout 286400</span></p><p style="text-align: justify;"><span style="background-color: #eeeeee; font-size: x-small;"></span></p><p style="text-align: justify;"><span style="background-color: #eeeeee; font-size: x-small;"></span></p><div style="text-align: justify;">-------------------</div><div style="text-align: justify;"><b>How to analyze op-test output :</b></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">Traverse this directory path: op-test/test-reports/test-run-$date</div><div style="text-align: justify;">There are are 3 log files to investigate the test failure or life-cycle of testsuite</div><div style="text-align: justify;"><div><span style="font-size: xx-small;"># pwd</span></div><div><span style="font-size: xx-small;">/root/fix_kdump_FalsePositives/op-test/test-reports/test-run-$DATE</span></div><div><span style="font-size: xx-small;">#</span></div></div><div style="text-align: justify;"><div><span style="font-size: xx-small;"># ls -1</span></div><div><span style="font-size: xx-small;">$DATE.log</span></div><div><span style="font-size: xx-small;">$DATE.main.log</span></div><div><span style="font-size: xx-small;">$DATE.debug.log</span></div><div><span style="font-size: xx-small;">#</span></div><div><br /></div><div>1 ) $DATE.log ====> You could see console related commands and outputs </div><div><br /></div><div><span style="font-size: xx-small;">For example :</span></div><div><div><span style="font-size: xx-small;">lssyscfg -m Serverx0 -r lpar --filter lpar_names=Serverx0-lp6 -F state</span></div><div><span style="font-size: xx-small;">lsrefcode -m Serverx0 -r lpar --filter lpar_names=Serverx0-lp6 -F refcode</span></div><div><span style="font-size: xx-small;">chsysstate -m Serverx0 -r lpar -n Server-lp6 -o shutdown --immed</span></div><div><br /></div><div>2) $DATE.main.log</div><div>If you add any statements in log.info , that will be logged in this file </div><div><br /></div><div><span style="font-size: xx-small;">log.info("=============== Testing kdump/fadump over ssh ===============")</span></div><div><br /></div><div>3) $DATE.debug.log</div><div>If you add any comments with log.debug, that will be logged in this file </div><div><br /></div><div><span style="font-size: xx-small;">log.debug("SACHIN_DEBUG: In loop1")</span></div></div><div><br /></div></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div>Listed are some interesting things about the op-test framework and its use cases:</div><div><ol><li>Testing hardware systems: The op-test framework is designed for testing hardware systems, particularly servers, using the OpenPOWER architecture. It includes a wide range of tests that cover different aspects of hardware functionality, such as power management, CPU, memory, and I/O.</li><li>Integration with OpenBMC: The op-test framework integrates with the OpenBMC project, an open-source implementation of the Baseboard Management Controller (BMC) firmware that provides out-of-band management capabilities for servers. This integration allows users to control and monitor server hardware using the OpenBMC interface, and to run tests on the hardware using the op-test framework.</li><li>UEFI and firmware testing: The op-test framework includes support for testing UEFI firmware and other low-level system components, such as the Hostboot bootloader. This allows users to test the system firmware and ensure that it is functioning correctly.</li><li>Easy to use: The op-test framework is designed to be easy to use, even for users who are not familiar with hardware testing. It uses a simple command-line interface and provides detailed documentation and tutorials to help users get started quickly.</li><li>Scalability: The op-test framework is designed to be scalable and can be used to test multiple systems in parallel. This makes it suitable for testing large server farms and data centers.</li><li>Community support: The op-test framework is an open-source project with an active community of developers and testers. The community provides regular updates and bug fixes, and is actively involved in improving the usability and functionality of the framework. The op-test community also provides support through various channels, including GitHub, mailing lists, and IRC.</li><li>Use cases: The op-test framework is used by various organizations and companies for testing hardware systems, including server manufacturers, data center operators, and cloud service providers. Some of the organizations that use the op-test framework include IBM, Google, and Rackspace.</li></ol><div><b>How to contribute to op-test framework open source community :</b></div></div><div><br /></div><div><div>1) mkdir kdump_xive_off_check<br /><br />2) cd kdump_xive_off_check<br /><br />3) git clone git@github.com:SACHIN-PB/op-test.git</div><div><br /></div><div> <span style="font-size: x-small;"> Fork the repository from master : <b>https://github.com/open-power/op-test</b></span></div><div><span style="font-size: x-small;"><b><br /></b></span></div><div><span style="font-size: x-small;"> NOTE: In Git, forking a repository means creating a copy of the original repository into your own GitHub account. </span></div><div><span style="font-size: x-small;"> This is typically done when you want to contribute to an open-source project or collaborate with other developers.</span><br /><br />4) git config user.email<br /><br />5) git config user.name</div><div><br /></div><div><span style="font-size: x-small;">NOTE: To get proper username and email . Please do the following setup at /root directory</span></div><div><div><span style="background-color: #cccccc; font-size: x-small;"># cat .gitconfig</span></div><div><span style="background-color: #cccccc; font-size: x-small;">[user]</span></div><div><span style="background-color: #cccccc; font-size: x-small;"> email = sachin@linux.XYZ.com</span></div><div><span style="background-color: #cccccc; font-size: x-small;"> name = Sachin P B</span></div><div><span style="background-color: #cccccc; font-size: x-small;">#</span></div><br />6) git branch<br /><br />7) git remote -v</div><div><span style="font-size: x-small;"> <span style="background-color: #cccccc;"> origin git@github.com:SACHIN-PB/op-test.git (fetch)</span></span></div><div><span style="background-color: #cccccc; font-size: x-small;"> origin git@github.com:SACHIN-PB/op-test.git (push)</span></div><div><br /></div><div>8) git remote add upstream git@github.com:open-power/op-test.git</div><div><br /></div><div>9) git remote -v</div><div><span style="background-color: #cccccc;"> <span style="font-size: x-small;"> origin git@github.com:SACHIN-PB/op-test.git (fetch)</span></span></div><div><span style="background-color: #cccccc; font-size: x-small;"> origin git@github.com:SACHIN-PB/op-test.git (push)</span></div><div><span style="background-color: #cccccc; font-size: x-small;"> upstream git@github.com:open-power/op-test.git (fetch)</span></div><div><span style="background-color: #cccccc; font-size: x-small;"> upstream git@github.com:open-power/op-test.git (push)</span></div><div><br /></div><div>10) git checkout -b "kdump_xive_off_check"</div><div><br /></div><div>11) git branch</div><div><br /></div><div>12) vi testcases/PowerNVDump.py</div><div><br /></div><div>13) git diff</div><div><br /></div><div>14) git status</div><div><br /></div><div>15) git add testcases/PowerNVDump.py</div><div><br /></div><div>16) git status</div><div><br /></div><div>17) git commit -s</div><div><br /></div><div>18) git branch</div><div><br /></div><div>19) git push origin kdump_xive_off_check</div><div><span style="background-color: #cccccc; font-size: x-small;">Enumerating objects: 7, done.</span></div><div><span style="background-color: #cccccc; font-size: x-small;">Counting objects: 100% (7/7), done.</span></div><div><span style="background-color: #cccccc; font-size: x-small;">Delta compression using up to 16 threads</span></div><div><span style="background-color: #cccccc; font-size: x-small;">Compressing objects: 100% (4/4), done.</span></div><div><span style="background-color: #cccccc; font-size: x-small;">Writing objects: 100% (4/4), 880 bytes | 880.00 KiB/s, done.</span></div><div><span style="background-color: #cccccc; font-size: x-small;">Total 4 (delta 3), reused 0 (delta 0), pack-reused 0</span></div><div><span style="background-color: #cccccc; font-size: x-small;">remote: Resolving deltas: 100% (3/3), completed with 3 local objects.</span></div><div><span style="background-color: #cccccc; font-size: x-small;">remote:</span></div><div><span style="background-color: #cccccc; font-size: x-small;">remote: Create a pull request for 'kdump_xive_off_check' on GitHub by visiting:</span></div><div><span style="background-color: #cccccc; font-size: x-small;">remote: https://github.com/SACHIN-PB/op-test/pull/new/kdump_xive_off_check</span></div><div><span style="background-color: #cccccc; font-size: x-small;">remote:</span></div><div><span style="background-color: #cccccc; font-size: x-small;">To github.com:SACHIN-PB/op-test.git</span></div><div><span style="background-color: #cccccc; font-size: x-small;"> * [new branch] kdump_xive_off_check -> kdump_xive_off_check</span></div><div><span style="background-color: #cccccc; font-size: x-small;">#</span></div></div><div><br /></div><div>20) Create PR using the link created at step 19 and request for the review</div><div><span style="font-size: x-small;"><span style="background-color: white;">Example https://github.com/open-power/op-test/pull/7XYZ4:</span></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRhdLelO5Vgi4y5LrU1Ss-zba6AuMNfN4OjxprZRVR3L1et4R9yM5LnN-qk21AwZOknmIFngCBbtootsC1OuHJ0CtMtF9CLtE6Y6PeroOJ0lsFA0poPVqO9ZbadnjR13D3iHJLlc_lMEN5HKNxv9PFKFT5YqRVXmaXlospLhNaDNET2DGykUG3ucuCOw/s1380/xive-pr-snapshot.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="857" data-original-width="1380" height="398" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRhdLelO5Vgi4y5LrU1Ss-zba6AuMNfN4OjxprZRVR3L1et4R9yM5LnN-qk21AwZOknmIFngCBbtootsC1OuHJ0CtMtF9CLtE6Y6PeroOJ0lsFA0poPVqO9ZbadnjR13D3iHJLlc_lMEN5HKNxv9PFKFT5YqRVXmaXlospLhNaDNET2DGykUG3ucuCOw/w640-h398/xive-pr-snapshot.png" width="640" /></a></div></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">21) You can update your PR by running these commands </div><div style="text-align: justify;"><span style="background-color: #cccccc;">git commit --amend<br />git push -f origin kdump_xive_off_check</span></div><div style="text-align: justify;">======================</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><span style="font-size: xx-small;">Reference:</span></div><div style="text-align: justify;"><span style="font-size: xx-small;">1) https://github.com/open-power/op-test/blob/master/testcases/RunHostTest.py</span></div><div style="text-align: justify;"><span style="font-size: xx-small;">2) https://github.com/avocado-framework-tests/avocado-misc-tests</span></div><div style="text-align: justify;"><span style="font-size: xx-small;">3) </span><span style="text-align: left;"><span style="font-size: xx-small;">https://avocado-framework.readthedocs.io/en/latest/</span></span></div>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-8306474358691344022023-03-29T17:45:00.007+05:302023-03-30T10:41:19.511+05:30Linux security and kernel Lockdown - kernel image access prevention feature<p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgrm28BXMRIaIXqa2q1PM7pRWgojlDooeiuFa5P6q0E43zGXinO7biurqyL66Aurct8upTPnhD65geqAvZ_Hqddl4XE3MgRI8mJsN_VRMqz4BkV6LQRFsTyYeRnvjbt8i_iUarhLVsdR9XTKOQOSbWsaL76Y_Bss4XBfqIDf9n7KicD9LM286Tm_5Smg/s1200/ld.jpg" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="431" data-original-width="1200" height="115" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgrm28BXMRIaIXqa2q1PM7pRWgojlDooeiuFa5P6q0E43zGXinO7biurqyL66Aurct8upTPnhD65geqAvZ_Hqddl4XE3MgRI8mJsN_VRMqz4BkV6LQRFsTyYeRnvjbt8i_iUarhLVsdR9XTKOQOSbWsaL76Y_Bss4XBfqIDf9n7KicD9LM286Tm_5Smg/w254-h115/ld.jpg" width="254" /></a></div><div style="text-align: justify;"><div>Linux has a long history of security-focused development and has been used in many high-security environments, such as military and government organizations. Linux is highly customizable, which allows administrators to tailor security configurations to their specific needs. For example, security modules like SELinux and AppArmor can be configured to enforce highly granular access control policies. Many Linux distributions include security-focused features, such as hardening patches and secure boot support, by default.The open-source nature of Linux allows for community-driven development and auditing, which can help to uncover security vulnerabilities and improve the overall security of the system. Linux containers, such as Docker and Kubernetes, have become increasingly popular in recent years and offer a more secure alternative to traditional virtualization solutions. Linux is widely used in cloud environments and has many built-in features for secure cloud deployments, such as network isolation and encryption. It is constantly being updated and improved with new security features and bug fixes, making it one of the most secure operating systems available.</div></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">The Kernel Lockdown feature is designed to prevent both direct and indirect access to a running kernel image, attempting to protect against unauthorized modification of the kernel image and to prevent access to security and cryptographic data located in kernel memory, whilst still permitting driver modules to be loaded. This is security feature, the Linux Security Module (LSM, nicknamed “lockdown”). It does promise to bring additional security to one of the most widely-used and hardened kernels on the market. The lockdown feature’s aim is to restrict various pieces of kernel functionality. There are two modes available to the lockdown module: Integrity and Confidentiality. When in Integrity mode, kernel features which would allow userland code to modify the running kernel are disabled. When in Confidentiality mode, userland code to extract confidential information from the kernel will be disabled.First off, it will restrict access to kernel features that may allow arbitrary code execution by way of code supplied by any application or service outside of the kernel (aka “userland”). The new feature will also block processes from reading/writing to /dev/mem and /dev/kmem memory, as well as block access to opening /dev/port (as a means to prevent raw ioport access). Other features include:</div><div style="text-align: justify;"><div><ol><li>Enforcing kernel module signatures.</li><li>Prevents even the root account from modifying the kernel code.</li><li>Kexec reboot (in case secure boot being enabled does not keep the secure boot mode in new kernel).</li><li>Lockdown of hardware that could potentially generate direct memory addressing (DMA).</li><li>Lockdown of KDADDIO, KDDELIO, KDENABIO and KDDISABIO console ioctls.</li></ol><div>where</div></div><div><ul><li><span style="font-size: x-small;">The KDADDIO, KDDELIO, KDENABIO, and KDDISABIO console ioctls are used to manage console input/output (I/O) on Linux systems. Here's a brief overview of each of these ioctls:</span></li><li><span style="font-size: x-small;">KDADDIO: This ioctl is used to add a new input/output device to the console. When a new device is added using KDADDIO, it can be used to send input to or receive output from the console.</span></li><li><span style="font-size: x-small;">KDDELIO: This ioctl is used to remove an input/output device from the console. When a device is removed using KDDELIO, it is no longer able to send input to or receive output from the console.</span></li><li><span style="font-size: x-small;">KDENABIO: This ioctl is used to enable input/output from a specific device on the console. When a device is enabled using KDENABIO, it can be used to send input to or receive output from the console.</span></li><li><span style="font-size: x-small;">KDDISABIO: This ioctl is used to disable input/output from a specific device on the console. When a device is disabled using KDDISABIO, it is no longer able to send input to or receive output from the console.</span></li></ul><div><span style="font-size: x-small;">NOTE: The "KD" in these console ioctls stands for Keyboard Display. The term "keyboard display" is used to refer to the console on a computer system, which includes the keyboard and screen used to interact with the system</span></div></div></div><p></p><p>If a prohibited or restricted feature is accessed or used, the kernel will emit a message that looks like:</p><p> <span style="background-color: #cccccc;">Lockdown: X: Y is restricted, see man kernel_lockdown.7</span></p><p>where X indicates the process name and Y indicates what is restricted. On an EFI-enabled x86 or arm64 machine, lockdown will be automatically enabled if the system boots in EFI Secure Boot mode.</p><p>Coverage: When lockdown is in effect, a number of features are disabled or have their use restricted. This includes special device files and kernel services that allow direct access of the kernel image:</p><p style="text-align: left;"><span style="background-color: #cccccc;"> /dev/mem<br /> /dev/kmem<br /> /dev/kcore<br /> /dev/ioports<br /> BPF<br /> kprobes</span></p><p>and the ability to directly configure and control devices, so as to prevent the use of a device to access or modify a kernel image:The use of module parameters that directly specify hardware parameters to drivers through the kernel command line or when loading a module.</p><p>The term "lockdown" refers to a set of security features in the Linux kernel that are designed to prevent even privileged users, such as the root user, from bypassing certain security restrictions. These features are intended to provide an additional layer of protection against malicious software and unauthorized access to sensitive information.</p><p>There are two main components to the lockdown feature:</p><p><b>Integrity measurement:</b> This feature prevents changes to the kernel's security settings, such as disabling secure boot or loading unsigned kernel modules, even by users with root privileges.</p><p><b>Confidentiality protection:</b> This feature prevents user space processes from accessing certain sensitive information, such as kernel memory or hardware resources, even if the processes are running with root privileges.</p><p>The lockdown feature is a powerful tool for enhancing the security of Linux systems, particularly in high-security environments or those where data privacy is a top concern. However, it can also limit the flexibility of the system, so it's important to carefully consider the trade-offs before enabling this feature.</p><p>---------------------------------------------------------</p><p>The lockdown feature and SELinux are both security features in the Linux kernel, but they serve different purposes and work independently of each other.</p><p style="text-align: justify;"><b>SELinux is a mandatory access control (MAC)</b> system that enforces a set of security policies to determine what processes and users can access specific resources, such as files or network ports. It operates by labeling resources with a security context and assigning labels to users and processes. The security policies defined in SELinux are enforced by the kernel and can prevent unauthorized access and other security breaches.</p><p style="text-align: justify;">The lockdown feature, on the other hand, is designed to prevent even privileged users, including those with root privileges, from bypassing certain security restrictions. It achieves this by restricting access to certain kernel features and preventing modifications to the kernel's security settings.</p><p style="text-align: justify;">When both SELinux and the lockdown feature are enabled, they work together to provide a comprehensive security solution. SELinux enforces mandatory access controls to restrict access to resources, while the lockdown feature ensures that even privileged users cannot bypass certain security restrictions. This can help to prevent security breaches caused by malicious software or unauthorized access to sensitive information.</p><p style="text-align: justify;">The combination of SELinux and the lockdown feature provides a powerful security solution for Linux systems. It's important to carefully configure and manage these features to ensure that they do not interfere with normal system operations or cause unintended consequences.</p><p style="text-align: justify;">The idea of effectively rendering the root account less capable of working with a system (on a kernel level), might be considered (to some) a disservice to Linux (and Linux administrators). However, in the realm of business, absolute security is a necessity — especially on machines that house sensitive business/customer data. When the root account is under a form of strict lockdown, malicious code would be significantly more challenging to run rampant on a system. This could lead to fewer data breaches. And because the kernel developers are making the lockdown feature “optional,” it is possible for enterprise admins to enable the feature on production machines that store such sensitive data. Conversely, on standard desktop machines (or developer machines) the feature can remain disabled.</p><p style="text-align: justify;">Linux kernel has several security features built into it to protect against various types of security threats. Some of the additional security features of Linux kernel include:</p><p style="text-align: justify;">1) <b>AppArmor</b>: AppArmor is a mandatory access control (MAC) system that restricts the capabilities of individual applications or processes. It can be used to enforce security policies that limit the actions of individual applications, such as restricting access to certain files or network resources.</p><p style="text-align: justify;">2) <b>Control Groups (cgroups):</b> cgroups provide a way to organize and manage system resources, such as CPU, memory, and I/O bandwidth, among different processes. This helps to prevent individual processes from monopolizing system resources, which can improve system performance and stability.</p><p style="text-align: justify;">3) <b>Kernel SamePage Merging (KSM)</b>: KSM allows multiple identical memory pages to be merged into a single page, reducing memory usage and improving system performance. However, this feature also presents a potential security risk, as it could allow an attacker to create a malicious page that looks like a legitimate page, thereby bypassing memory protection measures.</p><p style="text-align: justify;">4) <b>Executable Space Protection</b>: Executable Space Protection is a security feature that prevents execution of code from memory pages that are marked as data or stack. This helps to prevent buffer overflow and other types of attacks that rely on executing code in memory regions that are not intended for code execution.</p><p style="text-align: justify;">5) <b>Secure Boot</b>: Secure Boot is a security feature that ensures that only trusted software is executed during system boot-up. It uses cryptographic signatures to verify the authenticity of boot loaders and other critical components of the system, preventing unauthorized or malicious software from running at boot time.</p><p style="text-align: justify;">6) <b>Address Space Layout Randomization</b> (ASLR): This feature randomizes the memory layout of user space programs, making it more difficult for attackers to exploit vulnerabilities in the program's code.</p><p style="text-align: justify;">7) <b>Seccomp</b>: This feature provides a mechanism for filtering system calls that can be made by a process, allowing administrators to restrict the system calls that can be made by certain programs.</p><p style="text-align: justify;">8) <b>Trusted Platform Module (TPM)</b>: This is a hardware-based security feature that provides a secure storage area for cryptographic keys and other sensitive data. It can be used to enhance the security of system booting, disk encryption, and other security-related functions.</p><p style="text-align: justify;">9) <b>SELinux </b>similar to<b> </b>AppArmor. These are two popular security modules that provide mandatory access control (MAC) enforcement in the Linux kernel. They use security policies to define what resources, such as files and network ports, can be accessed by which processes and users.</p><p style="text-align: justify;">-----------------------------</p><p style="text-align: justify;">The Linux kernel includes a variety of cryptography algorithms that can be used to provide secure communication, storage, and other security-related functions. Here are some of the key cryptography algorithms and features in the Linux kernel:</p><p style="text-align: justify;">1) Advanced Encryption Standard (AES): AES is a symmetric encryption algorithm that is widely used for data encryption. The Linux kernel includes an implementation of AES that can be used by applications and other kernel subsystems.</p><p style="text-align: justify;">2) RSA: RSA is an asymmetric encryption algorithm that is used for digital signatures and key exchange. The Linux kernel includes an implementation of RSA that can be used by applications and other kernel subsystems.</p><p style="text-align: justify;">3) SHA: SHA (Secure Hash Algorithm) is a family of cryptographic hash functions that are used for digital signatures, data integrity checking, and other security-related functions. The Linux kernel includes implementations of several SHA algorithms, including SHA-1, SHA-256, and SHA-512.</p><p style="text-align: justify;">4) Random Number Generation: Random number generation is a critical component of many cryptographic functions. The Linux kernel includes several sources of entropy that are used to generate high-quality random numbers for use in cryptography algorithms.</p><p style="text-align: justify;">5) Cryptographic API: The Linux kernel includes a Cryptographic API that provides a standard interface for using cryptographic functions in kernel modules and applications. The API includes support for a wide range of cryptographic algorithms and features, including those listed above.</p><p style="text-align: justify;">6) Filesystem Encryption: The Linux kernel includes support for encrypting filesystems using the dm-crypt subsystem. This allows for encrypted storage of sensitive data and can be used to protect against data theft in the event of a system breach.</p><p style="text-align: justify;">7) IPSec: IPSec is a protocol suite for securing IP communications, including VPNs and other types of network connections. The Linux kernel includes support for IPSec, which can be used to secure network communications between Linux systems.</p><p style="text-align: justify;">Linux kernel has a wide range of built-in security features that can help to protect against various types of security threats. Linux-based security offers a wide range of benefits, including customizability, community-driven development, and a long history of use in high-security environments. These factors have helped to make Linux a popular choice for organizations looking to enhance the security of their systems and data.</p><p style="text-align: justify;"><span style="font-size: xx-small;">Reference:</span></p><p style="text-align: justify;"><span style="font-size: xx-small;">https://man7.org/linux/man-pages/man7/kernel_lockdown.7.html</span></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-7252682689246008792023-03-28T22:53:00.008+05:302023-03-29T08:00:37.210+05:30Non-Uniform Memory Access Architecture<p style="text-align: justify;">Non-Uniform Memory Access (NUMA) is a computer memory design used in multiprocessors, where the memory access time depends on the distance between the CPU and the memory. In a NUMA system, each CPU has access to its own local memory as well as remote memory, which can cause performance issues if not managed properly.</p><p>In the Non-Uniform Memory Access (NUMA) architecture, the path from processor to memory is non-uniform. This organization enables the construction of systems with a large number of processors, and hence the association with very large systems. A NUMA system with cache-coherent memory running a single OS image is still an SMP system. A general representation of the NUMA system is shown below.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXvrDzrw84D9P5o5YiOslMysGP89p64UQWazyZO3Kn-qL9xIkC1BF-QzHpJ-9H2aOAfOctewyF8PkX58J42fpV-T2qJUsJCu_O_8L38Mmw7bW7qzej-wzy4wVphfz0GWhLVeaFpkQV_ZjhekJK2o9YbvAfyhwY7ZqJKCsHQ1Ygrykz8GUk4lQu-ujchg/s885/numa2.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="275" data-original-width="885" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXvrDzrw84D9P5o5YiOslMysGP89p64UQWazyZO3Kn-qL9xIkC1BF-QzHpJ-9H2aOAfOctewyF8PkX58J42fpV-T2qJUsJCu_O_8L38Mmw7bW7qzej-wzy4wVphfz0GWhLVeaFpkQV_ZjhekJK2o9YbvAfyhwY7ZqJKCsHQ1Ygrykz8GUk4lQu-ujchg/w640-h198/numa2.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://www.qdpma.com/SystemArchitecture/NUMA.html" target="_blank">NUMA Architecture</a></td></tr></tbody></table><br /><p>The system is comprised of multiple nodes each with 2-4 processors, a memory controller, memory and perhaps IO. There might be a separate node controller, or the MC and NC could be integrated. The nodes could be connected by a shared bus, or may implement a cross-bar.</p><p style="text-align: justify;"> By classifying memory location bases on signal path length from the processor to the memory, latency and bandwidth bottlenecks can be avoided. This is done by redesigning the whole system of processor and chipset. AMD Opteron family was introduced featuring integrated memory controllers with each CPU owning designated memory banks. Each CPU has now its own memory address space. A NUMA optimized operating system such as ESXi allows workload to consume memory from both memory addresses spaces while optimizing for local memory access. Let’s use an example of a two CPU system to clarify the distinction between local and remote memory access within a single system.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigd0vHPoURvxxByUQ1A2-7YF91h-Sg07j7BxXORAb3xONstiimkaC60DuHfEWo4OY23kuc8AcivVoCDXn5K2R4DJ2pHOIuM2xD3LNDdCxulZ42sOIL9LH8FpEK92GrZU4G5roFJTRse1IELJiDsK851FQuUE8LJeYp2JdKWAJ8DWXi3mI4QtuZNW5Urw/s963/mem_NUma.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="412" data-original-width="963" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigd0vHPoURvxxByUQ1A2-7YF91h-Sg07j7BxXORAb3xONstiimkaC60DuHfEWo4OY23kuc8AcivVoCDXn5K2R4DJ2pHOIuM2xD3LNDdCxulZ42sOIL9LH8FpEK92GrZU4G5roFJTRse1IELJiDsK851FQuUE8LJeYp2JdKWAJ8DWXi3mI4QtuZNW5Urw/w640-h274/mem_NUma.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://frankdenneman.nl/2016/07/07/numa-deep-dive-part-1-uma-numa/" target="_blank"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><p style="text-align: justify;">The memory connected to the memory controller of the CPU1 is considered to be local memory. Memory connected to another CPU socket (CPU2)is considered to be foreign or remote for CPU1. Remote memory access has additional latency overhead to local memory access, as it has to traverse an interconnect (point-to-point link) and connect to the remote memory controller. As a result of the different memory locations, this system experiences “non-uniform” memory access time.</p><p><b>HP Prema:</b></p><p>A node comprises four processors, memory, IO and a pair of node controllers. There are three IOH devices in the system. The processors and memory on the CPU board are connected to the XNC board with the node controllers.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOTfWZY0fiiiyDpPz3od_ob5CQLxV3heK1jk_z4-K_9njM_Wt2o1Z3fwYhB5KO6IgcQymgowgJ2FL5RiazvOA-cQhqv99oWBBDH11xkB1EQOEAmwNrHn-vlc8EIQAnGtydpr2QSgxmzXoa42EUTO_hiUiMvL-9a-Mhqb9-bhVUVwWeQ2nHMj8j05pKnQ/s888/HP_prema.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="361" data-original-width="888" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOTfWZY0fiiiyDpPz3od_ob5CQLxV3heK1jk_z4-K_9njM_Wt2o1Z3fwYhB5KO6IgcQymgowgJ2FL5RiazvOA-cQhqv99oWBBDH11xkB1EQOEAmwNrHn-vlc8EIQAnGtydpr2QSgxmzXoa42EUTO_hiUiMvL-9a-Mhqb9-bhVUVwWeQ2nHMj8j05pKnQ/w640-h260/HP_prema.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://www.qdpma.com/SystemArchitecture/NUMA.html" target="_blank"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><br /><p>The "isolcpus" kernel parameter is used to isolate one or more CPUs from the kernel scheduler. This is typically used for running real-time or high-performance applications that require dedicated CPU resources. However, it does not have any direct relationship with NUMA nodes.</p><p>vi /etc/default/grub</p><p>Find the line that starts with "GRUB_CMDLINE_LINUX_DEFAULT" and add "numa=off" to the end of the line:</p><p>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash numa=off"</p><p>Regenerate the GRUB configuration file by running the following command:</p><p>update-grub</p><p>grub2-mkconfig -o /boot/grub2/grub.cfg</p><p>Reboot the system for the changes to take effect</p><p>Note that after disabling NUMA, the system will treat all memory as a single, uniform memory pool. This may not always improve performance</p><p>---------------</p><p>Certainly, here's an example of how to use the "isolcpus" command to isolate CPU cores from the kernel scheduler:</p><p>Find out the number of available CPU cores by running:</p><p>cat /proc/cpuinfo | grep processor | wc -l</p><p>To isolate one or more CPU cores from the kernel scheduler, append the "isolcpus" parameter to the kernel boot command in the GRUB configuration file. For example, to isolate CPU core 0, edit the GRUB configuration file by running:</p><p>/etc/default/grub</p><p>Add the "isolcpus" parameter followed by the CPU core number(s) to the end of the "GRUB_CMDLINE_LINUX_DEFAULT" line. For example:</p><p>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=0"</p><p>If you want to isolate multiple cores, separate them with commas. For example:</p><p>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=0,2"</p><p>save and update-grub</p><p>Reboot the system for the changes to take effect.</p><p>After isolating the specified CPU cores with the "isolcpus" parameter, you can assign them to a specific process using the "taskset" command. For example, to run a process on CPU core 0, run:</p><p><br /></p><p>taskset -c 0 <command></p><p>Note that isolating CPU cores can affect system performance, so it's important to test your application's performance before and after isolating CPU cores to see if it has any effect on performance.</p><p style="text-align: justify;"><span style="background-color: #cccccc;"># cat /proc/cpuinfo | grep processor | wc -l<br />16<br /># taskset -c 0 hostname<br />host1<br /># taskset -c 18 hostname<br />taskset: failed to set pid 11896's affinity: Invalid argument<br /># taskset -c 16 hostname<br />taskset: failed to set pid 12131's affinity: Invalid argument<br /># taskset -c 15 hostname<br />host1<br /># taskset -c 11,15 hostname<br />host1<br /> #</span></p><p>on PPC arch : /etc/grub.conf</p><p>Find the line that starts with "append" and add "isolcpus" parameter followed by the CPU core number(s) to the end of the line. For example</p><p>append="quiet splash isolcpus=0,2"</p><p>Ater isolating the specified CPU cores with the "isolcpus" parameter, you can assign them to a specific process using the "taskset" command</p><p>check your kernel supports isolcpus:</p><p>grep -i isolcpus /boot/config-$(uname -r)</p><p>This command searches for the "isolcpus" parameter in the kernel configuration file for the currently running kernel.</p><p>If the output of the command shows a line that looks like this:</p><p>CONFIG_ISOLCPU_PROC=y</p><p>then your kernel supports isolating CPU cores with the "isolcpus" parameter.</p><p>Some Linux distributions may not include the kernel configuration file in the /boot directory. In that case, you may need to install the "kernel-devel" or "kernel-source" package to access the kernel configuration file</p><p>NOTE: Another way of making CPU offline:<br />echo 0 > /sys/devices/system/cpu/cpu7/online (to offline cpu)<br />change cpu number with required cpu</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPEXiJolsmRGuuUf4EVo7RfeIkZ7uExXmSCMNhSYRrDvE2eBlm8zmH4pD1tL-o0ENZmMpL70BG5U0C_W-Lc_c5K_CqRlG-PL3r5YG9D9DYnnbChLkvmLNssN2Gv3bq062d60pD7A5o2VOqZjBgBv1NFDX_aCXy1DOeCJ-LfIV-cTNwsWFiu0Uw5FsuJg/s827/numa3.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="475" data-original-width="827" height="368" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPEXiJolsmRGuuUf4EVo7RfeIkZ7uExXmSCMNhSYRrDvE2eBlm8zmH4pD1tL-o0ENZmMpL70BG5U0C_W-Lc_c5K_CqRlG-PL3r5YG9D9DYnnbChLkvmLNssN2Gv3bq062d60pD7A5o2VOqZjBgBv1NFDX_aCXy1DOeCJ-LfIV-cTNwsWFiu0Uw5FsuJg/w640-h368/numa3.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://www.qdpma.com/SystemArchitecture/NUMA.html" target="_blank"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><p><b>Simultaneous multithreading (SMT)</b> is a processor design that combines hardware multithreading with superscalar processor technology. Simultaneous multithreading can use multiple threads to issue instructions each cycle.</p><p>Example: How enable SMT and check on power architecture(PPC): </p><p style="text-align: left;"><span style="background-color: #cccccc;"># cat smt.sh<br />while [ 1 ]<br />do<br /> ppc64_cpu --smt=off<br /> ppc64_cpu --smt<br /> ppc64_cpu --smt=on<br /> ppc64_cpu --smt<br /> ppc64_cpu --smt=2<br /> ppc64_cpu --smt<br /> ppc64_cpu --smt=4<br /> ppc64_cpu --smt<br />done</span></p><p>-----------------------------------END--------------------------------------------------------------</p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-28756929741098693892023-02-23T22:51:00.018+05:302023-02-25T22:47:31.743+05:30High Performance Network Adapters and protocols <p style="text-align: justify;">High performance network adapters are designed to provide fast and efficient data transfer between servers, storage systems, and other devices in a data center or high-performance computing environment. They typically offer advanced features such as high bandwidth, low latency, RDMA support, and offload capabilities for tasks such as encryption and compression. These adapters are often used in high-performance computing, cloud computing, and data center environments to support large-scale workloads and high-speed data transfer. Some examples of high-performance network adapters include:</p><p style="text-align: justify;"></p><ul><li>Mellanox ConnectX-6 and ConnectX-6 Dx</li><li>Intel Ethernet Converged Network Adapter X710 and X722</li><li>Broadcom BCM957810A1008G Network Adapter</li><li>QLogic QL45212HLCU-CK Ethernet Adapter</li><li>Solarflare XtremeScale X2522/X2541 Ethernet Adapter</li><li>Chelsio T6 and T6E-CR Unified Wire Adapters</li></ul><p></p><p style="text-align: justify;">High-performance network adapters typically use specialized protocols that are designed to provide low-latency and high-bandwidth communication between systems. Some examples of these protocols include:</p><p style="text-align: justify;"></p><ol><li>Remote Direct Memory Access (RDMA): A protocol that allows data to be transferred directly between the memory of one system and another, without involving the CPU of either system.</li><li>RoCE (RDMA over Converged Ethernet): An extension of RDMA that allows RDMA traffic to be carried over Ethernet networks.</li><li>iWARP: A protocol that provides RDMA capabilities over standard TCP/IP networks.</li><li>InfiniBand: A high-speed interconnect technology that provides extremely low-latency and high-bandwidth communication between systems.</li></ol><p></p><p style="text-align: justify;">These protocols are typically used in high-performance computing (HPC) environments, where low-latency and high-bandwidth communication is critical for achieving maximum performance. They are also used in other applications that require high-speed data transfer, such as machine learning, data analytics, and high-performance storage systems. Some examples of adapter features include:</p><p style="text-align: justify;"></p><ul><li>Advanced offloading capabilities: High-performance adapters can offload CPU-intensive tasks such as packet processing, encryption/decryption, and compression/decompression, freeing up server resources for other tasks.</li><li>Low latency: Many high-performance adapters are designed to minimize latency, which is especially important for applications that require fast response times, such as high-frequency trading, real-time analytics, and scientific computing.</li><li>Scalability: Some adapters support features such as RDMA and SR-IOV, which allow multiple virtual machines to share a single adapter while maintaining high performance and low latency.</li><li>Security: Many high-performance adapters have hardware-based security features such as secure boot, secure firmware updates, and hardware-based encryption/decryption, which can help protect against attacks and data breaches.</li><li>Management and monitoring: High-performance adapters often come with tools for monitoring and managing network traffic, analyzing performance, and troubleshooting issues.</li></ul><p></p><p style="text-align: justify;"><span style="background-color: #f7f7f8; color: #374151; text-align: justify; white-space: pre-wrap;">A network adapter, also known as a network interface card (NIC), is a hardware component that allows a computer or other device to connect to a network. It typically includes a connector for a cable or antenna, as well as the necessary electronics to transmit and receive data over the network. Network adapters can be internal, installed inside the computer or device, or external, connected via USB or other ports. They are used for wired or wireless connections and support different types of networks such as Ethernet, WiFi, Bluetooth, and cellular networks.</span></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNjd6DyhrVIZ1rQlqXHq0iMBpQG3TWD7i-OcmWIL_0Pd9GEtj54DzdpiJNZxr7ylt4HI0qGDgyo0H0g0Dqp7rjQgUb7s7XZaDNYO3_uQdI1wV5SHMNFQnhdOWgv69gFrO2N2aHD4_u7092u8X5ix2RShMuTAM6Ob9h391wJ9DxhVOS28XHupO7pgECMQ/s1165/networkCard1.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="682" data-original-width="1165" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNjd6DyhrVIZ1rQlqXHq0iMBpQG3TWD7i-OcmWIL_0Pd9GEtj54DzdpiJNZxr7ylt4HI0qGDgyo0H0g0Dqp7rjQgUb7s7XZaDNYO3_uQdI1wV5SHMNFQnhdOWgv69gFrO2N2aHD4_u7092u8X5ix2RShMuTAM6Ob9h391wJ9DxhVOS28XHupO7pgECMQ/w640-h374/networkCard1.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.howtogeek.com/764894/what-is-a-network-adapter/" target="_blank"><span style="font-size: xx-small;">source</span></a></td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;"><span style="background-color: #f7f7f8; color: #374151; text-align: justify; white-space: pre-wrap;"></span></div><span style="background-color: #f7f7f8; color: #374151; text-align: justify; white-space: pre-wrap;"><br /></span><p></p><span style="color: #374151;"><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">A host bus adapter (HBA) is a hardware component that connects a server or other device to a storage area network (SAN). It is responsible for managing the flow of data between the server and the storage devices on the SAN. HBAs typically include a connector for a Fibre Channel or iSCSI cable, as well as the necessary electronics to transmit and receive data over the SAN. They are used to connect servers to storage devices such as disk arrays, tape libraries, and other storage systems.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><br /></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">Common Network Protocols Used in Distributed Storage:
<ol><li><span style="background-color: #f7f7f8; white-space: pre-wrap;"><span>IB: used for the front-end storage network in the DPC scenario.</span></span></li><li><span style="background-color: #f7f7f8; white-space: pre-wrap;"><span>RoCE: used for the back-end storage network.</span></span></li><li><span style="background-color: #f7f7f8; white-space: pre-wrap;"><span>TCP/IP: used for service network.</span></span></li></ol></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
Network adapters are used to connect a computer or device to a network, while host bus adapters are used to connect a computer or device to a storage area network. Network adapters are used for data transmission over networks, while host bus adapters are used for data transmission over storage area networks. There are several network adapters that are commonly used in servers, and the best option will depend on the specific needs of the server and the network it will be connecting to. Some popular options include:</span></div><div style="text-align: justify;"><ol><li><span style="background-color: #f7f7f8; white-space: pre-wrap;">Intel Ethernet Converged Network Adapter X520-DA2: This is a 10 Gigabit Ethernet adapter that is designed for use in data center environments. It supports both copper and fiber connections and is known for its high performance and reliability.</span></li><li><span style="background-color: #f7f7f8; white-space: pre-wrap;">Mellanox ConnectX-4 Lx EN: This is another 10 Gigabit Ethernet adapter that is designed for use in data centers. It supports both copper and fiber connections and is known for its low latency and high throughput.</span></li><li><span style="background-color: #f7f7f8; white-space: pre-wrap;">Broadcom BCM57416 NetXtreme-E: This is a 25 Gigabit Ethernet adapter that is designed for use in data centers. It supports both copper and fiber connections and is known for its high performance and reliability.</span></li><li><span style="background-color: #f7f7f8; white-space: pre-wrap;">Emulex LPe1605A: This is a 16 Gbps Fibre Channel host bus adapter (HBA) that is designed for use in storage area networks (SANs). It supports both N_Port ID Virtualization (NPIV) and N_Port Virtualization (NPV) and is known for its high performance and reliability.</span></li></ol><span style="background-color: #f7f7f8; white-space: pre-wrap;"></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
IBM produces a wide range of servers for various types of environments, here are a few examples of IBM servers:</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8;"><ol style="white-space: pre-wrap;"><li><span>IBM Power Systems: These servers are designed for high-performance computing and big data workloads, and are based on the Power architecture. They support IBM's AIX, IBM i, and Linux operating systems.</span></li><li><span>IBM System x: These servers are designed for general-purpose computing and are based on the x86 architecture. They support a wide range of operating systems, including Windows and Linux.</span></li><li><span>IBM System z: These servers are designed for mainframe computing and support IBM's z/OS and z/VM operating systems.</span></li><li><span>IBM BladeCenter: These servers are designed for blade server environments and support a wide range of operating systems, including Windows and Linux.</span></li><li><span>IBM Storage: These servers are designed for storage and data management workloads, and support a wide range of storage protocols and operating systems.</span></li><li><span>IBM Cloud servers: IBM Cloud servers are designed for cloud-based computing and are based on the x86 architecture. They support a wide range of operating systems, including Windows and Linux.</span></li></ol><span style="white-space: pre-wrap;"><br /></span></span><span style="background-color: #f7f7f8; white-space: pre-wrap;">Emulex Corporation Device e228 is a network adapter produced by Emulex Corporation. It is an Ethernet controller, which means it is responsible for controlling the flow of data packets over an Ethernet network. The Emulex Corporation Device e228 is part of the Emulex OneConnect family of network adapters, which are designed for use in data center environments. These adapters are known for their high performance, low latency, and high throughput. They also provide advanced features such as virtualization support, Quality of Service (QoS) and offloads (TCP/IP, iSCSI, and FCoE) to improve network performance. It supports 10Gbps Ethernet and can be used in both copper and fiber connections. This adapter is typically used in servers and storage systems that require high-speed network connections and advanced features to support data-intensive applications. The "be2net" kernel driver is a Linux device driver that is used to control the Emulex Corporation Device e228 network adapter. A kernel driver is a type of low-level software that interfaces with the underlying hardware of a device, such as a network adapter. It provides an interface between the hardware and the operating system, allowing the operating system to communicate with and control the device. The "be2net" driver is specifically designed to work with the Emulex Corporation Device e228 network adapter, and is responsible for managing the flow of data packets between the device and the operating system. It provides the necessary functionality for the operating system to access the adapter's features and capabilities, such as configuring network settings, monitoring link status and performance, and offloading network processing tasks. The be2net driver is typically included with the Linux operating system and it's loaded automatically when the device is detected. It's also available as a separate package, that can be installed and configured manually.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8;"><span style="white-space: pre-wrap;"><br /></span></span><span style="background-color: #f7f7f8; white-space: pre-wrap;">The Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] is a network adapter produced by Mellanox Technologies. It is an Ethernet controller, which means it is responsible for controlling the flow of data packets over an Ethernet network. This adapter is part of the Mellanox ConnectX-5 Ex family of network adapters, which are designed for use in data center environments. These adapters are known for their high performance, low latency, and high throughput. They support 100 Gbps Ethernet, RoCE v2 and InfiniBand protocols and provide advanced features such as virtualization support, Quality of Service (QoS), and offloads to improve network performance. It's worth noting that the Mellanox ConnectX-5 Ex Virtual Function is a specific type of adapter that is designed to be used in virtualized environments. It allows multiple virtual machines to share a single physical adapter, thus providing better flexibility and resource utilization. This adapter is typically used in servers, storage systems, and other high-performance computing devices that require high-speed network connections and advanced features to support data-intensive applications such as big data analytics, machine learning, and high-performance computing.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">The Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function] and the Emulex Corporation Device e228 are both network adapters, but there are some key differences between them:
Speed and protocol support: The Mellanox ConnectX-5 Ex supports 100 Gbps Ethernet, RoCE v2 and InfiniBand protocols, while the Emulex Device e228 supports 10 Gbps Ethernet. This means that the Mellanox adapter is capable of higher data transfer speeds and can support multiple protocols for different types of networks.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">Advanced features: Both adapters offer advanced features such as virtualization support, Quality of Service (QoS), and offloads. However, the Mellanox ConnectX-5 Ex also supports features like hardware-based time stamping, hardware-based packet filtering and dynamic rate scaling.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
Target market: The Mellanox ConnectX-5 Ex is designed for use in data center environments, while the Emulex Device e228 is also designed for data center environments. Mellanox ConnectX-5 Ex is more geared towards high-performance computing and big data analytics, while the Emulex Device e228 is more geared towards general data center use.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">Virtualization: Mellanox ConnectX-5 Ex Virtual Function is a specific type of adapter that is designed to be used in virtualized environments, allowing multiple virtual machines to share a single physical adapter, thus providing better flexibility and resource utilization. Emulex Device e228 supports virtualization, but it does not have a specific virtual function version. </span><span style="background-color: #f7f7f8; white-space: pre-wrap;">In summary, the Mellanox ConnectX-5 Ex is a high-speed, high-performance network adapter that offers advanced features and support for multiple protocols, while the Emulex Device e228 is a lower-speed, general-purpose network adapter that is geared towards data center environments.</span></div><div style="text-align: justify;"><span style="white-space: pre-wrap;"><br /></span></div><div style="text-align: justify;"><div><span style="background-color: #f7f7f8; white-space: pre-wrap;">Mellanox Technologies produces networking equipment, including network adapters. Some of the Mellanox adapters that support CNA (Converged Network Adapter) are:</span></div><div><ol><li><span style="background-color: #f7f7f8; white-space: pre-wrap;">Mellanox ConnectX-5 CNA: This adapter supports both Ethernet and Fibre Channel over Ethernet (FCoE) on a single adapter, and provides high-performance, low-latency data transfer.</span></li><li><span style="background-color: #f7f7f8; white-space: pre-wrap;">Mellanox ConnectX-6 CNA: This adapter supports 100 GbE and 200 GbE speeds and provides hardware offloads for RoCE, iWARP and TCP/IP, in addition to supporting FC and FCoE protocols.</span></li><li><span style="background-color: #f7f7f8; white-space: pre-wrap;">Mellanox ConnectX-5 EN CNA: This adapter supports both Ethernet and InfiniBand protocols, providing high-performance, low-latency data transfer for data center and high-performance computing environments.</span></li><li><span style="background-color: #f7f7f8; white-space: pre-wrap;">Mellanox ConnectX-6 Lx CNA: This adapter supports 25 GbE and 50 GbE speeds, and provides hardware offloads for RoCE, iWARP, and TCP/IP, in addition to supporting FC and FCoE protocols.</span></li></ol></div><span style="background-color: #f7f7f8; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhO76T3xl6vgW75sB6IIiG8qlOeSdCWWfw-oQIDJloPid0lwWWVjYILV8JFAqhLyJj5C8DJ7QJUsz2rG7vS9TA6-XSl4EsaT0Gi1ebPoTcw2833GHFBnDTxM3ajBrCRR3BNz_cQQky3Ke5TnAjxq9MlgkMe8VO2U_uiKc9hosAN3lImsNa1CjdHB7M3nA/s697/connectx_6_nvidia.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="452" data-original-width="697" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhO76T3xl6vgW75sB6IIiG8qlOeSdCWWfw-oQIDJloPid0lwWWVjYILV8JFAqhLyJj5C8DJ7QJUsz2rG7vS9TA6-XSl4EsaT0Gi1ebPoTcw2833GHFBnDTxM3ajBrCRR3BNz_cQQky3Ke5TnAjxq9MlgkMe8VO2U_uiKc9hosAN3lImsNa1CjdHB7M3nA/w400-h260/connectx_6_nvidia.png" width="400" /></a></div><br /></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><b>Slingshot</b> is a high-performance network fabric developed by the company Cray, now owned by Hewlett Packard Enterprise. It is designed to provide low-latency and high-bandwidth communication between nodes in high-performance computing systems, such as supercomputers and data centers. It is based on a packet-switched network architecture, with each node connected to a network switch. It supports a range of network topologies, including fat-tree, hypercube, and dragonfly. The fabric is designed to be scalable, with support for thousands of nodes. It uses a range of advanced features to optimize performance, including adaptive routing, congestion control, and quality-of-service (QoS) mechanisms. It also includes support for features such as remote direct memory access (RDMA) and messaging passing interface (MPI) offload, which can further improve application performance. Overall, Slingshot is designed to provide high-performance, low-latency communication for demanding HPC workloads, making it a popular choice for large-scale scientific simulations, data analytics, and other compute-intensive applications.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><br /></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><b>RDMA Types</b>
As discussed before , there are three types of RDMA networks: Infiniband, RDMA over Converged Ethernet (RoCE), and iWARP.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBi0DBA3nhsaUo7rQMLY_wriQ-T-e7r7oYQ-oVtBHRHxpSFjHwMh16ko0HNWsRcYH3EYo7eLbgD-MElIN1zgbThdU9oYD7nPRumReP9SGzAUsWJKP7NeBqE5-yt9T1slbElMvtMvtb3cwXhqzG_CDe7pk2pgT2Uyeo71ocH27fgEXmV73MXO4w3_atUw/s657/rdma_traditinal4.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="360" data-original-width="657" height="219" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBi0DBA3nhsaUo7rQMLY_wriQ-T-e7r7oYQ-oVtBHRHxpSFjHwMh16ko0HNWsRcYH3EYo7eLbgD-MElIN1zgbThdU9oYD7nPRumReP9SGzAUsWJKP7NeBqE5-yt9T1slbElMvtMvtb3cwXhqzG_CDe7pk2pgT2Uyeo71ocH27fgEXmV73MXO4w3_atUw/w400-h219/rdma_traditinal4.png" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://support.huawei.com/enterprise/en/doc/EDOC1100203339"><span style="font-size: xx-small;">source</span></a></td></tr></tbody></table>
The InfiniBand network is specially designed for RDMA to ensure reliable transmission at the hardware level. The technology is advanced, but the cost is high. RoCE and iWARP are both Ethernet-based RDMA technologies, which enable RDMA with high speed, ultra-low latency, and extremely low CPU usage to be deployed on the most widely used Ethernet.</span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjNVcqx4FwAtAjpNP1tSd825rIP95Bc4nqy8v7GeZohKQFSjrBTEZ0FwMCpl74zs8Iv-Np8WDz3B5nlkRSw1w1VIhNU7-3W2qjJdeKCdafOGcPbYG7rACLMxRgBTE3VXZWOaQ8twsX2ctM2-z2AyMYN8hVYFOkNRZ88qTCAsXGGQUTENckC5vkAWi23A/s962/traditional-vs-rdma.png" style="margin-left: 1em; margin-right: 1em; text-align: center; white-space: normal;"><img border="0" data-original-height="584" data-original-width="962" height="243" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjNVcqx4FwAtAjpNP1tSd825rIP95Bc4nqy8v7GeZohKQFSjrBTEZ0FwMCpl74zs8Iv-Np8WDz3B5nlkRSw1w1VIhNU7-3W2qjJdeKCdafOGcPbYG7rACLMxRgBTE3VXZWOaQ8twsX2ctM2-z2AyMYN8hVYFOkNRZ88qTCAsXGGQUTENckC5vkAWi23A/w400-h243/traditional-vs-rdma.png" width="400" /></a></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><a href="https://core.vmware.com/resource/basics-remote-direct-memory-access-rdma-vsphere#section2" style="text-align: center; white-space: normal;" target="_blank"><span style="font-size: xx-small;">source</span></a></span></div><div style="text-align: justify;"><div>The three RDMA networks have the following characteristics:</div><div><ol><li>InfiniBand: RDMA is considered at the beginning of the design to ensure reliable transmission at the hardware level and provide higher bandwidth and lower latency. However, the cost is high because IB NICs and switches must be supported.</li><li>RoCE: RDMA based on Ethernet consumes less resources than iWARP and supports more features than iWARP. You can use common Ethernet switches that support RoCE NICs.</li><li>iWARP: TCP-based RDMA network, which uses TCP to achieve reliable transmission. Compared with RoCE, on a large-scale network, a large number of TCP connections of iWARP occupy a large number of memory resources. Therefore, iWARP has higher requirements on system specifications than RoCE. You can use common Ethernet switches that support iWARP NICs.</li></ol></div></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjE-BNB-09wba2InnV0kyNvh1awO1ulI8eBAL81POEVcgX8X9CxxKVPLIQ4qLMLpb6Cj-CVoKmLwYS3hqmOHKCnaDAgJb3AtO-ER-3w-yJ73TMxlGOlZ8qTpKpM2Ws29t4BVCwsBFSc_VGKCJbFKPwgX1xczdgeXSDEtfsSloTMf2_AfdmpGFvcfvx9pQ/s1377/comparison_ROCE_infi.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="297" data-original-width="1377" height="138" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjE-BNB-09wba2InnV0kyNvh1awO1ulI8eBAL81POEVcgX8X9CxxKVPLIQ4qLMLpb6Cj-CVoKmLwYS3hqmOHKCnaDAgJb3AtO-ER-3w-yJ73TMxlGOlZ8qTpKpM2Ws29t4BVCwsBFSc_VGKCJbFKPwgX1xczdgeXSDEtfsSloTMf2_AfdmpGFvcfvx9pQ/w640-h138/comparison_ROCE_infi.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="background-color: white; color: #494949; font-family: LT_regular, Arial; font-size: 14px; text-align: left;"><a href="https://support.huawei.com/enterprise/en/doc/EDOC1100203339" target="_blank">Comparison between RoCE and InfiniBand</a></span></td></tr></tbody></table><br /></span></div><div style="text-align: justify;"><b style="background-color: #f7f7f8; white-space: pre-wrap;">Infiniband is a high-performance, low-latency interconnect technology</b><span style="background-color: #f7f7f8; white-space: pre-wrap;"> used to connect servers, storage, and other data center equipment. It uses a switched fabric topology and supports both data and storage traffic. InfiniBand adapters are specialized network interface cards (NICs) that are designed to work with InfiniBand networks. Here are a few examples of InfiniBand adapters:</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><div style="white-space: normal;"><span style="white-space: pre-wrap;">
</span><span style="white-space: pre-wrap;">Mellanox ConnectX-4/5: These adapters support both 40 Gb/s and 100 Gb/s InfiniBand and provide high-performance, low-latency data transfer for data center and high-performance computing environments.</span></div><div style="white-space: normal;"><span style="white-space: pre-wrap;">
Mellanox ConnectX-6: This adapter supports 200 Gb/s InfiniBand, providing hardware offloads for RoCE, iWARP and TCP/IP, in addition to supporting FC and FCoE protocols.</span></div><div style="white-space: normal;"><span style="white-space: pre-wrap;">
</span><span style="white-space: pre-wrap;">Intel Omni-Path Architecture (OPA) 100 Series: This adapter supports 100 Gb/s InfiniBand and provides high-performance, low-latency data transfer for data center and high-performance computing environments.</span></div><div style="white-space: normal;"><span style="white-space: pre-wrap;">
Qlogic InfiniPath HTX: This adapter supports 10 Gb/s InfiniBand and provides high-performance, low-latency data transfer for data center and high-performance computing environments.</span></div><div style="white-space: normal;"><span style="white-space: pre-wrap;">
</span><span style="white-space: pre-wrap;">Mellanox ConnectX-4 Lx: This adapter supports 25 Gb/s and 50 Gb/s InfiniBand and provides hardware offloads for RoCE, iWARP, and TCP/IP, in addition to supporting FC and FCoE protocols.</span></div><div><span style="white-space: pre-wrap;"><br /></span></div></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;"><b>RoCE (RDMA over Converged Ethernet)</b> is a network protocol that allows for low-latency, high-throughput data transfer over Ethernet networks. It </span><span style="background-color: #f7f7f8; white-space: pre-wrap;">leverages Remote Direct Memory Access (RDMA) capabilities to accelerate communications between applications hosted on clusters of servers and storage arrays. </span><span style="background-color: #f7f7f8; white-space: pre-wrap;">It is based on the Remote Direct Memory Access (RDMA) protocol, which allows for direct memory access over a network without involving the CPU, resulting in low-latency and high-bandwidth data transfer. RoCE uses standard Ethernet networks and devices, so it is simpler to set up and manage than traditional RDMA over Infiniband. RoCE is designed for use in data center environments, and is particularly well-suited for use with high-performance computing and big data analytics applications, which require high-speed, low-latency data transfer. Some features of RoCE are:</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
<ul><li><span>Low-latency: RoCE allows for very low-latency data transfer, which is critical for high-performance computing and big data analytics applications.</span></li><li><span>High-throughput: RoCE allows for high-bandwidth data transfer, which is necessary for handling large amounts of data.</span></li><li><span>RDMA support: RoCE is based on the RDMA protocol, which allows for direct memory access over a network, resulting in low-latency and high-bandwidth data transfer.</span></li><li><span>Converged Ethernet: RoCE uses standard Ethernet networks and devices, making it simpler to set up and manage than traditional RDMA over Infiniband.</span></li><li><span>Quality of Service (QoS) support: RoCE can provide Quality of Service (QoS) feature, which allows for guaranteed bandwidth and low-latency for critical applications.</span></li><li><span>Virtualization support: RoCE can be used with virtualized environments, allowing multiple virtual machines to share a single physical adapter, thus providing better flexibility and resource utilization.</span></li></ul></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><b>RoCE Overview</b>
RDMA over Converged Ethernet (RoCE) is a network protocol that leverages Remote Direct Memory Access (RDMA) capabilities to accelerate communications between applications hosted on clusters of servers and storage arrays. RoCE incorporates the IBTA RDMA semantics to allow devices to perform direct memory-to-memory transfers at the application level without involving the host CPU. Both the transport processing and the memory translation and placement are performed by the hardware which enables lower latency, higher throughput, and better performance compared to software-based protocols.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><br /></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><b>Infiniband RDMA to RoCE : </b></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
Both InfiniBand RDMA and RoCE can implement remote memory access network protocols. The two currently have their own advantages and disadvantages on the market, and both are used in HPC cluster architecture or large-scale data centers.
Comparing the two, InfiniBand has better performance. But InfiniBand is a dedicated network technology. It cannot inherit the user's accumulation and platform of operation on the IP network, which causes the high cost in operation and maintenance. Therefore, carrying RDMA based on the traditional Ethernet network is also inevitable for the large-scale application of RDMA. To guarantee RDMA performance and network layer communication, many network switches use RoCEv2 to carry high-performance distributed applications.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
<b>CNA (Converged Network Adapter) </b>is a type of network adapter that supports multiple protocols, such as Ethernet and Fibre Channel over Ethernet (FCoE), on a single adapter. A CNA typically includes both a NIC and a Host Bus Adapter (HBA) <b>to support both data and storage traffic</b>. </span><span style="background-color: #f7f7f8; white-space: pre-wrap;">When using a CNA with SRIOV (Single Root I/O Virtualization) and ROCE (RDMA over Converged Ethernet), multiple virtual functions (VFs) can be created on the CNA, each with its own MAC address, VLAN ID, and other network attributes. Each VF can be assigned to a different virtual machine (VM) or a container, and each VM or container can have its own network configuration and parameters.
Each VF can be configured to support ROCE, allowing for low-latency, high-throughput data transfer over Ethernet networks. This can be particularly useful in high-performance computing and big data analytics environments, where low-latency and high-bandwidth data transfer is critical.</span></div><div style="text-align: justify;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIOo4m0WkDtm2fUic4cRGG5GndhLaLkkkfh_8q_rQuIG4nwNhR_G4p_MdDZY7HL_ECkJ9rDUCcNf33XuYPZKksUOaedFpv9QnPYrGabmJPd9svz0k6DQ2T-MSmznpa7QuyAe2UzuZAhInL-3Q-M6Ak_04Ctn4UjOXu8H_Z4_LyH-UKYNhPecKioqjglQ/s867/converged_network_adapter.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="542" data-original-width="867" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIOo4m0WkDtm2fUic4cRGG5GndhLaLkkkfh_8q_rQuIG4nwNhR_G4p_MdDZY7HL_ECkJ9rDUCcNf33XuYPZKksUOaedFpv9QnPYrGabmJPd9svz0k6DQ2T-MSmznpa7QuyAe2UzuZAhInL-3Q-M6Ak_04Ctn4UjOXu8H_Z4_LyH-UKYNhPecKioqjglQ/w640-h400/converged_network_adapter.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://en.vcenter.ir/network/converged-network-adapter-cna/" target="_blank"><span face=""Open Sans", sans-serif" style="background-color: white; color: #444444; font-size: 15px; text-align: start;">converged network </span><span face="Open Sans, sans-serif" style="color: #04a3ed;"><span style="background-color: white; border-color: initial; border-image: initial; border-style: initial; box-sizing: border-box; font-size: 15px; text-align: start; transition-duration: 0.3s; transition-property: all; transition-timing-function: ease-in-out;">adapter</span></span><span face=""Open Sans", sans-serif" style="background-color: white; color: #444444; font-size: 15px; text-align: start;"> (CNA)</span></a></td></tr></tbody></table></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;"><b>SRIOV with ROCE on a CNA </b>can provide the following benefits:
Improved resource utilization: By allowing multiple VMs or containers to share a single physical adapter, SRIOV with ROCE on a CNA can improve resource utilization and reduce costs.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">Improved network performance: ROCE allows for low-latency, high-throughput data transfer over Ethernet networks, which can improve network performance in high-performance computing and big data analytics environments.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
Fine-grained control of network resources: SRIOV with ROCE on a CNA allows for fine-grained control of network resources, allowing each VM or container to have its own network configuration.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGyEIB3CEN8xKiU4zXnBysyPmOtvkeT4wq8GQyrT_iv4DLGGqOxZuLTqLPsHpaZAhdk4wzFWxZWqUfCg61hg8_6v9TbkY1T7G8scJuXDRXviA5vuRJvgKs3NiKxX9xSWQtahB9mauMoBaEweeV1CkJ2eOX7Ohq3oI0ByFbhRW-J7alu5cnScQ-SKm6nw/s709/SRIOV.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="709" data-original-width="446" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGyEIB3CEN8xKiU4zXnBysyPmOtvkeT4wq8GQyrT_iv4DLGGqOxZuLTqLPsHpaZAhdk4wzFWxZWqUfCg61hg8_6v9TbkY1T7G8scJuXDRXviA5vuRJvgKs3NiKxX9xSWQtahB9mauMoBaEweeV1CkJ2eOX7Ohq3oI0ByFbhRW-J7alu5cnScQ-SKm6nw/w402-h640/SRIOV.png" width="402" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><h3 class="storytitle" style="background-color: white; color: sienna; font-family: "Trebuchet MS", Verdana, "Courier New", sans-serif; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px; padding: 0px 15px; text-align: left;"><span style="font-size: xx-small;"><a href="https://infrastructureadventures.wordpress.com/2010/12/08/io-virtualization-overview-cna-sr-iov-vn-tag-and-vepa/" target="_blank">I/O Virtualization Overview: CNA, SR-IOV</a></span></h3></td></tr></tbody></table>
</span><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbDM00Xp1WAib165CPxgNP5f8YreM0NycYjb-2UhYPlwZ2kOPwBmJKKHlV1_6uorAWsE9DXCfDPpZWIhDksclaskCgQ-tHg07yxTo8IkWed2VnzAfHBFLREMeO3fs_aXPGvNjNVCP22HWtPQYPzEj6joun9nqv4bnFiZ_xNlO18ofrLKmM59JstHC3zw/s1027/table.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="440" data-original-width="1027" height="274" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbDM00Xp1WAib165CPxgNP5f8YreM0NycYjb-2UhYPlwZ2kOPwBmJKKHlV1_6uorAWsE9DXCfDPpZWIhDksclaskCgQ-tHg07yxTo8IkWed2VnzAfHBFLREMeO3fs_aXPGvNjNVCP22HWtPQYPzEj6joun9nqv4bnFiZ_xNlO18ofrLKmM59JstHC3zw/w640-h274/table.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span face=""Open Sans", Arial, Helvetica, sans-serif" style="background-color: white; color: #19191a; font-size: 14px; text-align: justify;"><a href="https://community.fs.com/blog/roce-vs-infiniband-vs-tcp-ip.html" target="_blank">Differences between RoCE, Infiniband RDMA, and TCP/IP.</a></span></td></tr></tbody></table>
The fastest network adapter available today depends on the specific application and the network infrastructure. Generally, there are different types of network adapters that support different speeds and protocols, and each one is suitable for different use cases.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">For example, for data center environments, 100 GbE (gigabit ethernet) adapters are currently considered as the fastest option, providing high-bandwidth and low-latency data transfer. These adapters use the latest technologies such as SFP28 and QSFP28 connectors and support both copper and fiber cabling. Mellanox ConnectX-6, Intel Omnipath and Marvell FastLinQ are some examples of 100 GbE adapters.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
For High-Performance Computing (HPC) and Artificial Intelligence (AI) applications, Infiniband adapters are considered as the fastest option providing low-latency and high-bandwidth data transfer. Mellanox ConnectX-6 HDR and Intel OPA 100 series are examples of 200 Gb/s Infiniband adapters.
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">For storage, Fibre Channel (FC) and Fibre Channel over Ethernet (FCoE) adapters are considered as the fastest option providing low-latency and high-bandwidth data transfer. Mellanox ConnectX-6, Emulex Gen 6 Fibre Channel and Qlogic Gen 6 Fibre Channel are examples of these adapters.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
Supercomputer systems, like the Summit and Sierra, developed by Oak Ridge National Laboratory and Lawrence Livermore National Laboratory, respectively, use a high-performance interconnect technology called Infiniband for their internal communication. Mellanox Technologies is the company that provides the Infiniband adapters and host bus adapters (HBAs) for these supercomputers.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">Summit and Sierra use Mellanox's Connect-IB adapter which supports 100 Gb/s InfiniBand and provides hardware offloads for RoCE, iWARP and TCP/IP. The Connect-IB adapters are designed to handle the high-bandwidth and low-latency requirements of large-scale supercomputing applications.
The Host Bus Adapter (HBA) Mellanox ConnectX-4 Lx is used for these systems. ConnectX-4 Lx is a single-port 100 Gb/s InfiniBand adapter that supports both 25 Gb/s and 50 Gb/s speeds. The adapters provide hardware offloads for RoCE, iWARP, and TCP/IP, in addition to supporting FC and FCoE protocols.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">Frontier is a planned supercomputer that is being developed by Oak Ridge National Laboratory and Cray Inc. It is the world's fastest supercomputer in 2021. According to the information available, Frontier uses high-performance interconnect technology called Slingshot, developed by Cray, for its internal communication. Slingshot is a next-generation interconnect technology that promises to provide low-latency, high-bandwidth, and high-message-rate data transfer.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
In terms of network adapters and host bus adapters (HBAs), the information available is not specific, but it's known that Cray has collaborated with Mellanox Technologies to provide the network interconnect technology for Frontier. This suggests that Mellanox's InfiniBand and/or Slingshot adapters may be used in Frontier.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
</span><span style="background-color: #f7f7f8; white-space: pre-wrap;">A host bus adapter (HBA) is a specialized type of network interface card (NIC) that connects a host computer to a storage area network (SAN). HBAs provide a bridge between the host computer and the storage devices, allowing the host to access and manage the storage devices as if they were locally attached.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">
Here are a few key things to know to get familiar with storage HBAs:
<ul><li><span>Protocols: HBAs support different storage protocols such as Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), and iSCSI. FC and FCoE are commonly used in enterprise environments, while iSCSI is more commonly used in smaller, SMB environments.
Speed: HBAs are available in different speeds, such as 8 Gb/s, 16 Gb/s, and 32 Gb/s. Higher speeds provide faster data transfer and improved performance.
</span></li><li><span>Multi-Path Support: HBAs often support multi-path I/O, which allows multiple paths to the storage devices to be used for failover and load balancing.
Compatibility: HBAs are designed to work with specific operating systems, so it's important to check the compatibility of the HBA with the operating system you are using.
</span></li><li><span>Management and Monitoring: Many HBAs include management and monitoring software that allows administrators to view and configure the HBA's settings, such as Fibre Channel zoning, and to monitor the performance of the HBA and the storage devices it is connected to.
Driver and Firmware: HBA's require driver and firmware to work properly, so it's important to ensure that the HBA has the latest driver and firmware updates installed.
</span></li><li><span>Vendor Support: It's important to consider the vendor support of the HBA, as well as the warranty and technical support options available, as these can be critical factors when choosing an HBA.
Architecture: Some HBAs are based on ASIC (Application-Specific Integrated Circuit) while others on FPGA (Field-Programmable Gate Array) architecture, both have their own pros and cons.</span></li></ul></span><span style="background-color: #f7f7f8; white-space: pre-wrap;">Power10 is the latest generation of IBM's Power Architecture designed for high-performance computing and big data workloads, and is intended to deliver significant performance and efficiency improvements over its predecessor, Power9. </span><span style="background-color: #f7f7f8; white-space: pre-wrap;">Some of the key features of the Power10 architecture include:
<ul><li><span>Higher core count: Power10 processors have a higher core count than Power9 processors, which allows for more parallel processing and improved performance.
</span></li><li><span>Improved memory bandwidth: Power10 processors have more memory bandwidth than Power9 processors, which allows for faster data transfer between the processor and memory.
Enhanced security features: Power10 processors include enhanced security features, such as hardware-enforced memory encryption and real-time threat detection, to protect against cyber-attacks.
</span></li><li><span>Improved energy efficiency: Power10 processors are designed to be more energy efficient than Power9 processors, which can help to reduce power consumption and cooling costs.
Optimized for AI workloads: Power10 processors are optimized for AI workloads and have better support for deep learning and other AI-related tasks.
</span></li><li><span>More flexible and open: Power10 architecture is more flexible and open. It supports more operating systems, and it has more open interfaces and more standard protocols to connect to other devices. Example: IBM Power Systems AC922.
</span></li></ul></span><span style="background-color: #f7f7f8; white-space: pre-wrap;"></span><span style="background-color: #f7f7f8; white-space: pre-wrap;"></span><span style="background-color: #f7f7f8; white-space: pre-wrap;">AI workloads refer to tasks that involve the use of artificial intelligence and machine learning algorithms, such as:</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><ul><li><span>Natural Language Processing (NLP): This includes tasks such as speech recognition, text-to-speech, and machine translation.
</span></li><li><span>Computer Vision: This includes tasks such as image recognition, object detection, and facial recognition.
Predictive analytics: This includes tasks such as forecasting, anomaly detection, and fraud detection.
</span></li><li><span>Robotics: This includes tasks such as navigation, object manipulation, and decision making.
Recommender Systems: This includes tasks such as personalized product recommendations, content recommendations, and sentiment analysis.
</span></li><li><span>Generative Models: These include tasks such as image and video generation, text generation and music generation.
Reinforcement learning: These include tasks such as game playing, decision making and control systems.
</span></li><li><span>Deep Learning: These include tasks such as Image and speech recognition, natural language processing and predictive analytics.
</span></li></ul></span></div></span><span style="color: #374151;"><div style="text-align: justify;"><br /></div></span><span style="color: #374151;"><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">These are just a few examples of AI workloads, there are many more possible applications of AI in various industries such as healthcare, finance, transportation, and manufacturing. As AI technology continues to advance, new possibilities for AI workloads will continue to emerge.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><br /></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">OpenMPI and UCX are both middleware for high-performance computing, but they are not directly connected to adapter design. However, they can utilize hardware-specific features and optimizations of network adapters to improve performance.</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><br /></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">MPI (Message Passing Interface) and AI (Artificial Intelligence) are interrelated because MPI can be used to distribute the computational workload of AI applications across multiple nodes in a distributed computing environment. Many AI algorithms, such as deep learning, machine learning, and neural networks, require a significant amount of computational resources, memory, and data storage. These algorithms can be parallelized and run in a distributed environment using MPI, which allows them to take advantage of the computing power of multiple nodes. MPI can be used to distribute the data and the workload of AI applications across multiple nodes, enabling parallel processing and reducing the time required to complete the computation. This can significantly improve the performance of AI applications and enable researchers to train and optimize more complex models. Moreover, MPI can be integrated with other libraries, such as OpenMP, CUDA, and UCX, to further improve the performance of AI applications. For example, CUDA is a parallel computing platform that enables programmers to use GPUs (Graphics Processing Units) for general-purpose processing, and MPI can be used to distribute the workload across multiple GPUs and nodes. In summary, MPI provides a scalable and efficient way to distribute the computational workload of AI applications across multiple nodes, enabling researchers and developers to build and run more complex models and achieve better performance.
The choice of MPI communication method that is best suited for AI workloads depends on the specific characteristics of the workload and the system architecture. However, some general guidelines can help in selecting the appropriate MPI communication method for AI workloads. For AI workloads that involve large amounts of data, non-blocking point-to-point communication and collective communication methods are generally preferred. Non-blocking point-to-point communication methods, such as MPI_Isend and MPI_Irecv, allow the application to continue processing while the communication is in progress, which can help reduce the overall communication time. Collective communication methods, such as MPI_Allreduce and MPI_Allgather, can also be highly effective in AI workloads, as they enable efficient data sharing and synchronization among multiple nodes. These methods can be used to distribute the workload of an AI application across multiple nodes, enabling parallel processing and reducing the time required to complete the computation. Additionally, the choice of MPI communication method may also depend on the underlying system architecture. For example, on a system with a high-speed interconnect, such as InfiniBand, the use of MPI communication methods that take advantage of the RDMA (Remote Direct Memory Access) capabilities of the interconnect, such as UCX, can provide significant performance benefits. The best MPI communication method for AI workloads depends on the specific characteristics of the workload and the system architecture. However, non-blocking point-to-point communication and collective communication methods are generally preferred for AI workloads that involve large amounts of data, and the use of RDMA-enabled MPI communication methods can provide significant performance benefits on high-speed interconnects.
</span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;"><br /></span></div><div style="text-align: justify;"><span style="background-color: #f7f7f8; white-space: pre-wrap;">The mapping of adapters in supercomputers and network adapters is an important aspect of designing and building a supercomputer. In general, supercomputers use high-performance network adapters that can handle large amounts of data at high speeds, with low latency and high bandwidth. The choice of network adapter depends on the specific requirements of the supercomputer, such as the type and size of data being processed, the number of nodes in the system, and the desired performance characteristics. Some of the common network adapters used in supercomputers include InfiniBand adapters, Ethernet adapters, and Omni-Path adapters. These adapters are typically integrated with the server hardware, either as separate network interface cards (NICs) or as part of the motherboard design. </span>These adapters provide low-latency, high-bandwidth interconnects between nodes in a cluster, enabling parallel computing and large-scale data processing. In addition to high-performance interconnects, HPC also relies on specialized hardware accelerators like GPUs, FPGAs, and ASICs to offload compute-intensive tasks from the CPU and improve overall system performance. These accelerators are often used in combination with high-performance network adapters to enable faster data transfer and processing in HPC environments.</div></span>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-90395374152151046602023-02-23T20:13:00.000+05:302023-02-23T20:13:39.115+05:30IBM offloaded Watson Health assets to investment firm Francisco Partners<p style="text-align: justify;">IBM Watson is a question-answering computer system capable of answering questions posed in natural language, developed in IBM's DeepQA project by a research team led by principal investigator David Ferrucci. Watson was named after IBM's founder and first CEO, industrialist Thomas J. Watson. IBM’s then- CEO Ginni Rometty called the project a “moon shot,” but her replacement was less enthused about the business. The computer system was initially developed to answer questions on the quiz show Jeopardy!.</p><p style="text-align: justify;">IBM launched Watson Health in early 2015 and made a series of acquisitions that cost $4 billion. They included Merge Healthcare, Truven Health Analytics, Phytel, and Explorys. IBM sold Watson Health for $1B, which is 25% of what it paid to acquire four strong businesses. The assets involved include Health Insights, MarketScan, Clinical Development, Social Program Management, Micromedex, and imaging software products. IBM offloaded Watson Health this year because it doesn't have the requisite vertical expertise in the healthcare sector.</p><p style="text-align: justify;">Talking at stock market analyst Bernstein's 38th Annual Strategic Decisions Conference, the big boss was asked to outline the context for selling the healthcare data and analytics assets of the business to private equity provider Francisco Partners for $1 billion in January.</p><p><b> The Watson brand will be carrier for AI.</b></p><p style="text-align: justify;">It's a question of verticals versus horizontals. IBM believes that they are best positioned to take these technologies.They will always have an industry lens but through their consulting team. They want to work on technologies that are horizontal across all industries."</p><p style="text-align: justify;">Verticals should belong to people who really have all of the domain expertise, they have credibility in that vertical. And healthcare companies, people in medical devices, they will have the credibility to carry out how AI is applied to health in depth i.e AI as applied to healthcare, to financial services, to compliance, in that case, regulatory compliance, is going to be a massive market.</p><p style="text-align: justify;">To succeed in health, they need doctors and nurse practitioners to speak to the buyers of Watson Health. That's not the IBM go-to-market field force, so there's a misalignment. Ditto in Promontory, that needed ex-regulators and accountants to go talk to people worrying about financial compliance. So, that's a little bit different than IBM. IBM still sells Watson solutions in financial services, advertising, business automation, and video streaming and hosting. As for AI in the enterprise where inflation, labor costs and the world undergoing a "demographic shift" means that "there are fewer people with the skills" and so AI and automation will be "applied to more and more domains." Trend is going to reverse in the next few decades."</p><p style="text-align: justify;">IBM’s Watson Health is being sold for parts. The technology giant has agreed to sell the division’s data and analytics assets to private equity firm Francisco Partners. Terms weren’t disclosed, although Bloomberg values the deal at more than $1 billion. Launched in 2015, Watson Health’s goal was to revolutionize medicine through AI. After years of pricey expansion — it spent more than $4 billion on acquisitions, per Axios — and reports of ineffectiveness, the unit scaled back its ambitions.</p><p style="text-align: justify;">Once viewed as a flagship of AI in medcine and life science, IBM Watson Health couldn't live up to its ambitious promises to transform everything from drug discovery to cancer care. It would be interesting to see how the new firm who bought this giant from IBM will transform its data and analytic assets and realize their full potential.</p><p><br /></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-46447640877209788262023-02-23T19:54:00.001+05:302023-02-23T19:54:38.409+05:30BIG MPI<p style="text-align: justify;">In order to describe a structured region of memory, the routines in the MPI standard use a (count, datatype) pair. The C specification for this convention uses an int type for the count. Since C int types are nearly always 32 bits large and signed, counting more than 2 power 31 elements poses a challenge. Instead of changing the existing MPI routines, and all consumers of those routines, the MPI Forum asserts that users can build up large datatypes from smaller types. To evaluate this hypothesis and to provide a user-friendly solution to the large-count issue, we have developed BigMPI, a library on top of MPI that maps large count MPI-like functions to MPI-3 standard features. BigMPI demonstrates a way to perform such a construction, reveals shortcomings of the MPI standard, and uncovers bugs in MPI implementations</p><p style="text-align: justify;">References:</p><p style="text-align: justify;">https://www.mcs.anl.gov/papers/P5210-1014.pdf</p><p style="text-align: justify;">https://github.com/jeffhammond/BigMPI</p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-42598022513644588322023-02-23T19:50:00.003+05:302023-02-25T09:44:18.503+05:30MPI [ Message Passing Interfaces] - behind the scenes <p style="text-align: justify;">Parallel computing is accomplished by splitting up large and complex tasks across multiple processors. In order to organize and orchestrate parallel processing, our program must consider automatically decomposing the problem at hand and allowing the processors to communicate with each other when necessary while performing their work. This introduces a new overhead, the synchronization and the communication itself.</p><p style="text-align: justify;">Computing parallelism can be roughly classified as Distributed Memory (DM) or Shared Memory(SM) class. In Distributed Memory (DM), each processor has its own memory which are connected through a network that can exchange data, thus, limiting the DM performance and scalability. In Shared</p><p style="text-align: justify;">Memory (SM), each processor can access all of the memory, resulting in automatic distribution of proce durally iterations over several processors - autoparallelization, explicit distribution of work over the pro cessors by compiler directives or function calls to threading libraries. If this overhead is not accounted. It can create several issues like bottlenecks in the parallel computer design and load imbalances.</p><p style="text-align: justify;">MPI is an API for message passing between entities with separated memory spaces - processes. The standard doesn't care where those processes run - it could be on networked computers (clusters), it could be on a big shared memory machine or it could be any other architecture that provide the same semantics (e.g. IBM Blue Gene)</p><p style="text-align: justify;">OpenMPI is a widely used message passing interface (MPI) library for parallel computing. It provides an abstraction layer that allows application developers to write parallel code without worrying about the underlying hardware details. However, OpenMPI also provides support for hardware-specific optimizations, including those for network adapters. For example, it supports the use of high-speed interconnects such as InfiniBand and RoCE, and it can take advantage of hardware offload capabilities such as Remote Direct Memory Access (RDMA).</p><p style="text-align: justify;">UCX (Unified Communication X) is another middleware library for communication in distributed systems. It is designed to be highly scalable and to support a wide range of hardware platforms, including network adapters. UCX provides a portable API that allows applications to take advantage of hardware-specific features of network adapters, such as RDMA and network offloading. UCX can also integrate with other system-level libraries such as OpenMPI and hwloc to optimize performance on specific hardware configurations.</p><p style="text-align: justify;">Hwloc (Hardware Locality) is a library for topology discovery and affinity management in parallel computing. It provides a portable API for discovering the hierarchical structure of the underlying hardware, including network adapters, and it allows applications to optimize performance by binding threads and processes to specific hardware resources. Hwloc can be used in conjunction with OpenMPI and UCX to optimize communication and data movement on high-performance computing systems.</p><p style="text-align: justify;">TCP/IP is a family of networking protocols. IP is the lower-level protocol that's responsible for getting packets of data from place to place across the Internet. TCP sits on top of IP and adds virtual circuit/connection semantics. With IP alone you can only send and receive independent packets of data that are not organized into a stream or connection. It's possible to use virtually any physical transport mechanism to move IP packets around. For local networks it's usually Ethernet, but you can use anything. There's even an RFC specifying a way to send IP packets by carrier pigeon.</p><p style="text-align: justify;">Sockets is a semi-standard API for accessing the networking features of the operating system. Your program can call various functions named socket, bind, listen, connect, etc., to send/receive data, connect to other computers, and listen for connections from other computers. You can theoretically use any family of networking protocols through the sockets API--the protocol family is a parameter that you pass in--but these days you pretty much always specify TCP/IP. (The other option that's in common use is local Unix sockets.)</p><p style="text-align: justify;">When you are interested to write a parallel programming application, you should probably not be looking at TCP/IP or sockets as those things are going to be much lower level than you want. You'll probably want to look at something like MPI or any of the PGAS languages like UPC, Co-array Fortran, Global Arrays, Chapel, etc. They're going to be far easier to use than essentially writing your own networking layer.</p><p style="text-align: justify;">When you use one of these higher level libraries, you get lots of nice abstractions like collective operations, remote memory access, and other features that make it easier to just write your parallel code instead of dealing with all of the OS stuff underneath. It also makes your code portable between different machines/architectures.</p><p style="text-align: justify;">MPI is free to use any available communication path(s) for MPI messages in the new communicator; the socket is only used for the initial handshaking. </p><p style="text-align: justify;">A common problem is the one of two processes each opening connections to each other. The socket code assume that the sockets are bidirectional, thus only one socket is needed by each pair of connected processes, not one socket for each member of the pair. Then it should refactor the states and state machine into a clear set of VC connection states and connection states.</p><p style="text-align: justify;">There are three related objects used during a connection event. They are the connection itself (a structure specific to the communication method, sockets in the case of this note), the virtual connection, and the process group to which the virtual connection belongs</p><p style="text-align: justify;">If a socket connection between two processes is established, there are always two sides: The connecting side and the accepting side. The connecting side sends an active message to the accepting side. This sides first accepts the connection. However, if both processes try to connect to each other (head-to-head situation), the processes n have both, a connecting and an accepting connection. In this situation, one of the connections is refused/discarded - while the other connection is established. This is decided on the accepting side.</p><p style="text-align: justify;"><b>State machines for establishing a connection:</b></p><p style="text-align: justify;"><b>Connect side :</b></p><p style="text-align: justify;">The connecting side tries to establish a connection by sending an active message to the remote side. If the connection is accepted, the pg_id is send to the remote side. Now, the process waits, until the connection is finally accepted or refused. For this decision, the remote side requires the gp_id . Based on the answer from the remote side (ack = yes or ack = no) the connection is connected or closed.</p><p style="text-align: justify;"><b>Accept side:</b></p><p style="text-align: justify;">The accept side receives a connection request on the listening socket. In the first instance, it accepts the connection an allocates the required structures (conn, socket). Then, the connection waits for the pg_id of the remote side to assign the socket-connection to a VC. The decision, if a connection is accepted or refused, is based on the following steps:</p><p></p><ol style="text-align: left;"><li style="text-align: justify;">The VC has to aktive connection (vc->conn == NULL) : The new connection is accepted</li><li style="text-align: justify;">The VC has an aktive connection</li><li style="text-align: justify;">If my_pg_id < remote_pg_id: accept and discard other connection</li><li style="text-align: justify;">If my_pg_id > remote_pg_id: refuse </li></ol><div style="text-align: justify;"><br /></div><div style="text-align: justify;">The answer is send to the remote note.</div><p></p><p style="text-align: justify;">Data Types Required by the MPI Standard:</p><div class="separator" style="clear: both; text-align: justify;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhCNqKDMp6H_AKO7dELIGc0Z8QhggYRcsKlXsegHvfOTyjkPnlmzeeQEWAKwt2BqIlTSVMgXfofg83vdPY2Cwzscs-SJokiLvJef7roXikiKIiN1Mg3t6RpiYoypBkfV1MqFJad2Oezla4WOu9ZboxV4GTkdJjdPDyfEYRy2snROk4SZ0-LFjr5Wz8AwQ=s892" style="margin-left: 1em; margin-right: 1em;"><img alt="Source" border="0" data-original-height="818" data-original-width="892" height="586" src="https://blogger.googleusercontent.com/img/a/AVvXsEhCNqKDMp6H_AKO7dELIGc0Z8QhggYRcsKlXsegHvfOTyjkPnlmzeeQEWAKwt2BqIlTSVMgXfofg83vdPY2Cwzscs-SJokiLvJef7roXikiKIiN1Mg3t6RpiYoypBkfV1MqFJad2Oezla4WOu9ZboxV4GTkdJjdPDyfEYRy2snROk4SZ0-LFjr5Wz8AwQ=w640-h586" title="source" width="640" /></a></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">MPI point-to-point communication sends messages between two different MPI processes. One process performs a send operation while the other performs a matching read</div><div style="text-align: justify;"><br /></div><div class="separator" style="clear: both; text-align: justify;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhlwlNmkDG399kN6yldRd4nc1KAxZ8THJZjqe2iV4ZxH1s72dl4UHhsBAZenjxOsVGDzpQXiXlQ7Hnss9Z5ko58bGsL1xnoUOs1fg9PzZoevRvcQ36iNqBFalti1AQv2zQ13dob0xssTf1euhrZ3UQ_Zbe-W9e3ktdSvh34V10ieHF5btfiunotJp-evA=s1151" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="287" data-original-width="1151" height="100" src="https://blogger.googleusercontent.com/img/a/AVvXsEhlwlNmkDG399kN6yldRd4nc1KAxZ8THJZjqe2iV4ZxH1s72dl4UHhsBAZenjxOsVGDzpQXiXlQ7Hnss9Z5ko58bGsL1xnoUOs1fg9PzZoevRvcQ36iNqBFalti1AQv2zQ13dob0xssTf1euhrZ3UQ_Zbe-W9e3ktdSvh34V10ieHF5btfiunotJp-evA=w400-h100" width="400" /></a></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">MPI collectives: MPI provides a set of routines for communication patterns that involve all the processes of a certain communicator, so-called collectives. The two main advantages of using collectives are:</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">1) Less programming effort. </div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">2) Performance optimization, as the implementations are usually efficient, especially if optimized for specific architectures</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">For collective communication, significant gains in performance can be achieved by implementing topology- and performance-aware collectives.</div><div style="text-align: justify;"><br /></div><div>Three common blocking collectives are Barrier(), Bcast() and Reduce().</div><div><br /></div><div><ol style="text-align: left;"><li>Allreduce(). Combination of reduction and a broadcast so that the output is available for all processes.</li><li>Scatter(). Split a block of data available in a root process and send different fragments to each process.</li><li>Gather(). Send data from different processes and aggregate it in a root process.</li><li>Allgather(). Similar to Gather() but the output is aggregated in buffers of all the processes.</li><li>Alltoall(). All processes scatter data to all processes.</li></ol></div><div><br /></div><div><br /></div><div><br /></div><p><span style="font-size: xx-small;">Reference:</span></p><p><span style="font-size: xx-small;">https://www.sciencedirect.com/topics/computer-science/point-to-point-communication</span></p><p><span style="font-size: xx-small;">https://wiki.mpich.org/mpich/index.php/Establishing_Socket_Connections</span></p><p><span style="font-size: xx-small;">https://aist-itri.github.io/gridmpi/publications/cluster04-slide.pdf</span></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-47794702280667884842023-02-23T19:36:00.008+05:302023-02-23T19:36:59.531+05:30High performance computing<p> High-Performance Computing (HPC or supercomputer) is omnipresent in today’s society. For example, every time you watch Netflix, the recommendation algorithm leverages HPC resources remotely to offer you personalized suggestions. HPC stands for High-Performance Computing. The ability to carry out large scale computations to solve complex problems, that either need to process a lot of data, or to have a lot of computing power at their disposal. Basically, any computing system that doesn’t fit on a desk can be described as HPC. </p><p>HPC systems are actually networks of processors. The key principle of HPC lies in the possibility to run massively parallel code to benefit from a large acceleration in runtime. A common HPC capability is around 100,000 cores. Most HPC applications are complex tasks which require the processors to exchange their results. Therefore, HPC systems need very fast memories and a low-latency, high-bandwidth communication systems (>100Gb/s) between the processors as well as between the processors and the associated memories.</p><p>We can differentiate two types of HPC systems: the homogeneous machines and the hybrid ones. Homogeneous machines only have CPUs while the hybrids have both GPUs and CPUs. Tasks are mostly run on GPUs while CPUs oversee the computation. </p><p><br /></p><p>They have more computing power since GPUs can handle millions of threads simultaneously and are also more energy efficient. GPUs have faster memories, require less data transfer and are capable to exchange with other GPUs, which is the most energy-intensive part of the machine.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGyWip56XdMWTq2R6FoNmZQo2EnNQSZH4GOmim-zngB2Os1iVYqOJRqy1BYK4zYQjQZTPDXMHI_pnr3OISxL-MFuYq_HNb8a4XAfH4Qc-ot47lqyCqd1boVMmgIIlxJiyi_vH-B24Ae_G8k6El2YVhwRkRX1rk3uDFuUqEK1MTCyY5BoHbMKVLIPvh1g/s1055/HPC-computes.PNG" style="margin-left: auto; margin-right: auto;"><span style="color: black;"><img border="0" data-original-height="635" data-original-width="1055" height="386" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGyWip56XdMWTq2R6FoNmZQo2EnNQSZH4GOmim-zngB2Os1iVYqOJRqy1BYK4zYQjQZTPDXMHI_pnr3OISxL-MFuYq_HNb8a4XAfH4Qc-ot47lqyCqd1boVMmgIIlxJiyi_vH-B24Ae_G8k6El2YVhwRkRX1rk3uDFuUqEK1MTCyY5BoHbMKVLIPvh1g/w640-h386/HPC-computes.PNG" width="640" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://medium.com/quantonation/a-beginners-guide-to-high-performance-computing-ae70246a7af"><span style="color: black;">source</span></a></td></tr></tbody></table><br /><div>High Performance Computing used to be strictly defined with high speed network to allow strong interconnections between cores. The rise of AI applications led to an architecture based on more independent clusters but still massively parallel.</div><div><br /></div><div>HPC systems also include the software stack. That can be divided into three categories. First the user environment encompasses the applications known as workflows. Then the middleware linking applications and their implementation on the hardware. It includes the runtimes and frameworks. Last, the Operating system, at system level with the job scheduler, management software for load balancing and data availability. Its role is to assign tasks to the processors and organize the exchange of data between the processors and the memories to ensure the best performance.</div><div><br /></div><div><br /></div><div><div><b>HPC applications</b></div><div>HPC provides many benefits and value when used for commercial and industrial applications. Applications that can be classified in five categories:</div><div><br /></div><div>- Fundamental research aims to improve scientific theories to better understand natural or other phenomena. HPC enables more advanced simulations leading to breakthrough discoveries.</div><div><br /></div><div>- Design simulation allows industries to digitally improve the design of their products and test their properties. It enables companies to limit prototyping and testing, making the designing process quicker and less expensive.</div><div><br /></div><div>- Behavior prediction enables companies to predict the behavior of a quantity which they can’t impact but depend on, such as the weather or the stock market trends. HPC simulations are more accurate and can look farther into the future thanks to their superior computing abilities. It is especially important for predictive maintenance and weather forecasts.</div><div><br /></div><div>- Optimization is a major HPC use case. It can be found in most professional fields, from portfolio optimization to process optimization, to most manufacturing challenges faced by the industry.</div><div><br /></div><div>HPC is more and more used for data analysis. Business models, industrial processes and companies are being built on the ability to connect, analyze and leverage data, making supercomputers a necessity in analyzing massive amounts of data.</div><div><br /></div></div><div>The 5 fields of HPC Applications.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYdpHYIrQsvpI3IW_-akjnfvFDDO6jIHgRC7baiQjU1EiJJi24nIZ3SAOuWTJDufpbWG4QJlW-4bmTaFuyQB_VHivSRfGAuFsdVULNzLy_eq6WfLVn_kxS-fbPBv_3HbQj1qwYzdQ_Ngayyejl3QysOiaKOWePOnrJUBQL0YePSAxG5-UuMnnQYNu4qQ/s635/1.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="280" data-original-width="635" height="141" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYdpHYIrQsvpI3IW_-akjnfvFDDO6jIHgRC7baiQjU1EiJJi24nIZ3SAOuWTJDufpbWG4QJlW-4bmTaFuyQB_VHivSRfGAuFsdVULNzLy_eq6WfLVn_kxS-fbPBv_3HbQj1qwYzdQ_Ngayyejl3QysOiaKOWePOnrJUBQL0YePSAxG5-UuMnnQYNu4qQ/s320/1.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBd9ZBl6tJvePqBS_HB9h5CXjaKrQLOgOKSK281dA7TO9ZgKRPEhRx-hLk_1APgCtid446AueyMY6HfQ7smy8tLZv6HIRHnSn-veB_1OMcoXLEXBmqfzKtmQUoTFoMcHxA5-OEfoaQSdXWxpOg-oARMLxctgKCEfPpQGWFiCOa2WT6zQRapQqkjQNN3w/s630/2.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="266" data-original-width="630" height="135" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBd9ZBl6tJvePqBS_HB9h5CXjaKrQLOgOKSK281dA7TO9ZgKRPEhRx-hLk_1APgCtid446AueyMY6HfQ7smy8tLZv6HIRHnSn-veB_1OMcoXLEXBmqfzKtmQUoTFoMcHxA5-OEfoaQSdXWxpOg-oARMLxctgKCEfPpQGWFiCOa2WT6zQRapQqkjQNN3w/s320/2.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrB27Mg2X8Dhct3p9CMjheS7Fsc4Hxg46hd897oXMAItCmnnlGCwORDq3AARd6R-k84gAWdfu5_ahM1XravJ3KW94OvabgldJ56z23NoxOPdLyqgoVG3vKYEeO1qHaLXmRFCjsVnWwNEdj5GBReAxU5TsX7iInJKbBZhkNd7iF9HKL87-S7qUwVu2tIQ/s640/3.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="272" data-original-width="640" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrB27Mg2X8Dhct3p9CMjheS7Fsc4Hxg46hd897oXMAItCmnnlGCwORDq3AARd6R-k84gAWdfu5_ahM1XravJ3KW94OvabgldJ56z23NoxOPdLyqgoVG3vKYEeO1qHaLXmRFCjsVnWwNEdj5GBReAxU5TsX7iInJKbBZhkNd7iF9HKL87-S7qUwVu2tIQ/s320/3.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-f0TNkT1qmtTfnq8psAhsmKJ_Y-ynOQXQ4fJg50jRwb_FTt22xiLtMRqQF3qFwwaA7WB5_Xo1XqBak_3CXJrLlMrDnA94olIQCjwc-Usb4WpdYaOfL0dLo9pL08m1TuVbbkejOgBtu8r3A0nQuEFs9c5KU8zuWkIlaLokFXlGm464-l19Sc1M2sd4Dg/s627/4.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="267" data-original-width="627" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-f0TNkT1qmtTfnq8psAhsmKJ_Y-ynOQXQ4fJg50jRwb_FTt22xiLtMRqQF3qFwwaA7WB5_Xo1XqBak_3CXJrLlMrDnA94olIQCjwc-Usb4WpdYaOfL0dLo9pL08m1TuVbbkejOgBtu8r3A0nQuEFs9c5KU8zuWkIlaLokFXlGm464-l19Sc1M2sd4Dg/s320/4.PNG" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjL01t3V4lrS86Y5WWPRy-Eo3yd8d8Z4WhLEBVhwOdb-Um6GwY1aKC9No9MVU-krSSsc048eRPaXTu4B8DBHa47zPjnpcQD-RguP0wqAwxskFbyddY6xqnctoh9p7-L1LUik1uwEQGJT9p7eZ39s66w5QzJlUnQrW461w6Gvfejia32bXmigknsZJbnrw/s633/5.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="273" data-original-width="633" height="138" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjL01t3V4lrS86Y5WWPRy-Eo3yd8d8Z4WhLEBVhwOdb-Um6GwY1aKC9No9MVU-krSSsc048eRPaXTu4B8DBHa47zPjnpcQD-RguP0wqAwxskFbyddY6xqnctoh9p7-L1LUik1uwEQGJT9p7eZ39s66w5QzJlUnQrW461w6Gvfejia32bXmigknsZJbnrw/s320/5.PNG" width="320" /></a></div><br /><div><br /></div><div><div>Another major application for HPC is in the fields of medical and material advancements. For instance, HPC can be deployed to:</div><div><br /></div><div>Combat cancer: Machine learning algorithms will help supply medical researchers with a comprehensive view of the U.S. cancer population at a granular level of detail.</div><div><br /></div><div>Identify next-generation materials: Deep learning could help scientists identify materials for better batteries, more resilient building materials and more efficient semiconductors.</div><div><br /></div><div>Understand patterns of disease: Using a mix of artificial intelligence (AI) techniques, researchers will identify patterns in the function, cooperation, and evolution of human proteins and cellular systems.</div></div><div><br /></div><div>HPC needs are skyrocketing. A lot of sectors are beginning to understand the economic advantage that HPC represents and therefore are developing HPC applications. </div><div><br /></div><div>Industrial companies in the field of aerospace, automotive, energy or defence are working on developing digital twins of a machine or a prototype to test certain properties. This requires a lot of data and computing power in order to accurately represent the behavior of the real machine. This will, moving forward, render prototypes and physical testing less and less standard.</div><div><br /></div><div>The HPC dynamics and industrial landscape</div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGbci252mAd5GyLKFbbUMw6E1NnUrifNx4YpDtHGVRwFnP2ghB820x924gkUCRfY-qznaD1RJN94XhOMtnzq5h2eOPx3Hzfx-Vu5FraTvPQsBqQ0riPeH5sfZ7xlTLlI5ZB4_XlKWe2ILaH-NHomJaJudxDV6ECKF5rrAjfr8Xx7QZzMODmDGTe7OJJg/s1089/hpc-landscape.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="499" data-original-width="1089" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGbci252mAd5GyLKFbbUMw6E1NnUrifNx4YpDtHGVRwFnP2ghB820x924gkUCRfY-qznaD1RJN94XhOMtnzq5h2eOPx3Hzfx-Vu5FraTvPQsBqQ0riPeH5sfZ7xlTLlI5ZB4_XlKWe2ILaH-NHomJaJudxDV6ECKF5rrAjfr8Xx7QZzMODmDGTe7OJJg/w640-h294/hpc-landscape.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://medium.com/quantonation/a-beginners-guide-to-high-performance-computing-ae70246a7af">source</a></td></tr></tbody></table><br /><div><div>The limits of a model :</div><div>Unfortunately, supercomputers are revealing some limits. First of all, some problems are not currently solvable by a supercomputer. The race to the exascale (a supercomputer able to realize 10^18 floating point operations per second) is not necessarily going to solve this issue. Some problems or simulations might remain unsolvable, or at least, unsolvable in an acceptable length of time. For example, in the case of digital twins or molecular simulation, calculations have to be greatly simplified in order for current computers to be able to make them in an acceptable length of time (for product or drug design).</div><div><br /></div><div>Moreover, a second very important challenge is the power consumption. The consumption of computing and data centers represents 1% of power consumption in the world and this is bound to significantly increase. It shows that this model is unsustainable in the long term, especially since exascale supercomputers will most surely consume more than current ones. Not only is it technically unsustainable, it is also financially so. Indeed, a supercomputer can cost as much as 10mUSD per year in electricity consumption.</div><div><br /></div><div>The new chips revolution</div><div>CPUs and GPUs are not the only solutions to tackle the two previously stated issues.</div><div><br /></div><div>Although most efforts are focused on developing higher-performance CPU and GPU-powered supercomputers in order to reach the exascale, new technologies, in particular “beyond Silicon”, are emerging. Innovative chip technologies could act as accelerators like GPUs did in the 2010s and significantly increase the computing power. Moreover, some technologies, such as quantum processors for example, would be able to solve new categories of problems that are currently beyond our reach.</div><div><br /></div><div>In addition, 70% of the energy consumption in a HPC is accounted for by the processors. Creating new chips, more powerful and more energy efficient would enable us to solve both problems at once. GPUs were the first step towards this goal. Indeed, for some applications, GPUs can replace up to 200 or 300 CPUs. Although one GPU individually consumes a bit more than a CPU (400W against 300W approximately), overall, a hybrid supercomputer will consume less than a homogeneous supercomputer of equal performance.</div><div><br /></div><div>The model needs to be reinvented to include disruptive technologies. Homogeneous supercomputers should disappear, and it is already underway. In 2016, only 10 out of the supercomputers in the Top500 were hybrid. By 2020, within only four years, it rose to 333 out of 500, including 6 in the top 10.</div><div><br /></div><div>At Quantonation, are convinced that innovative chips integrated in hetereogeneous supercomputing architectures, as well as optimized softwares and workflows, will be key enablers to face societal challenges by significantly increasing sustainability and computing power. We trust that these teams are ready to face the challenge and be part of the future of compute:</div></div><div><br /></div><div><div><ol style="text-align: left;"><li>Pasqal’s neutral atoms quantum computer, highly scalable and energy efficient;</li><li>Lighton’s Optical Processing Unit, a special purpose ligh-based AI chip fitted for tasks such as Natural Language Processing;</li><li>ORCA Computing’s fiber based photonic systems for simulation and fault tolerant quantum computing;</li><li>Quandela’s photonic qubit sources that will fuel next generation of photonic devices;</li><li>QuBit Pharmaceuticals software suites leveraging HPC and quantum computing resources to accelerate drug discovery ;</li><li>Multiverse Computing’s solutions using disruptive mathematics to resolve finance’s most complex problems on a range of classical and quantum technologies.</li></ol></div></div><div><br /></div><div><br /></div><div><div style="text-align: justify;">IBM Cloud HPC IaaS for building HPC environments using IBM’s Virtual Private Cloud (VPC). It enables you to create your own configuration for Compute Instances; High-performance Storage and Networking like Public Gateways, Load Balancers and Routers. Multiple connectivity options are available upto 80Gbps and IBM Cloud offers the highest level of security and encryption with FIPS 140-2 Level 4. Also available is IBM Code Engine, a fully managed serverless platform to run containers, applications or batch jobs.</div><div style="text-align: justify;">– Spectrum Computing provides intelligent dynamic hybrid cloud capabilities which enables organizations to use cloud resources according to defned policies. Spectrum LSF and Symphony allows you to burst workloads to the cloud, dynamically provision cloud resources and intelligently move data to manage egress costs. It also enables the ability for auto scaling to take full advantage of consumption-based pricing and pay for cloud resources only when they are needed.</div><div style="text-align: justify;">– Spectrum Scale is an enterprise grade High Performance File System (HPFS) that delivers scalable capacity and performance to handle demanding data analytics, content repositories and HPC workloads. Spectrum Scale architecture allows it to handle tens of thousands of clients, billions of fles and petabytes of data written and retrieved as fles or objects with low latency. Optionally, IBM Aspera can be used for high speed data movement using the FASP protocol.</div><div><br /></div><div><br /></div><div><br /></div><div><div><b>Use Cases</b></div><div>– Financial Services: Monte Carlo simulation, risk modeling, actuarial sciences</div><div>– Health and Life Sciences: Genome analysis, drug discovery, bio-sequencing, clinical treatments, molecular modeling</div><div>– Automotive: Vehicle drag coeffcient analysis, crash simulation, engine combustion analysis, air flow modeling</div><div>– Aerospace: Structural, fluid dynamics, thermal, electromagnetic and turbine flow analysis</div><div>– Electronic Design Automation (EDA): Integrated Circuit (IC) and Printed Circuit Board (PCB) design and analysis</div><div>– Oil and Gas: Subsurface terrain modeling, reservoir simulation, seismic analysis</div><div>– Transportation: Routing logistics, supply chain optimization</div><div>– Energy & Utility: Severe storm prediction, climate, weather and wind modelling</div><div>– Education/Research: High energy physics, computational chemistry </div></div><div><br /></div><p>The reign and modern challenges of the Message Passing Interface (MPI)</p><p>All good, but why do you guys doing numerical linear algebra and parallel computing always use the Message Passing Interface to communicate between the processors?”</p><p>MPI begun about 25 years ago and has been since then, undoubtedly, the “King” of HPC. What were the characteristics of MPI that made it the de-facto language of HPC?</p><p>MPI-3 has added several interfaces to enable more powerful communication scheduling, for example nonblocking collective operations and neighborhood collective operations. </p><p>Much of the big data community moved from single-nodes to parallel and distributed computing to process larger amounts of data using relatively short-lived programs and scripts. Thus, programmer productivity only played a minor role in MPI/HPC while it was one of the major requirements for big data analytics. While MPI codes are often orders-of-magnitude faster than many big data codes, they also take much longer to develop. And that is most often a good trade-off.</p><p>MPI I/O has been introduced nearly two decades ago to improve the handling of large datasets in parallel settings. It is successfully used in many large applications and I/O libraries such as HDF-5. </p><p>MPI predates the time when the use of accelerators became commonplace. However, when these accelerators are used in distributed-memory settings such as computer clusters, then MPI is the common way to program them. The current model, often called MPI+X (e.g., MPI+CUDA), combines traditional MPI with accelerator programming models (e.g., CUDA, OpenACC, OpenMP etc.) in a simple way. In this model, MPI communication is performed by the CPU. Yet, this can be inefficient and inconvenient and recently, we have proposed a programming model called distributed CUDA (dCUDA) to perform communication from within a CUDA compute kernel [3]. This allows to use the powerful GPU warp scheduler for communication latency hiding. In general, integrating accelerators and communication functions is an interesting research topic.</p><p>Programming at the transport layer, where every exchange of data has to be implemented with lovingly hand-crafted sends and receives or gets and puts, is an incredibly awkward fit for numerical application developers, who want to think in terms of distributed arrays, data frames, trees, or hash tables. </p><p>Everyone uses MPI” has made it nearly impossible for even made-in-HPC-land tools like Chapel or UPC to make any headway, much less quite different systems like Spark or Flink, meaning that HPC users are largely stuck with using an API which was a big improvement over anything else available 25 years ago, </p><p>Chapel is a modern programming language designed for productive parallel computing at scale. Chapel's design and implementation have been undertaken with portability in mind, permitting Chapel to run on multicore desktops and laptops, commodity clusters, and the cloud, in addition to the high-end supercomputers for which it was originally undertaken.</p><p><span style="font-size: xx-small;">Reference:</span></p><p><span style="font-size: xx-small;">https://medium.com/quantonation/a-beginners-guide-to-high-performance-computing-ae70246a7af</span></p><p><span style="font-size: xx-small;">https://github.com/ljdursi/mpi-tutorial/blob/master/presentation/presentation.md</span></p><p><span style="font-size: xx-small;">https://github.com/chapel-lang/chapel</span></p><p><span style="font-size: xx-small;">https://blog.xrds.acm.org/2017/02/message-passing-interface-mpi-reign-modern-challenges/</span></p><p>================</p></div>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-12862361603124207112023-02-23T19:24:00.001+05:302023-02-28T11:58:57.866+05:30Test log Analytics with Elasticsearch and Kibana <p>Software Test Analytics is the process of collecting and analyzing data from software testing activities to improve the quality and efficiency of the testing process. This can include metrics such as test coverage, defect density, and test execution time, as well as data on test automation and test case management. The goal of software test analytics is to identify trends and patterns in the data that can be used to make informed decisions about how to improve the testing process, such as where to focus testing efforts, which tests to automate, and how to optimize test case design.</p><p>Test log analysis is the process of collecting, analyzing, and interpreting data from test logs in order to identify patterns, trends, and issues that can help improve the quality and performance of software systems. This can include data such as test results, error messages, performance metrics, and other relevant information. The goal of test log analysis is to help identify and resolve issues that may be impacting the performance or functionality of the system, and to improve the overall quality of the software. Common techniques used in test log analysis include statistical analysis, machine learning, and data visualization.</p><p>Insights that can be gained from analyzing test case output logs are listed below: </p><p>1) Identifying which test cases are passing and which are failing, and the reasons for the failures. This can help you to focus your testing efforts on the areas of the application that need the most attention.</p><p>2) Understanding the performance of the application under test. This can include metrics such as response time, memory usage, and CPU usage, which can help you to identify and fix performance bottlenecks.</p><p>3) Identifying patterns in the test case data that indicate potential issues with the application under test. For example, if a large number of test cases are failing in a particular module, it could indicate a problem with that module that needs to be investigated.</p><p>4) Identifying areas of the application that are not being adequately tested. This can help you to create new test cases to cover these areas, and to ensure that the application is thoroughly tested before release.</p><p>5) Identifying where automation can improve the testing process. With the help of logs, you can automate test cases which are repetitive, time-consuming or prone to human errors.</p><p>6) To get the most value out of test case output logs, it's important to have a systematic and automated way of collecting and analyzing the data. This can include using tools such as log analyzers, data visualization tools, and automated reporting tools.</p><p><b>Open-source frameworks that can be used for test result analytics: </b></p><p>ELK Stack: ELK stands for Elasticsearch, Logstash, and Kibana. Elasticsearch is a search engine, Logstash is a log aggregator, and Kibana is a data visualization tool. By using the ELK stack, you can collect, store, and analyze large volumes of test result data in real-time.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXIeweInxo1zhZd_iEMh2WbT_wlmAppLwwS9LV_D9ntFZnqvahYkAS54Y9u_PmeU6OSa7PkLaDg0aXo6dPaaShiJ1QktR1sWviR_csJxnVq3xhH_4Y3lDtYfo8YjKQP814UuKppdRuQrL2yKRaz3vbC75VCsr2d8-cw2JDst1bl3GTqdK3VFdXBA4lWA/s1000/elastci_stack.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="400" data-original-width="1000" height="100" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXIeweInxo1zhZd_iEMh2WbT_wlmAppLwwS9LV_D9ntFZnqvahYkAS54Y9u_PmeU6OSa7PkLaDg0aXo6dPaaShiJ1QktR1sWviR_csJxnVq3xhH_4Y3lDtYfo8YjKQP814UuKppdRuQrL2yKRaz3vbC75VCsr2d8-cw2JDst1bl3GTqdK3VFdXBA4lWA/w249-h100/elastci_stack.png" width="249" /></a></div><br /><p></p><p>The ELK stack (Elasticsearch, Logstash, and Kibana) can be integrated with a CI (Continuous Integration) framework in several ways to show Test analytics.</p><p>1.<span style="white-space: pre;"> </span>Logstash: Logstash can be used to collect and parse log files generated by the CI framework. You can configure Logstash to read log files from the CI server and to parse the data into a format that can be indexed by Elasticsearch.</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinjQTmZxAy0_ywTKbJDIkqJJF6PIxVwGQWw0Ube12DubQ6g4nKU5wMSfb0Pq2_QQQRnO6VwfCrhCogU2JaRfygc82JcCV9QoD7kek6Cf6U7n69WH2RQKhRSg7WNbfa-wggw3oocLWAvyCjwjMj0xOkEPnz0NtpNhUblfsWh4ksU_nHqLU2dgRk4jSsyA/s1000/Kibana_Logo.webp" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="1000" data-original-width="1000" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinjQTmZxAy0_ywTKbJDIkqJJF6PIxVwGQWw0Ube12DubQ6g4nKU5wMSfb0Pq2_QQQRnO6VwfCrhCogU2JaRfygc82JcCV9QoD7kek6Cf6U7n69WH2RQKhRSg7WNbfa-wggw3oocLWAvyCjwjMj0xOkEPnz0NtpNhUblfsWh4ksU_nHqLU2dgRk4jSsyA/w111-h111/Kibana_Logo.webp" width="111" /></a>2.<span style="white-space: pre;"> </span>Kibana: Kibana can be used to visualize the data collected by Logstash. You can create a dashboard in Kibana that displays metrics such as build time, test execution time, and pass/fail rate.</p><p><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit0SUwrbEWYD6mQX8v-6_Y1Yd1tgrMJ31TDYnRsLCOpqnWai-TGsu2x_00DTDjZMESx6OQ8t7IHcvkAcBn3N_xNxHOkqDJ-GenpU3ZcflniYmeHjwfgnlNnKCvENKF82JQZ0yYF3LjIYSyzTUdbGb7GNoqCDqQXhHrYUnaGQxsIgubJ_GYapxTF993Dg/s3422/elasticsearch_logo.png" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" data-original-height="1781" data-original-width="3422" height="101" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEit0SUwrbEWYD6mQX8v-6_Y1Yd1tgrMJ31TDYnRsLCOpqnWai-TGsu2x_00DTDjZMESx6OQ8t7IHcvkAcBn3N_xNxHOkqDJ-GenpU3ZcflniYmeHjwfgnlNnKCvENKF82JQZ0yYF3LjIYSyzTUdbGb7GNoqCDqQXhHrYUnaGQxsIgubJ_GYapxTF993Dg/w193-h101/elasticsearch_logo.png" width="193" /></a></p><p>3.<span style="white-space: pre;"> </span>Elasticsearch: Elasticsearch can be used to store and index the data collected by Logstash. You can use Elasticsearch to search and analyze the data, and to create complex queries and visualizations.</p><p>4.<span style="white-space: pre;"> </span>Integrate with CI/CD tool: You can integrate ELK stack with your CI/CD tool, for example, Jenkins, Travis or CircleCI. You can configure the CI tool to send log files to Logstash, or directly to Elasticsearch.</p><p>-------------------</p><p>After Installation and configuring the ELK stack and Jenkins, the next step for analytics on test reports logs would be to start analyzing and visualizing the data. Here are some steps you can take:</p><p>1.<span style="white-space: pre;"> </span>Verify data collection: Ensure that the data is being collected correctly and that the logs are being indexed in Elasticsearch.</p><p>2.<span style="white-space: pre;"> </span>Create visualizations: Use Kibana to create visualizations such as line charts, bar charts, and pie charts to represent the data in a meaningful way. These visualizations can be used to represent metrics such as build time, test execution time, and pass/fail rate.</p><p>3.<span style="white-space: pre;"> </span>Create dashboards: Use Kibana to create dashboards that display multiple visualizations in a single view. These dashboards can be used to monitor the build and test results in real-time and to analyze the data over time.</p><p>4.<span style="white-space: pre;"> </span>Define alerts: Set up alerts in Kibana to notify you when certain conditions are met, such as a high number of test failures.</p><p>5.<span style="white-space: pre;"> </span>Analyze the data: Use Elasticsearch to create complex queries and to analyze the data in more detail. This can be used to identify patterns and trends in the data that can help improve the testing process.</p><p>6.<span style="white-space: pre;"> </span>Improve test coverage: Use the data to identify areas of the application that are not being adequately tested and to focus your testing efforts on those areas.</p><p>7.<span style="white-space: pre;"> </span>Identify and fix defects: Use the data to identify the root cause of test failures and to fix defects in the application</p><p>------------------------</p><p>We can ingest data with Python on Elastic search instead of using logstash which is good for realtime data ingestion. Prerequisites: Get elasticsearch packages</p><p style="text-align: left;">python -m pip install elasticsearch<br />python -m pip install elasticsearch-async</p><p><b>Some commonly used elasticsearch APIs listed below :</b></p><p></p><ol style="text-align: left;"><li>es.index: Used to index a document into an index.</li><li>es.search: Used to search for documents in an index based on a query.</li><li>es.get: Used to retrieve a document from an index by its ID.</li><li>es.delete: Used to delete a document from an index by its ID.</li><li>es.update: Used to update a document in an index.</li><li>es.count: Used to count the number of documents that match a query without returning the actual documents.</li><li>es.exists: Used to check if a document exists in an index.</li><li>es.bulk: Used to execute multiple index, update, or delete requests in a single HTTP request.</li><li>es.create: Used to create a new index.</li><li>es.delete_index: Used to delete an existing index.</li><li>es.get_mapping: Used to retrieve the mapping of a document type in an index.</li><li>es.put_mapping: Used to define the mapping of a document type in an index.</li><li>es.cluster.health: Used to retrieve information about the health of the Elasticsearch cluster.</li><li>es.cluster.state: Used to retrieve the current state of the Elasticsearch cluster.</li><li>es.nodes.info: Used to retrieve information about the nodes in the Elasticsearch cluster.</li><li>es.nodes.stats: Used to retrieve statistics about the nodes in the Elasticsearch cluster.</li><li>es.termvectors: Used to retrieve information about the terms in a document.</li><li>es.mtermvectors: Used to retrieve information about the terms in multiple documents.</li><li>es.explain: Used to explain how a particular document matches a query.</li><li>es.mget: Used to retrieve multiple documents from an index by their IDs</li></ol><p></p><p><b>Example</b> : Python code to use the Elasticsearch Python client to index a test log report into an Elasticsearch index:</p><p>Pre-requisite: Elasticsearch Python client installed (pip install elasticsearch)</p><p>+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++</p><p># cat my_test_dataToES.py</p><p>#Let me create a sample test report data and push it to ElasticsearchDB</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><span style="background-color: #eeeeee;">from datetime import datetime<br />from elasticsearch import Elasticsearch<br /># create an Elasticsearch client instance<br />es = Elasticsearch("http://sachin.bengaluru.com:9200")<br /><br /># create an Elasticsearch index to store the test log report<br />es.indices.create(index='test_logs_sachin', ignore=400)<br /><br /># define the test log report as a Python dictionary<br />test_log = {<br /> 'test_name': 'login_test',<br /> 'status': 'failed',<br /> 'error_message': 'Invalid credentials',<br /> 'timestamp': datetime.now()<br />}<br /># index the test log report into the Elasticsearch index<br />es.index(index='test_logs_sachin', body=test_log)<br />#</span><br /><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><p>Execute the python script </p><p style="text-align: left;"><span style="background-color: #cccccc;"># python3 my_test_dataToES.py</span></p><p style="text-align: left;"><span style="background-color: #cccccc;">#</span></p><p style="text-align: left;"><span style="background-color: #cccccc;">+++++++++++++++++++++++++++++++++++++++++++++++++++++++++</span></p><p style="text-align: left;"><span style="background-color: #cccccc;">Check the kibana dashboard and create ==> Create index pattern</span></p><p style="text-align: left;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1ur2DcD1CMsnmXsXKWetLWykkoTVo9rIhbgKdrCkILkh5KQWoqToNO0aa4DbBPtxzWu3qnyFUMNjmgHE5ggl9A7xqm8nN_GLQc2Uv3BIGik9m6EGt0hit_pn-q3jhi3Lu65l4DwxdXcs2CMSmFFrOLDTFZm8McAjA7HqDqwF9pdLmjuoT8tdJsiWzkQ/s1548/new_index_ES.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="757" data-original-width="1548" height="312" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1ur2DcD1CMsnmXsXKWetLWykkoTVo9rIhbgKdrCkILkh5KQWoqToNO0aa4DbBPtxzWu3qnyFUMNjmgHE5ggl9A7xqm8nN_GLQc2Uv3BIGik9m6EGt0hit_pn-q3jhi3Lu65l4DwxdXcs2CMSmFFrOLDTFZm8McAjA7HqDqwF9pdLmjuoT8tdJsiWzkQ/w640-h312/new_index_ES.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><p>Visualizing testing results can help you to quickly and easily identify patterns and trends in the data, and can provide valuable insights into the performance and quality </p><p>1) Test Execution Progress: Creating a graph or chart that shows the progress of test execution over time can help you to identify trends in test case pass/fail rates, and to identify areas of the application that are not being adequately tested.</p><p>2) Test Case Pass/Fail Rates: Creating a graph or chart that shows the pass/fail rate for each test case can help you to quickly identify which test cases are passing and which are failing. This can help you to focus your testing efforts on the areas of the application that need the most attention.</p><p>3) Defect Density: Creating a graph or chart that shows the number of defects per unit of code can help you to identify areas of the application that are prone to defects and to identify patterns in the types of defects that are being found.</p><p>4) Test Execution Time: Creating a graph or chart that shows the execution time for each test case can help you to identify performance bottlenecks and to optimize test case design.</p><p>5) Test Automation: Creating a graph or chart that shows the percentage of test cases that are automated can help you to identify areas of the application that can benefit from test automation.</p><p>6) Test coverage : Creating a graph or chart that shows how much of the application is being tested by your test suite can help you identify the areas that are not being covered and focus on increasing test coverage.</p><p>-----------------------------------------------</p><p>NOTE: </p><p>Arrays: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html</a></p><p>In Elasticsearch, there is no dedicated array data type. Any field can contain zero or more values by default, however, all values in the array must be of the same data type. For instance:</p><p>an array of strings: [ "one", "two" ]</p><p>an array of integers: [ 1, 2 ]</p><p>an array of arrays: [ 1, [ 2, 3 ]] which is the equivalent of [ 1, 2, 3 ]</p><p>an array of objects: [ { "name": "Mary", "age": 12 }, { "name": "John", "age": 10 }]</p><p>Arrays of objects: Arrays of objects do not work as you would expect: you cannot query each object independently of the other objects in the array. If you need to be able to do this then you should use the nested data type instead of the object data type. This is explained in more detail in Nested.</p><p><b>Detailed insights from test result logs :</b></p><p>1) Root Cause Analysis: By analyzing the logs of failed test cases, you can identify the root cause of the failure, such as an issue with the application under test, a problem with the test case design, or an environment issue. This can help you to quickly fix the problem and prevent similar issues in the future.</p><p>2) Correlation Analysis: By analyzing the logs of multiple test cases, you can identify patterns and correlations between test results, such as the relationship between test execution time and the number of defects found. This can help you to identify areas of the application that are prone to defects and to optimize test case design.</p><p>3) Regression Analysis: By analyzing the logs of test cases that have been executed over time, you can identify trends in test case pass/fail rates, and to identify areas of the application that are not being adequately tested. This can help you to focus your testing efforts on the areas of the application that need the most attention.</p><p>4) Log Parsing: By parsing the logs, you can extract relevant information such as test case name, status, execution time, error messages, and stack trace. This information can be further analyzed to identify trends and patterns that can help improve the testing process.</p><p>5) Anomaly Detection: By analyzing the logs, you can identify anomalies or unexpected behavior in the test results. This can help you to identify potential issues with the application under test and to quickly fix them before they become major problems.</p><p><b>Machine Learning:</b> You can use machine learning techniques such as clustering, classification, or prediction to analyze test results logs. This can help you to identify patterns and insights that would be difficult to discover manually.</p><p>Natural Language Processing: By using NLP techniques, you can extract useful information from unstructured test result logs. This information can be used to identify patterns and insights that would be difficult to discover manually.</p><p>These techniques can be implemented using machine learning libraries and frameworks such as scikit-learn, TensorFlow, or PyTorch. It's also important to have a good understanding of the data, cleaning and preprocessing it before training the model.</p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-45914219880699806532023-02-20T21:14:00.016+05:302023-02-26T12:25:48.017+05:30HPC workloads on Kubernetes cluster with Volcano batch scheduler <p style="text-align: justify;">Kubernetes is primarily designed for managing containerized workloads, and it can also be used for managing High-Performance Computing (HPC) clusters. However, using Kubernetes for HPC clusters may require additional customization and configuration.</p><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div>Performance: Kubernetes is not specifically designed for high-performance computing workloads, and it may not provide the same level of performance as specialized HPC schedulers. To improve performance, you may need to customize the Kubernetes scheduling policies and resource management.</div><div><br /></div><div>Networking: HPC workloads often require high-bandwidth, low-latency networking between compute nodes. Kubernetes networking is designed for containerized workloads, and it may not provide the required level of performance for HPC workloads. You may need to use specialized network plugins or customize the Kubernetes networking configuration to achieve the desired performance.</div><div><br /></div><div>Storage: HPC workloads often require large amounts of high-performance storage. Kubernetes provides built-in support for persistent volumes, but it may not be optimized for HPC workloads. You may need to use specialized storage solutions or customize the Kubernetes storage configuration.</div><div><br /></div><div>Resource Management: HPC clusters often require strict resource management to ensure that workloads are efficiently utilizing compute resources. Kubernetes provides a range of resource management features, such as quotas and limits, but you may need to customize the configuration to meet the specific needs of your HPC workloads</div></div><p style="text-align: justify;">----------------------</p><p style="text-align: justify;">Kubernetes Batch: Kubernetes Batch is a built-in batch scheduler that is included in the core Kubernetes distribution. It provides basic batch job management functionality, such as job creation, monitoring, and cleanup.</p><p style="text-align: justify;">In addition to Kubernetes Batch, there are several other batch schedulers that can be used with Kubernetes for managing batch jobs, including:</p><p style="text-align: justify;"></p><ul><li>Apache Airflow: Apache Airflow is a popular open-source platform for creating, scheduling, and monitoring workflows. It can be used to manage batch jobs in Kubernetes using the KubernetesExecutor.</li><li>Apache Spark: Apache Spark is a distributed computing framework that includes a built-in scheduler for managing batch jobs. Spark can be run on Kubernetes using the Spark operator, which provides support for managing Spark jobs as Kubernetes native resources.</li><li>HTCondor: HTCondor is a widely-used batch scheduler in the high-performance computing (HPC) community. It can be used with Kubernetes through the HTCondor-Kubernetes integration, which allows HTCondor to manage Kubernetes pods as HTCondor jobs.</li><li>Slurm: Slurm is another popular batch scheduler in the HPC community. It can be used with Kubernetes through the Slurm-Kubernetes integration, which allows Slurm to manage Kubernetes pods as Slurm jobs.</li><li>Valcano: Valcano is a new batch scheduler designed specifically for Kubernetes. It provides advanced job scheduling and resource management capabilities, such as backfill scheduling and intelligent resource allocation</li></ul><p></p><p style="text-align: justify;">---------------------------------------------------------------</p><div><b>Volcano is a Kubernetes native batch system and job scheduler designed to handle deep learning, machine learning, and high-performance computing (HPC) workloads. It was developed by Alibaba Cloud and was released as an open-source project in 2018.</b></div><div><br /></div><div>Here are some of the features that make Volcano special for HPC workloads:</div><div><ul style="text-align: left;"><li>Resource Management: Volcano has advanced resource management capabilities that allow it to efficiently schedule and manage HPC workloads across a large number of nodes. It can allocate GPUs, CPUs, memory, and other resources required for running HPC workloads in a distributed computing environment.</li><li>Performance Optimization: Volcano is designed to optimize the performance of HPC workloads by minimizing resource contention and reducing the time it takes to start and complete jobs. It uses advanced scheduling algorithms to allocate resources and optimize job execution times.</li><li>Custom Schedulers: Volcano allows users to create custom schedulers and customize job scheduling policies to meet their specific requirements. This makes it easier for users to configure the scheduler to meet the specific needs of their HPC workloads.</li><li>Job Prioritization: Volcano supports job prioritization based on various factors such as job dependencies, job age, and user-defined priorities. This ensures that higher priority jobs are executed first, and that resources are allocated efficiently across the cluster.</li><li>Workflow Support: Volcano supports complex workflow management, allowing users to define dependencies between jobs and execute them in a specific order. This is particularly useful for HPC workloads that require a series of jobs to be executed in a specific sequence.</li><li>Overall, Volcano is a powerful job scheduler that is optimized for HPC workloads in Kubernetes environments. Its advanced resource management, performance optimization, and custom scheduling capabilities make it an ideal choice for running complex deep learning, machine learning, and HPC workloads in Kubernetes.</li></ul></div><div><br /></div><div>------------------</div><p style="text-align: justify;">Integrating the Volcano scheduler into a Kubernetes cluster involves following steps:</p><p style="text-align: justify;">Install the Volcano components: The first step is to install the Volcano components on the Kubernetes cluster. This can be done using the Volcano Helm chart, which includes all the necessary components such as the Volcano scheduler, admission controllers, and CRDs (Custom Resource Definitions).</p><p style="text-align: justify;">Install Volcano components with Helm chart:</p><p style="text-align: justify;"># Add the Volcano Helm repository<br /><span style="font-size: xx-small;">helm repo add volcano https://volcano.sh/charts<br />helm repo update</span><br /><br /># Install the Volcano components<br /><span style="font-size: xx-small;">helm install volcano volcano/volcano</span></p><p style="text-align: justify;"><span style="font-size: x-small;"><b>NOTE: HELM is package manager for Kubernetes</b></span></p><p style="text-align: justify;">Configure the scheduler: Once the Volcano components are installed, the next step is to configure the Volcano scheduler. This involves setting the scheduling policies, priority classes, and other parameters that govern how the scheduler allocates resources to jobs.</p><p style="text-align: justify;">Configure the scheduler with YAML file:</p><p style="text-align: justify;"><span style="font-size: x-small;">apiVersion: scheduling.volcano.sh/v1beta1<br />kind: Queue<br />metadata:<br /> name: default<br />spec:<br /> # Set the scheduling policy for the queue<br /> schedulingPolicy:<br /> type: "PriorityPolicy"<br /> priorityPolicy:<br /> defaultPriority: 50<br /> # Set the resource limits for the queue<br /> resources:<br /> limits:<br /> cpu: "16"<br /> memory: "64Gi"<br /> requests:<br /> cpu: "2"<br /> memory: "8Gi"</span></p><p style="text-align: justify;">Apply the YAML file with the following command:</p><p style="text-align: justify;"><span style="font-size: x-small;">kubectl apply -f queue.yaml</span></p><p style="text-align: justify;">Define the job templates: After configuring the scheduler, the next step is to define the job templates that will be used to create batch jobs. This involves specifying the Docker image, command, arguments, and resource requirements for each job template.</p><p style="text-align: justify;">Define job templates with YAML file:</p><p style="text-align: justify;"><span style="font-size: x-small;">apiVersion: batch.volcano.sh/v1alpha1<br />kind: Job<br />metadata:<br /> name: pi<br />spec:<br /> template:<br /> spec:<br /> containers:<br /> - name: pi<br /> image: perl<br /> command: ["perl"]<br /> args: ["-Mbignum=bpi", "-wle", "print bpi(2000)"]<br /> resources:<br /> limits:<br /> cpu: "2"<br /> memory: "8Gi"<br /> requests:<br /> cpu: "1"<br /> memory: "4Gi"<br /> # Set the queue name to use for this job<br /> queue: default</span></p><p style="text-align: justify;">Apply the YAML file with the following command:</p><p style="text-align: justify;"><span style="font-size: x-small;">kubectl apply -f job.yam</span></p><p style="text-align: justify;">---------------------</p><p style="text-align: justify;">Create batch jobs: Once the job templates are defined, batch jobs can be created using the Kubernetes API or the kubectl command-line tool. When a batch job is created, the Volcano scheduler will allocate resources to the job based on the scheduling policies and resource requirements defined in the job template. Create batch jobs with kubectl command:</p><p style="text-align: justify;"><span style="font-size: x-small;">kubectl create job pi --image=perl -- perl -Mbignum=bpi -wle 'print bpi(2000)'</span></p><p style="text-align: justify;">Monitor and manage batch jobs: Finally, the batch jobs can be monitored and managed using the Kubernetes API or the kubectl command-line tool. This includes viewing the status of running jobs, scaling up or down the number of replicas, and deleting completed jobs.</p><p style="text-align: justify;"><br /></p><p style="text-align: justify;">Monitor and manage batch jobs with kubectl command:</p><p style="text-align: justify;"># View the status of all batch jobs</p><p style="text-align: justify;"><span style="font-size: x-small;">kubectl get jobs.batch.volcano.sh</span></p><p style="text-align: justify;"><br /></p><p style="text-align: justify;"># View the logs for a specific batch job</p><p style="text-align: justify;">kubectl logs job/pi</p><p style="text-align: justify;"><br /></p><p style="text-align: justify;"># Scale up or down the number of replicas for a batch job</p><p style="text-align: justify;">kubectl scale job/pi --replicas=10</p><p style="text-align: justify;"><br /></p><p style="text-align: justify;"># Delete a completed batch job</p><p style="text-align: justify;">kubectl delete job/pi</p><p style="text-align: justify;">-----------------------------</p><p style="text-align: justify;">kubectl get jobs.batch.volcano.sh</p><div style="text-align: justify;">This command will display a table with information about all the Volcano jobs currently running, including the job name, namespace, queue, status, and creation time.</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">If you want to see more details about a specific job, you can use the following command</div><div style="text-align: justify;"><br /></div><div style="text-align: justify;"><div>kubectl describe job.batch.volcano.sh job-name</div><div><br /></div></div><p style="text-align: justify;">--------------------------------</p><p style="text-align: justify;">The rdma/hca_shared resource in Kubernetes refers to the HCA (Host Channel Adapter) Shared Device Plugin. This device plugin enables Kubernetes to detect and utilize InfiniBand and RDMA network interfaces on the host nodes, allowing containers to access RDMA resources through a standard Kubernetes API. By using the HCA Shared Device Plugin in a Kubernetes cluster, applications can take advantage of RDMA technology for high-performance networking and low-latency communication. This is especially useful for applications that require high-throughput data transfer or require low-latency communication, such as big data analytics, high-performance computing, and machine learning workloads. So, rdma/hca_shared configuration in Kubernetes is used to enable RDMA support in the cluster, which can improve the performance of certain types of applications that require fast, low-latency networking.</p><p style="text-align: justify;">--------------------Add resources to a Kubernetes cluster using YAML file-----</p><p style="text-align: justify;">To add Nvidia GPU resources to a Kubernetes cluster using YAML file, you will need to modify the spec section of the deployment or pod YAML file to include the necessary configuration options. Here's an example YAML file for adding Nvidia GPU resources to a Kubernetes deployment:</p><p style="text-align: justify;"><span style="font-size: x-small;">apiVersion: apps/v1<br />kind: Deployment<br />metadata:<br /> name: my-deployment<br />spec:<br /> replicas: 1<br /> selector:<br /> matchLabels:<br /> app: my-app<br /> template:<br /> metadata:<br /> labels:<br /> app: my-app<br /> spec:<br /> containers:<br /> - name: my-container<br /> image: my-image<br /> resources:<br /> limits:<br /> nvidia.com/gpu: 1<br />---------------</span></p><p style="text-align: justify;">In the example above, the resources section specifies that the container needs one Nvidia GPU resource. You can adjust the number of GPUs by changing the value of nvidia.com/gpu. You can also specify the GPU type by using the nvidia.com/gpu-type resource limit.</p><p style="text-align: justify;">You can apply this YAML file to your Kubernetes cluster using the kubectl apply -f command </p><p style="text-align: justify;">The most commonly used Kubernetes commands:</p><p style="text-align: justify;"></p><ol><li><span style="font-size: x-small;">kubectl create: creates a resource from a file or from stdin.</span></li><li><span style="font-size: x-small;">kubectl apply: applies changes to an existing resource.</span></li><li><span style="font-size: x-small;">kubectl get: retrieves information about one or more resources.</span></li><li><span style="font-size: x-small;">kubectl describe: provides detailed information about a specific resource or a set of resources.</span></li><li><span style="font-size: x-small;">kubectl delete: removes one or more resources from the cluster.</span></li><li><span style="font-size: x-small;">kubectl logs: displays logs from a specific pod or container.</span></li><li><span style="font-size: x-small;">kubectl exec: runs a command inside a container in a specific pod.</span></li><li><span style="font-size: x-small;">kubectl port-forward: forwards one or more local ports to a pod.</span></li><li><span style="font-size: x-small;">kubectl rollout: manages rolling updates of a deployment.</span></li></ol><div><div>You can use the kubectl logs command with the -f option to stream logs in real-time, similar to the tail -f command. Here's the syntax for using kubectl logs with the -f option:</div><div><br /></div><div>kubectl logs -f <pod-name> <container-name></div><div>kubectl logs -f my-pod my-container</div><div><br /></div><div>This will start streaming the logs from the my-container container in the my-pod pod in real-time. You can stop the log streaming by pressing Ctrl+C.</div></div><div><br /></div><div><br /></div><div><div>---------------------------------------------------------------------------------------------</div><div><br /></div><div><div>Version of Volcano scheduler installed on K8s cluster is : </div><div><br /></div><div style="text-align: left;"><span style="font-size: x-small;">$ kubectl get deployment -n volcano-system volcano-scheduler -o=jsonpath='{.spec.template.spec.containers[0].image}' | cut -d':' -f2<br />1.7<br />$</span></div><div><br /></div><div>----------details ----------------</div><div>$ kubectl describe deployment -n volcano-system volcano-scheduler</div><div><span style="font-size: xx-small;">Name: volcano-scheduler</span></div><div><span style="font-size: xx-small;">Namespace: volcano-system</span></div><div><span style="font-size: xx-small;">CreationTimestamp: Fri, 03 Feb 2023 00:38:14 -0500</span></div><div><span style="font-size: xx-small;">Labels: app=volcano-scheduler</span></div><div><span style="font-size: xx-small;">Annotations: deployment.kubernetes.io/revision: 1</span></div><div><span style="font-size: xx-small;">Selector: app=volcano-scheduler</span></div><div><span style="font-size: xx-small;">Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable</span></div><div><span style="font-size: xx-small;">StrategyType: RollingUpdate</span></div><div><span style="font-size: xx-small;">MinReadySeconds: 0</span></div><div><span style="font-size: xx-small;">RollingUpdateStrategy: 25% max unavailable, 25% max surge</span></div><div><span style="font-size: xx-small;">Pod Template:</span></div><div><span style="font-size: xx-small;"> Labels: app=volcano-scheduler</span></div><div><span style="font-size: xx-small;"> Service Account: volcano-scheduler</span></div><div><span style="font-size: xx-small;"> Containers:</span></div><div><span style="font-size: xx-small;"> volcano-scheduler:</span></div><div><span style="font-size: xx-small;"> Image: sachin-power-hpc-docker-local/volcanosh/vc-scheduler:1.7</span></div><div>----------------</div></div><div>NOTE: This command describes the deployment object for the Volcano Scheduler in the volcano-system namespace and provides detailed information about the deployment, including the image tag and version.</div><div><br /></div><div>You can get the version of Kubernetes installed in your cluster by running the following command:</div><div><br /></div><div style="text-align: left;"><span style="font-size: x-small;">$ kubectl version --short | grep Server<br />Server Version: v1.23.4<br />$</span></div><div><p>After creating the job - we can check the job name as shown in Step 2 </p><p>Step 1 : Submit Job - spec defines scheduler=volcano</p><p style="text-align: left;"><span style="font-size: xx-small;">[spb@k8s-masterNode]#kubectl create -f openmpi-example.yaml</span><br /><span style="font-size: x-small;">job.batch.volcano.sh/nj-ompi-job created</span></p><p>Step 2: get the name of job submitted by volcano scheduler . So, It's vcjob </p><p style="text-align: left;"><span style="font-size: xx-small;">[spb@k8s-masterNode]#kubectl get vcjob -o custom-columns=:.metadata.name</span><br /><span style="font-size: x-small;">nj-ompi-job</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#kubectl get vcjob -o custom-columns=NAME:.metadata.name</span><br /><span style="font-size: x-small;">NAME</span><br /><span style="font-size: x-small;">nj-ompi-job</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#</span></p><p>Step 3 : Just observe - how the vcjob changes it's state from Pending ---> Running--->Completed</p><p style="text-align: left;"><span style="font-size: xx-small;">[spb@k8s-masterNode]#kubectl get vcjob</span><br /><span style="font-size: x-small;">NAME STATUS MINAVAILABLE RUNNINGS AGE</span><br /><span style="font-size: x-small;">nj-ompi-job Pending 2 5s</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#kubectl get vcjob</span><br /><span style="font-size: x-small;">NAME STATUS MINAVAILABLE RUNNINGS AGE</span><br /><span style="font-size: x-small;">nj-ompi-job Running 2 2 14s</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#kubectl get vcjob</span><br /><span style="font-size: x-small;">NAME STATUS MINAVAILABLE RUNNINGS AGE</span><br /><span style="font-size: x-small;">nj-ompi-job Running 2 3 37s</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#kubectl get vcjob</span><br /><span style="font-size: x-small;">NAME STATUS MINAVAILABLE RUNNINGS AGE</span><br /><span style="font-size: x-small;">nj-ompi-job Completed 2 90s</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#</span></p><p><br /></p><p>Step 4 : Based on the spec defined in yaml file - you can dispaly the vcjob </p><p style="text-align: left;"><span style="font-size: xx-small;">[spb@k8s-masterNode]#kubectl get vcjob -o custom-columns='NAME:.metadata.name,MinAvailable:.spec.minAvailable,SCHED:.spec.schedulerName'</span><br /><span style="font-size: x-small;">NAME MinAvailable SCHED</span><br /><span style="font-size: x-small;">nj-ompi-job 2 volcano</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#</span></p><p>Step 5 : Get the status of VC job </p><p style="text-align: left;"><span style="font-size: xx-small;">[spb@k8s-masterNode]#kubectl get vcjob nj-ompi-job -n kube-system -o jsonpath='{.status.conditions[?(@.status=="Completed")].status}'</span><br /><span style="font-size: x-small;">Completed</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#kubectl get vcjob nj-ompi-job -n kube-system -o jsonpath='{.status.conditions[-1:].status}'</span><br /><span style="font-size: x-small;">Completed</span><br /><span style="font-size: x-small;">[spb@k8s-masterNode]#</span></p></div><div><div>Get the status of a specific job:</div><div><span style="font-size: xx-small;">kubectl get job.batch.volcano.sh job-name -o yaml</span></div><div><br /></div><div>This command will display the job status in YAML format, which can be useful for debugging errors or issues with the job</div><div><br /></div><div>Get the logs for a specific pod in the job</div><div><span style="font-size: xx-small;">kubectl logs job-name-pod-name</span></div><div><br /></div><div>Get detailed information about the job and its pods:</div><div><span style="font-size: xx-small;">kubectl describe job.batch.volcano.sh job-name</span></div><div><span style="font-size: xx-small;">kubectl describe pod job-name-pod-name</span></div><div><br /></div><div>View the events for a specific job:</div><div><br /></div><div><span style="font-size: xx-small;">kubectl get events --field-selector involvedObject.name=job-name</span></div><div><br /></div><div><b>Example:</b></div><div><br /></div><div><div><span style="font-size: xx-small;">$ kubectl get events --field-selector involvedObject.name=smpi-build-0 -n smpici</span></div><div><span style="font-size: xx-small;">LAST SEEN TYPE REASON OBJECT MESSAGE</span></div><div><span style="font-size: xx-small;">20m Normal Scheduled pod/smpi--build-0 Successfully assigned smpici/smpi-build-0 to workerNode</span></div><div><span style="font-size: xx-small;">20m Normal Pulling pod/smpi-build-0 Pulling image "sachin-power-hpc-docker-local/master:rhel_8.6"</span></div><div><span style="font-size: xx-small;">20m Normal Pulled pod/smpi-build-0 Successfully pulled image "sachin-power-hpc-docker-local/master:rhel_8.6" in 2.374936782s</span></div><div><span style="font-size: xx-small;">20m Normal Created pod/smpi--build-0 Created container build</span></div><div><span style="font-size: xx-small;">20m Normal Started pod/smpi-build-0 Started container build</span></div><div><span style="font-size: xx-small;">$</span></div></div><div><br /></div><div>View the resource usage of a specific pod:</div><div><span style="font-size: xx-small;">kubectl top pod job-name-pod-name</span></div><div><br /></div><div>This command will display the CPU and memory usage of a specific pod in the job, which can help with diagnosing performance issues or resource constraints.</div></div><div><br /></div><div><b>Example : </b></div><div><div><span style="font-size: xx-small;">$ kubectl get pods -n smpici</span></div><div><span style="font-size: xx-small;">NAME READY STATUS RESTARTS AGE</span></div><div><span style="font-size: xx-small;">smpi-37-build-0 1/1 Running 0 16m</span></div><div><span style="font-size: xx-small;">$</span></div><div><span style="font-size: xx-small;">$ kubectl top pod smpi-weekly-ibm-smpi-37-build-0 -n smpici</span></div><div><span style="font-size: xx-small;">NAME CPU(cores) MEMORY(bytes)</span></div><div><span style="font-size: xx-small;">smpi-37-build-0 2m 208Mi</span></div><div><span style="font-size: xx-small;">$</span></div></div><div><br /></div><div><b>Example :</b> How to <b>delete the pod </b>that completed successfully</div><div><br /></div><div><div><span style="font-size: xx-small;">$ kubectl get pods -n smpici --field-selector=status.phase==Succeeded</span></div><div><span style="font-size: xx-small;">NAME READY STATUS RESTARTS AGE</span></div><div><span style="font-size: xx-small;">smpi-30-k8s-mpimaster-0 0/1 Completed 0 5d2h</span></div><div><span style="font-size: xx-small;">$</span></div><div><span style="font-size: xx-small;">$ kubectl delete pods -n smpici --field-selector=status.phase==Succeeded</span></div><div><span style="font-size: xx-small;">pod "smpi-30-k8s-mpimaster-0" deleted</span></div><div><span style="font-size: xx-small;">$</span></div></div><div><br /></div><div>-------------------------Vela, IBM’s first AI-optimized, cloud-native supercomputer.----------------------</div><div><br /></div><div><div><br /></div><div>IBM has announced the release of Vela, a new security technology and AI supercomputer. The technology is designed to assist organizations in detecting and responding to cyber threats in real time.</div><div><br /></div><div>It is also IBM’s first AI-optimized, cloud-native supercomputer, designed for developing and training large-scale AI models. It combines various security tools and technologies to provide a unified view of a company’s security posture.</div><div><br /></div><div>Vela is intended to assist organizations in improving their security posture by providing them with the information they need to quickly identify and respond to threats. The technology analyzes massive amounts of security data and provides actionable insights to security teams using artificial intelligence and machine learning algorithms. It integrates various security tools and technologies and employs artificial intelligence and machine learning algorithms to provide security teams with actionable insights.</div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCv5D0uTl1jdb3pVf0XznGXPwXNOshW0yglPkYXuLvZ7rmsD7U4YD36kEDDxFiguNGYlkWOpfwsHjxT-FWL1eyxWL2X5Boc381IUpiPhHcf6PqZviYjScj8R1RJG6XTVAxoIbyw05pPALHFcUstWjK2jYuw7-pocWL8Z7pjINMiQqfUp5wGffl-40_nQ/s1008/cloud_native_hpc-AI-stack.png" style="margin-left: auto; margin-right: auto;"><img alt="https://research.ibm.com/blog/AI-supercomputer-Vela-GPU-cluster" border="0" data-original-height="542" data-original-width="1008" height="344" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCv5D0uTl1jdb3pVf0XznGXPwXNOshW0yglPkYXuLvZ7rmsD7U4YD36kEDDxFiguNGYlkWOpfwsHjxT-FWL1eyxWL2X5Boc381IUpiPhHcf6PqZviYjScj8R1RJG6XTVAxoIbyw05pPALHFcUstWjK2jYuw7-pocWL8Z7pjINMiQqfUp5wGffl-40_nQ/w640-h344/cloud_native_hpc-AI-stack.png" title="https://research.ibm.com/blog/AI-supercomputer-Vela-GPU-cluster" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://research.ibm.com/blog/AI-supercomputer-Vela-GPU-cluster" target="_blank"><span style="font-size: xx-small;">source</span></a></td></tr></tbody></table><div><br /></div></div><div><br /></div></div><div>A comparison of the traditional HPC software stack and Cloud-native AI stack are shown in diagram. But the downside of cloud native stack with virtualization, historically, is that it reduces node performance.</div><div><br /></div><div><br /></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrizcpkDXspbB0JsI2zHFis2Hfx8tDTn_H5RpxG92rcvoyPHCPh0E7dnso3V83OIPtbKC0GI7aC9cQz9ZcR63oA-RUkt7BduFwJlQAR4C2G8Gxj-CNg3vKpuHnBhZHz_WQBAerCNwmqaqlRk30CykvuUBGXJE7p_droyOdTcfaKuFri-a_Eco0WLrTwA/s955/vela.png" style="font-size: 20px; margin-left: auto; margin-right: auto;"><span face="IBM Plex Sans, Helvetica Neue, Arial, sans-serif" style="color: #121619;"><img border="0" data-original-height="718" data-original-width="955" height="482" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrizcpkDXspbB0JsI2zHFis2Hfx8tDTn_H5RpxG92rcvoyPHCPh0E7dnso3V83OIPtbKC0GI7aC9cQz9ZcR63oA-RUkt7BduFwJlQAR4C2G8Gxj-CNg3vKpuHnBhZHz_WQBAerCNwmqaqlRk30CykvuUBGXJE7p_droyOdTcfaKuFri-a_Eco0WLrTwA/w640-h482/vela.png" width="640" /></span></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://research.ibm.com/blog/AI-supercomputer-Vela-GPU-cluster" target="_blank"><span face="IBM Plex Sans, Helvetica Neue, Arial, sans-serif" style="color: #121619; font-size: xx-small;">source</span></a></td></tr></tbody></table><div><br /></div><p></p><p style="text-align: justify;"><br /></p><p style="text-align: justify;">This work includes configuring the bare-metal host for virtualization with support for Virtual Machine Extensions (VMX), single-root IO virtualization (SR-IOV), and huge pages. With AI-optimized processor, the IBM AIU [Artificial Intelligence Unit] - The era of cloud-native AI supercomputing has only just begun. The IBM AIU is not a graphics processor. It was specifically designed and optimized to accelerate matrix and vector computations used by deep learning models. The AIU can solve computationally complex problems and perform data analysis at speeds far beyond the capability of a CPU. Deploying AI to classify cats and dogs in photos is a fun academic exercise. But it won’t solve the pressing problems we face today. For AI to tackle the complexities of the real world — things like predicting the next Hurricane /natural calamities or whether we’re heading into a recession — we need enterprise-quality, industrial-scale hardware. IBM AIU takes us one step closer. </p><p style="text-align: justify;">There are many real-world examples of Kubernetes clusters being used to run HPC workloads. Here are a few examples:</p><p style="text-align: justify;"></p><ul><li>The National Energy Research Scientific Computing Center (NERSC), which is part of the US Department of Energy, uses Kubernetes to manage its HPC resources. NERSC uses Kubernetes to manage both traditional HPC workloads and machine learning workloads, and has reported significant improvements in resource utilization and efficiency since adopting Kubernetes.</li><li>Argonne National Laboratory's Theta supercomputer, which is one of the fastest supercomputers in the world, uses Kubernetes to manage its containerized workloads. Theta uses Kubernetes to run a variety of scientific simulations, including simulations of earthquakes and climate models.</li><li style="text-align: justify;">Oak Ridge National Laboratory's Summit supercomputer, which is the most powerful supercomputer in the world, also uses Kubernetes to manage its containerized workloads.Summit uses Kubernetes to run scientific simulations and other HPC workloads. The Summit supercomputer at Oak Ridge National Laboratory uses IBM Spectrum MPI as its primary MPI implementation for running parallel workloads. It is is capable of running a variety of HPC workloads using MPI, including simulations, data analysis, and machine learning applications.</li><li>The University of Cambridge's High Performance Computing Service uses Kubernetes to manage its HPC resources. The service uses Kubernetes to run a variety of HPC workloads, including simulations of fluid dynamics and other scientific applications.</li></ul><div style="text-align: justify;">One of the most commonly used container runtime engines on Summit is Singularity, which is an open-source container platform designed specifically for HPC workloads. Singularity is optimized for running MPI workloads, and provides a number of features specifically tailored to HPC, including support for Infiniband networking and the ability to run on a wide range of HPC systems. In addition to Singularity, Summit also supports other container runtime engines, including Docker and Shifter. However, Singularity is the preferred container runtime engine for running HPC workloads on Summit, and is used by many researchers and organizations to run MPI workloads.</div><div style="text-align: justify;"><div><br /></div><div><div>There are a number of emerging container runtime technologies that are being developed for use in HPC environments. One example is Charliecloud, which is an open-source container runtime that is designed specifically for HPC workloads. Charliecloud provides a lightweight and secure environment for running containerized workloads on HPC systems, and is optimized for performance and flexibility.</div><div><br /></div><div>Another emerging technology is called udocker, which is a user-space container engine that is designed to provide a simple and lightweight way to run containerized workloads on HPC systems. Udocker is optimized for use in environments where users may not have root access to the system, and provides a number of features specifically tailored to HPC workloads, including support for MPI and Infiniband networking.</div></div><div><br /></div><div><div>Frontier is a new supercomputer that is currently being developed at Oak Ridge National Laboratory. </div><div>One of the key innovations of the Frontier supercomputer is its use of new "next-generation" runtime engines that are being developed specifically for HPC workloads. These new runtime engines are designed to provide improved performance, scalability, and flexibility compared to traditional runtime engines like MPI. One of the most promising new runtime engines being developed for Frontier is called Unified Parallel C (UPC). UPC is a high-level parallel programming language that is designed to simplify the development of parallel algorithms for HPC workloads. It provides a number of features specifically tailored to HPC, including support for shared memory programming and optimized communication primitives.</div></div><div><br /></div><div>AWS provides a wide range of compute, storage, and network resources that are optimized for running HPC workloads in the cloud. By leveraging these services, researchers and engineers can easily and cost-effectively run complex simulations and other HPC workloads, while also benefiting from the scalability and flexibility of cloud computing. Some examples:</div><div><div><ul><li>Amazon EC2: This is AWS's flagship compute service, which allows users to rent virtual machines (instances) with a variety of different configurations and capabilities. EC2 provides a wide range of instance types that are optimized for different workloads, including HPC workloads. These instance types offer high-performance CPUs, GPUs, and FPGAs, as well as high-speed network connectivity.</li><li>Amazon Elastic File System (EFS): This is a fully managed cloud file storage service that is designed for HPC workloads. EFS provides a scalable and highly available file system that can be accessed from multiple instances simultaneously, which is important for parallel computing workloads.</li><li>Amazon S3: This is a highly scalable and durable object storage service that can be used to store and retrieve large data sets for HPC workloads. S3 provides a simple API that can be used to access data from anywhere, and supports a variety of data formats and access patterns.</li><li>AWS ParallelCluster: This is an open-source HPC cluster management tool that can be used to deploy and manage HPC clusters on AWS. ParallelCluster provides a simple interface for configuring and launching HPC clusters, and supports a variety of different schedulers and software packages.</li><li>Amazon FSx for Lustre: This is a fully managed file system service that provides high-performance Lustre file systems for HPC workloads. FSx for Lustre provides scalable performance and high availability, and can be used to store and manage large data sets for parallel computing workloads.</li></ul><div>To run weather data analysis using HPC cloud services on AWS, you would need to follow these general steps:</div></div></div><div><div><ul><li>Provision the necessary compute resources: Using services such as Amazon EC2, you would need to launch instances that are optimized for your specific workload. For weather data analysis, you may require high-performance CPUs, GPUs, or FPGAs, as well as high-speed network connectivity to move data in and out of the instances.</li><li>Store the data: You would need to store the weather data in a highly scalable and durable storage system. Amazon S3 is a popular choice for storing large data sets in the cloud.</li><li>Configure the software stack: Once the compute resources and data storage are set up, you would need to configure the software stack. This would involve installing and configuring the necessary software packages, including any libraries or tools that are required for weather data analysis.</li><li>Submit the analysis jobs: You would then submit the analysis jobs to the HPC cluster using a batch scheduler such as Slurm or AWS ParallelCluster. The scheduler will manage the allocation of compute resources and ensure that the jobs are executed in a timely and efficient manner.</li><li>Retrieve the results: Once the analysis jobs are completed, you would retrieve the results from the storage system and perform any post-processing or analysis that is necessary.</li></ul><div>Additionally, there are a number of third-party tools and services available that are specifically designed for weather data analysis in the cloud, such as the Unidata Cloud-Ready Data Services, which provides a suite of cloud-based tools and services for working with weather data. There are some configuration details for a typical HPC software stack that might be used for the above example -weather data analysis :</div></div></div><div><div><ul><li>Operating System: Many HPC workloads run on Linux-based operating systems, such as CentOS or Ubuntu. These operating systems are generally lightweight and optimized for high-performance computing. Linux distributions, such as CentOS and Ubuntu, are commonly used in HPC environments, particularly in academic and research settings where open source software is often preferred. RHEL (Red Hat Enterprise Linux) and SLES (SUSE Linux Enterprise Server) are actually commonly used operating systems in HPC environments, particularly in the commercial sector. Both RHEL and SLES are enterprise-grade operating systems that offer long-term support and stability, which are important features in HPC environments where system uptime and reliability are critical.</li><li>Cluster Management Software: There are a variety of cluster management software packages available for HPC workloads, including Slurm, Torque, and LSF. These tools provide a way to manage the allocation of compute resources and schedule jobs on the cluster.</li><li>MPI Library: For parallel computing workloads, you would typically need to install a Message Passing Interface (MPI) library, such as OpenMPI or MPICH. These libraries allow multiple processes to communicate with each other and coordinate their work on the cluster.</li><li>Compiler Toolchain: A compiler toolchain is necessary for building and executing code on the cluster. Commonly used compilers include GCC, Clang, and Intel Compiler.</li><li>Data Processing Libraries: For weather data analysis, you would typically need to install a variety of data processing libraries, such as NetCDF, HDF5, and GRIB. These libraries allow you to read, write, and manipulate weather data in a variety of formats.</li><li>Visualization Tools: Once the weather data analysis is complete, you may want to visualize the results using tools such as Matplotlib, Paraview, or Visit. These tools allow you to create visualizations and animations that can help you better understand the results of your analysis.</li></ul></div></div><div>Steps to be followed in AWS to run HPC workloads :</div><div><div><ul><li>Choose an AWS instance: Select an Amazon EC2 instance type that suits your computational needs. EC2 provides a range of instance types with varying CPU, memory, and storage capacities. For a weather data analysis, you may need a high-memory instance type with multiple CPUs and GPUs, depending on the size of your data.</li><li>Install software dependencies: Install the necessary software dependencies on your EC2 instance, such as the operating system, compilers, libraries, and analysis tools. You can either install these manually or use a configuration management tool like Ansible or Chef to automate the process.</li><li>Move data to AWS: Transfer your weather data from your on-premises environment to the AWS cloud using AWS Storage services such as Amazon S3 or EFS. You can also use AWS Direct Connect to establish a dedicated network connection between your on-premises environment and your AWS resources.</li><li>Configure your software: Set up your weather analysis software, including the input/output paths, data formats, and algorithm parameters. You can run your analysis either on a single EC2 instance or on a cluster of instances using HPC job schedulers like Slurm or LSF.</li><li>Monitor and optimize performance: Monitor the performance of your weather analysis to identify bottlenecks and optimize performance. You can use AWS CloudWatch to monitor CPU, memory, and network utilization, as well as application-level metrics like response times and error rates.</li><li>Generate reports and visualizations: Once your analysis is complete, you can generate reports and visualizations using tools like Matplotlib, Paraview, or Visit. You can save the results to Amazon S3 or EFS, or use other AWS services like AWS Lambda to trigger alerts or actions based on the results.</li></ul><div>Some of the commonly used HPC workloads on AWS cloud :</div></div></div><div><ul><li>Genomics and bioinformatics: AWS offers a range of genomic and bioinformatics tools and services that can handle large-scale data processing and analysis tasks. These include tools for sequence alignment, variant calling, gene expression analysis, and genome assembly.</li><li>Computational chemistry and materials science: AWS provides a range of high-performance computing (HPC) resources, such as GPU-enabled instances, that can handle complex simulations and calculations. These resources can be used to run molecular dynamics simulations, quantum chemistry calculations, and other computational chemistry and materials science workflows.</li><li>Financial modeling and simulation: AWS provides a range of compute and storage resources that can be used for financial modeling and simulation tasks. These resources can be used to run Monte Carlo simulations, backtesting, portfolio optimization, and other financial analysis workflows.</li><li>Weather and climate modeling: AWS provides a range of weather and climate data services, such as Amazon Forecast, that can be used to generate forecasts and predictions. These services can be used to run weather and climate simulations, analyze large datasets, and generate forecasts for various applications.</li><li>Machine learning and AI: AWS provides a range of machine learning and AI services, such as Amazon SageMaker and Amazon Rekognition, that can be used to train and deploy machine learning models. These services can be used for natural language processing, image and video analysis, and other AI workflows.</li></ul><div><div>Area of research in HPC is the development of container-based cloud environments that are designed to support scientific workloads. These environments aim to provide a flexible and scalable platform for running scientific applications, while also addressing challenges such as data movement, job scheduling, and resource management. As mentioned earlier , Singularity is designed to support the needs of scientific and HPC workloads, and provides features such as secure image management, container mobility, and seamless integration with HPC schedulers. Another area of research is the development of new programming models and frameworks that are designed to simplify the development of HPC applications in container-based environments. For example, the Big Data and Extreme Computing (BDEC) project is developing a framework that provides a high-level abstraction layer for managing and orchestrating distributed scientific workflows. The project has the potential to make it easier and more efficient to manage large-scale scientific workflows in a variety of domains.</div><div><br /></div></div></div></div><p></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-36674264917554387742022-06-10T09:37:00.024+05:302022-07-18T17:04:55.889+05:30Frontier supercomputer powered by AMD is the fastest and first exascale machine<p style="text-align: justify;">Exascale computing is the next milestone in the development of supercomputers. Able to process information much faster than today’s most powerful supercomputers, exascale computers will give scientists a new tool for addressing some of the biggest challenges facing our world, from climate change to understanding cancer to designing new kinds of materials. </p><p style="text-align: justify;">One way scientists measure computer performance is in floating point operations per second (FLOPS). These involve simple arithmetic like addition and multiplication problems. Their performance in FLOPS has so many zeros - researchers instead use prefixes like Giga, Tera, Exa. where “Exa” means 18 zeros. That means an exascale computer can perform more than 1,000,000,000,000,000,000 FLOPS, or 1 exaFLOP. DOE is deploying the United States’ first exascale computers: Frontier at ORNL and Aurora at Argonne National Laboratory and El Capitan at Lawrence Livermore National Laboratory.</p><p style="text-align: justify;">Exascale supercomputers will allow scientists to create more realistic Earth system and climate models. They will help researchers understand the nanoscience behind new materials. Exascale computers will help us build future fusion power plants. They will power new studies of the universe, from particle physics to the formation of stars. And these computers will help ensure the safety and security of the United States by supporting tasks such as the maintenance of US nuclear deterrent.</p><p style="text-align: justify;">For decades, the performance maximization has been the chief concern of both the hardware architects and the software developers. Due to end of performance scaling by increasing CPUs clock frequencies (i.e Moore's Law), Industries transit from single-core to multi-core and many-core architectures. As a result, the hardware acceleration and the use of co-processors together with CPU are becoming a popular choice to gain the performance boost while keeping the power budget low. This includes both the new customized hardware for particular application domain such as Tensor Processing Unit (TPU), Vision Processing Unit (VPU) and Neural Processing Unit (NPU); and the modifications in existing platforms such as Intel Xeon Phi co-processors, general purpose GPUs and Field Programmable Gate Array (FPGA)s. Such accelerators together with main processors and memory, constitute a heterogeneous system. However, this heterogeneity has raised unprecedented difficulties posed to performance and energy optimization of modern heterogeneous HPC platforms. </p><p style="text-align: justify;">The focus of maximizing the performance of HPC in terms of completing the hundreds of trillion Floating Point Operations Per Second (FLOPS) has led the supercomputers to consume an enormously high amount of energy in terms of electricity and for cooling down purposes. As a consequence, current HPC systems are already consuming Megawatts of energy. Energy efficiency is becoming an equally important design concern with performance in ICT. Current HPC systems are already consuming Megawatts of energy. For example, the world’s powerful supercomputer like Summit consumes around 13 Megawatts of power which is roughly equivalent to the power draw of roughly over 10000 households. Because of such high power consumption, future HPC systems are highly likely to be power constrained. For example, DOE aims to deploy this exascale supercomputer capable of performing 1 million trillion ( or <span face="arial, sans-serif" style="background-color: white; color: #5f6368; font-size: 14px; font-weight: bold;">10<span style="position: relative; top: -0.4em; vertical-align: baseline;">18</span></span>) floating-point operations per second in a power envelope of 20-30 megawatts. Initial target was to deliver a double precision exaflops of compute capability for 20 megawatts of power and other target was 2 exaflops for 29 megawatts of power when it’s running at full power. Taking into consideration the above-mentioned factors, HPE Cray <span style="text-align: left;">designed </span><span style="text-align: left;">Frontier supercomputer powered by AMD </span><span style="text-align: left;">for </span><span style="text-align: left;">growing accelerated computational needs and power constraints.</span></p><p>The Frontier supercomputer, built at the Department of Energy's Oak Ridge National Laboratory in Tennessee, has now become the world's first known supercomputer to demonstrate a processor speed of 1.1 exaFLOPS (1.1 quintillion floating point operations per second, or FLOPS). The Frontier supercomputer's exascale performance is enabled by world's most advanced pieces of technology from HPE and AMD.</p><p>Frontier supercomputer powered by AMD is the first exascale machine meaning it can process more than a quintillion calculations per second with an HPL score of 1.102 Exaflop/s. Based on the latest HPE Cray EX235a architecture and equipped with AMD EPYC 64C 2GHz processors, the system has 8,730,112 total cores and a power efficiency rating of 52.23 gigaflops/watt. It relies on gigabit ethernet for data transfer. </p><p>Exascale is the next level of computing performance. By solving calculations five times faster than today’s top supercomputers—exceeding a quintillion [ or <span face="arial, sans-serif" style="background-color: white; color: #5f6368; font-size: 14px; font-weight: bold;">10<span style="position: relative; top: -0.4em; vertical-align: baseline;">18 </span></span> ] calculations per second—exascale systems will enable scientists to develop new technologies for energy, medicine, and materials. The Oak Ridge Leadership Computing Facility will be home to one of America’s first exascale systems, Frontier, which will help guide researchers to new discoveries at exascale.</p><p>It's based on HPE Cray’s new EX architecture and Slingshot interconnect with optimized 3rd Gen AMD EPYC™ CPUs for HPC and AI, and AMD Instinct™ 250X accelerators. It delivers linepack (double precision floating point – FP64) compute performance of 1.1 EFLOPS (ExaFLOPS). </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbfnDNMCmvp-4e1RHC472_xc6gP_QorWKRmvum7T480JmtdG1Fr0HVNn2NT0qBHJgFPQNGzmwcBaFIZLhbu7043s2T3X7T5lMORaEaT16FdfMsEqVYKjhRnbY5qOBdiT5QLaFQhyS4c5p5LGvzRqTFkDQIs720ZZG6Q7I09uSVF2c7zQ6qfgZ64V3uWw/s1179/frontier_supercomp.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="549" data-original-width="1179" height="298" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbfnDNMCmvp-4e1RHC472_xc6gP_QorWKRmvum7T480JmtdG1Fr0HVNn2NT0qBHJgFPQNGzmwcBaFIZLhbu7043s2T3X7T5lMORaEaT16FdfMsEqVYKjhRnbY5qOBdiT5QLaFQhyS4c5p5LGvzRqTFkDQIs720ZZG6Q7I09uSVF2c7zQ6qfgZ64V3uWw/w640-h298/frontier_supercomp.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.olcf.ornl.gov/wp-content/uploads/2019/05/frontier_specsheet.pdf" target="_blank"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><br /><p>The Frontier test and development system (TDS) secured the first place in the Green500 list, delivering 62.68 gigaflops/watt power-efficiency from a single cabinet of optimised 3rd Gen AMD EPYC processors and AMD Instinct MI250x accelerators. It could lead to breakthroughs in medicine, astronomy, and more. </p><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="BLOG_video_class" height="266" src="https://www.youtube.com/embed/etVzy1z_Ptg" width="320" youtube-src-id="etVzy1z_Ptg"></iframe></div><br /><p></p><p>The HPE/AMD system delivers 1.102 Linpack exaflops of computing power in a 21.1-megawatt power envelope, an efficiency of 52.23 gigaflops per watt. Frontier only uses about 29 megawatts at its very peak. During a test, Frontier ran at 1.1 exaflops and could go as high as 2 exaflops.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3PmfFXPukdNTKylG-WmNZJINknNq6LQoU_tFcVmKLOpb1tZv1Jr_mrkx9TG3ZyUX-vLbDgcwxPEXUFKzxPrzVMc9cGa2y3ARYnjOQsJA4Rxkapbeog54Hc2hwNk31zF0UC-R5VUwAXHkLYcX51yV6fucvy5TXcUYr7eSisgHrEX5D9-e0hlZ1IiobxA/s1181/Fronntier_hardware.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="653" data-original-width="1181" height="354" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3PmfFXPukdNTKylG-WmNZJINknNq6LQoU_tFcVmKLOpb1tZv1Jr_mrkx9TG3ZyUX-vLbDgcwxPEXUFKzxPrzVMc9cGa2y3ARYnjOQsJA4Rxkapbeog54Hc2hwNk31zF0UC-R5VUwAXHkLYcX51yV6fucvy5TXcUYr7eSisgHrEX5D9-e0hlZ1IiobxA/w640-h354/Fronntier_hardware.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.olcf.ornl.gov/wp-content/uploads/2019/05/frontier_specsheet.pdf" target="_blank"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><br /><p><br /></p><p>Node diagram:</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj92uRqXnFM_0sQicA7u_kVyt0Zw2FdfnEDidyG5p1JejmT1adi0YzEhdmXILtSo4lFV7eG_r3hdFTe_LBYyDmemteTb-IPnbrAk8PiqPUU5WC5VLUWyDiw-PvrK9JINnMnwnHe-ShKlhJmurnLBz_ro4Uk7Syv8WdW8ThWUzbuTDG8Ip6V9AjL3yxDCg/s1075/Frontier_arch.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="709" data-original-width="1075" height="422" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj92uRqXnFM_0sQicA7u_kVyt0Zw2FdfnEDidyG5p1JejmT1adi0YzEhdmXILtSo4lFV7eG_r3hdFTe_LBYyDmemteTb-IPnbrAk8PiqPUU5WC5VLUWyDiw-PvrK9JINnMnwnHe-ShKlhJmurnLBz_ro4Uk7Syv8WdW8ThWUzbuTDG8Ip6V9AjL3yxDCg/w640-h422/Frontier_arch.PNG" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.olcf.ornl.gov/frontier/" target="_blank"><span style="font-size: xx-small;">Source</span></a></td></tr></tbody></table><p style="text-align: justify;">These are HPE Cray EX systems has 74 cabinets of this — 9,408 nodes. Each node has one CPU and four GPUs. The GPUs are the [AMD] MI250Xs. The CPUs are an AMD Epyc CPU. It’s all wired together with the high-speed Cray interconnect, called Slingshot. And it’s a water-cooled system. Recently good efforts towards using computational fluid dynamics to model the water flow in the cooling system. These are incredibly instrumented machines with liquid cooling dynamically adjust to the workloads. There’s sensors that are monitoring temperatures where even down to the individual components on the individual node-boards, so they can adjust the cooling levels up and down to make sure that the system stays at a safe temperature. It was estimated to provide over 60 gigaflops-per-watt for the single cabinet run.</p><p style="text-align: justify;">This is the datacenter where they formerly had the Titan supercomputer. So they removed that supercomputer and refurbished this datacenter. That needed more power and more cooling. So they brought in 40 megawatts of power to the datacenter and have 40 megawatts of cooling available. Frontier really only uses about 29 megawatts of that at its very peak. And this Supercomputer is even a little bit quieter than Summit because they’re going to liquid-cooled with no fans and no rear doors where exchanging heat with the room. It’s 100 percent liquid cooled, and the [fan] noise generated from storage systems that are also HPE and are air-cooled.</p><p style="text-align: justify;">At OLCF [Oak Ridge Leadership Computing Facility], they have the Center for Accelerated Application Readiness, we call it CAAR. Its vehicle for application readiness. That group supports eight apps for the OLCF and 12 apps for the Exascale Computing Project. Frontier was OLCF-5, the next system will be OLCF-6.</p><p style="text-align: justify;">The result was confirmed in a benchmarking test called High-Performance Linpack (HPL). As impressive as that sounds, the ultimate limits of Frontier are even more staggering, with the supercomputer theoretically capable of a peak performance of 2 quintillion calculations per second. Among all these massively powerful supercomputers, only Frontier has achieved true exascale performance, at least where it counts, according to TOP500. S<span style="text-align: left;">ome of the most exciting things are the work in artificial intelligence and those workloads. Plan for research teams to develop better treatments for different diseases, how to improve efficacies of treatments, and these systems are capable of digesting just incredible amounts of data. Thousands of laboratory reports or pathology reports, can draw inferences across these reports that no human being could ever do but that a supercomputer can do. </span><span style="text-align: left;">They still have Summit here, a previous Top500 number-one system, an IBM/Nvidia machine. </span><span style="text-align: left;">It’s highly utilized at this point. </span><span style="text-align: left;">They will at least run it for a year and overlap with Frontier </span><span style="text-align: left;"> so that we can make sure that Frontier is up and stable and give people time to transition their data and their applications over to the new system.</span></p><p style="text-align: justify;">With the Linpack exaflops milestone achieved by the Frontier supercomputer at Oak Ridge National Laboratory, the United States is turning its attention to the next crop of exascale machines, some 5-10x more performant than Frontier. At least one such system is being planned for the 2025-2030 timeline, and the DOE is soliciting input from the vendor community to inform the design and procurement process. That can solve scientific problems 5 to 10 times faster – or solve more complex problems, such as those with more physics or requirements for higher fidelity – than the current state-of-the-art systems. These future systems will include associated networks and data hierarchies. A capable software stack will meet the requirements of a broad spectrum of applications and workloads, including large-scale computational science campaigns in modeling and simulation, machine intelligence, and integrated data analysis. They expect these systems to operate within a power envelope of 20–60 MW. These systems must be sufficiently resilient to hardware and software failures, in order to minimize requirements for user intervention. This could include the successor to Frontier (aka OLCF-6), the successor to Aurora (aka ALCF-5), the successor to Crossroads (aka ATS-5), the successor to El Capitan (aka ATS-6) as well as a future NERSC system (possibly NERSC-11). Note that of the “predecessor systems,” only Frontier has been installed so far. A key thrust of the DOE supercomputing strategy is the creation of an Advanced Computing Ecosystem (ACE) that enables “integration with other DOE facilities, including light source, data, materials science, and advanced manufacturing. The next generation of supercomputers will need to be capable of being integrated into an ACE environment that supports automated workflows, combining one or more of these facilities to reduce the time from experiment and observation to scientific insight.</p><p style="text-align: justify;"><span style="text-align: left;">The original CORAL contract called for three pre-exascale systems (~100-200 petaflops each) with at least two different architectures to manage risk. Only two systems – Summit at Oak Ridge and Sierra at Livermore – were completed in the intended timeframe, using nearly the same heterogeneous IBM-Nvidia architecture. CORAL-2 took a similar tack, calling for two or three exascale-class systems with at least two distinct architectures. The program is procuring two systems – Frontier and El Capitan – both based on a similar heterogenous HPE AMD+AMD architecture. The redefined Aurora – which is based on the heterogenous HPE Intel+Intel architecture – becomes the “architecturally-diverse” third system (although it technically still belongs to the first CORAL contract).</span></p><p style="text-align: justify;"><span style="text-align: left;">-----</span></p><p><span style="font-size: xx-small;">Reference:</span></p><p><span style="font-size: xx-small;">https://www.olcf.ornl.gov/wp-content/uploads/2019/05/frontier_specsheet.pdf</span></p><p><span style="font-size: xx-small;">https://www.researchgate.net/figure/Power-consumption-by-top-10-supercomputers-over-time-based-on-the-results-published_fig2_350176475</span></p><p><span style="font-size: xx-small;">https://www.youtube.com/watch?v=HvJGsF4t2Tc</span></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-89080287352700012142022-02-14T19:13:00.010+05:302022-09-15T11:26:02.843+05:30Open MPI with hierarchical collectives (HCOLL) Algorithms <p><span style="text-align: justify;">MPI, an acronym for Message Passing Interface, is a library specification for parallel computing architectures, which allows for communication of information between various nodes and clusters. Today, MPI is the most common protocol used in high performance computing (HPC).</span></p><p style="text-align: justify;">The Open MPI Project is an open source Message Passing Interface implementation that is developed and maintained by a consortium of academic, research, and industry partners. Open MPI is therefore able to combine the expertise, technologies, and resources from all across the High Performance Computing community in order to build the best MPI library available. Open MPI offers advantages for system and software vendors, application developers and computer science researchers.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjwbL9Q_J12vCEP2OlByMRIcDXFI8UiE-ec97iC9cWTREUyJwBH1V_NPE_1nOvp4S38N9fJEQYm1uK_Zzfd1OS7ZZ-21mVKBC8AOr3AO-TBLFu8agPWb-4e6xr-5BgI1mlrLSYIfI_kznDVy5R--2OjqbFJywIXBlV4lc3o_QXmXXXAEP5e86sWtoipCw=s924" style="margin-left: auto; margin-right: auto;"><img alt="https://developer.nvidia.com/blog/benchmarking-cuda-aware-mpi/" border="0" data-original-height="513" data-original-width="924" height="357" src="https://blogger.googleusercontent.com/img/a/AVvXsEjwbL9Q_J12vCEP2OlByMRIcDXFI8UiE-ec97iC9cWTREUyJwBH1V_NPE_1nOvp4S38N9fJEQYm1uK_Zzfd1OS7ZZ-21mVKBC8AOr3AO-TBLFu8agPWb-4e6xr-5BgI1mlrLSYIfI_kznDVy5R--2OjqbFJywIXBlV4lc3o_QXmXXXAEP5e86sWtoipCw=w640-h357" title="source" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://developer.nvidia.com/blog/benchmarking-cuda-aware-mpi/"><span style="font-size: xx-small;">source</span></a></td></tr></tbody></table><p style="text-align: justify;">Open MPI is developed in a true open source fashion by a consortium of research, academic, and industry partners. Latest version of Open MPI: Version 4.1.</p><p>Download OpenMPI from link <span style="font-size: xx-small;"><a href="https://www.open-mpi.org/software/ompi/v4.1/">https://www.open-mpi.org/software/ompi/v4.1/</a></span></p><p>Example <a href="https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.1.tar.gz]">https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.1.tar.gz</a> </p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj2P1T6h2smqy5SNfPLR8uYhttaJo8La-4Nw1gkI73aif48nMxPjN-NiuCEvcQrauo80ifN307sSws0ZdCZ8rMhC6zlJSTpZhvq-BPchgnTLu6MpdcxfJoMUYgCeWvR_J2fX-d-HthS6O9FdZxuMkYXeLoGcoeW7Yr2HJcuUiSnzWGmlCLn1yPMZuRM8w=s916" style="margin-left: auto; margin-right: auto;"><img alt="source" border="0" data-original-height="791" data-original-width="916" height="552" src="https://blogger.googleusercontent.com/img/a/AVvXsEj2P1T6h2smqy5SNfPLR8uYhttaJo8La-4Nw1gkI73aif48nMxPjN-NiuCEvcQrauo80ifN307sSws0ZdCZ8rMhC6zlJSTpZhvq-BPchgnTLu6MpdcxfJoMUYgCeWvR_J2fX-d-HthS6O9FdZxuMkYXeLoGcoeW7Yr2HJcuUiSnzWGmlCLn1yPMZuRM8w=w640-h552" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/hpc/setup-mpi" target="_blank"><span style="font-size: xx-small;">source</span></a></td></tr></tbody></table><p>NOTE: NVIDIA Mellanox HPC-X is a comprehensive software package that includes MPI and SHMEM communications libraries. HPC-X uses 'hcoll' library for collective communication and 'hcoll' is enabled by default in HPC-X on Azure HPC VMs and can be controlled at runtime by using the parameter[-mca coll_hcoll_enable 1]</p><p><b>How to install UCX :</b></p><p>Unified Communication X (UCX) is a framework of communication APIs for HPC. It is optimized for MPI communication over InfiniBand and works with many MPI implementations such as OpenMPI and MPICH.</p><p style="text-align: left;"></p><ul style="text-align: left;"><li>wget https://github.com/openucx/ucx/releases/download/v1.4.0/ucx-1.4.0.tar.gz</li><li>tar -xvf ucx-1.4.0.tar.gz</li><li>cd ucx-1.4.0</li><li>./configure --prefix=<ucx-install-path> </li><li>make -j 8 && make install</li></ul><p></p><p><b>Optimizing MPI collectives and hierarchical communication algorithms (HCOLL):</b></p><p style="text-align: justify;">MPI Collective communication primitives offer a flexible, portable way to implement group communication operations. They are widely used across various scientific parallel applications and have a significant impact on the overall application performance. Refer configuration parameters to optimize collective communication performance using HPC-X and HCOLL library for collective communication.</p><p style="text-align: justify;">As an example, if you suspect your tightly coupled MPI application is doing an excessive amount of collective communication, you can try enabling hierarchical collectives (HCOLL). To enable those features, use the following parameters.</p><p><br /></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="background-color: #cccccc;">-mca coll_hcoll_enable 1 -x HCOLL_MAIN_IB=<MLX device>:<Port></span></p></blockquote><p><b>HCOLL :</b></p><p>Scalable infrastructure: Designed and implemented with current and emerging “extreme-scale” systems in mind</p><p style="text-align: left;"></p><ul style="text-align: left;"><li>Scalable communicator creation, memory consumption, runtime interface</li><li>Asynchronous execution</li><li>Blocking and non-blocking collective routines</li><li>Easily integrated into other packages</li><li>Successfully integrated into OMPI – “hcoll” component in “coll” framework</li><li>Successfully integrated in Mellanox OSHMEM</li><li>Experimental integration in MPICH</li><li>Host level hierarchy awareness</li><li>Socket groups, UMA groups</li><li>Exposes Mellanox and InfiniBand specific capabilities</li></ul><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiG4zDYh53UVC4w5S058ZkG7TKRuVV5cwvVbKMVJMnSyySxB6J1oYcmmgYFup7ein5ZPUZ27h643IUNXtZe9dGwcW7C7uHA6Jyl2gpnB5y_VcF-Zo8K9x3MgyFKs2oXdnOxs8Az13SXG9mp2cv3y2Zz8Ai1x9rLWKuOvNWUtQRrGCZylNttC0HjmqE3yA=s728" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="728" height="422" src="https://blogger.googleusercontent.com/img/a/AVvXsEiG4zDYh53UVC4w5S058ZkG7TKRuVV5cwvVbKMVJMnSyySxB6J1oYcmmgYFup7ein5ZPUZ27h643IUNXtZe9dGwcW7C7uHA6Jyl2gpnB5y_VcF-Zo8K9x3MgyFKs2oXdnOxs8Az13SXG9mp2cv3y2Zz8Ai1x9rLWKuOvNWUtQRrGCZylNttC0HjmqE3yA=w640-h422" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="http://mug.mvapich.cse.ohio-state.edu/static/media/mug/presentations/18/bureddy-mug-18.pdf" target="_blank"><span style="font-size: xx-small;">source</span></a></td></tr></tbody></table><p></p><p><b>How to build OpenMPI with HCOLL</b></p><p>Install UCX as described above and build with HCOLL as shown below </p><p><b><span style="font-size: x-small;">Steps:</span></b></p><p></p><ol style="text-align: left;"><li>./configure --with-lsf=/LSF_HOME/10.1/ --with-lsf-libdir=/LSF_HOME/10.1/linux3.10-glibc2.17-ppc64le/lib/ --disable-man-pages --enable-mca-no-build=btl-uct --enable-mpi1-compatibility --prefix $MY_HOME/openmpi-4.1.1/install --with-ucx=/ucx-install_dir CPPFLAGS=-I/ompi/opal/mca/hwloc/hwloc201/hwloc/include --cache-file=/dev/null --srcdir=. --disable-option-checking</li><li>make </li><li>make install</li></ol><p></p><p>---------------------------<b>Set Test Environment</b>------------------------------------------------</p><p></p><ol style="text-align: left;"><li> export PATH=$MY_HOME/openmpi-4.1.1/install/bin:$PATH</li><li> export LD_LIBRARY_PATH=$MY_HOME/openmpi- 4.1.1/install/lib:/opt/mellanox/hcoll/lib:/opt/mellanox/sharp/lib:$LD_LIBRARY_PATH</li><li> export OPAL_PREFIX=$MY_HOME/openmpi-4.1.1/install</li></ol><div>NOTE: It may be necessary to explicitly pass LD_LIBRARY_PATH as mentioned in (3)</div><p></p><p>-------------- <b>How to run mpi testcase without HCOLL</b>--------------------------------------</p><p>1) Use these --mca option to disable HCOLL</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;">--mca coll_hcoll_enable 0 </span></p><p style="text-align: left;"><span style="background-color: #cccccc;">--mca coll_hcoll_priority 0 </span></p></blockquote><p>2) Add <span style="background-color: #cccccc;">--mca coll_base_verbose 10</span> to get more details </p><p>3) Add <span style="background-color: #cccccc;">-x LD_LIBRARY_PATH</span> to get the proper path as shown below</p><p><br /></p><p>-----------------------------<b>Execute Testcase</b> ----------------------------------</p><p><b>Testcase source: </b> https://github.com/jeffhammond/BigMPI/tree/master/test</p><p>$MY_HOME/openmpi-4.1.1/install/bin/mpirun --np 4 --npernode 1 --host host01,host02,host03,host04 -x LD_LIBRARY_PATH -x BIGCOUNT_MEMORY_PERCENT=6 -x BIGCOUNT_MEMORY_DIFF=10 -x HCOLL_RCACHE=^ucs -mca coll_hcoll_enable 0 --mca coll_hcoll_priority 0 test_allreduce_uniform_count</p><p style="text-align: left;"><span style="background-color: #cccccc;">--------------------------------------------------------------------------<br /><br />INT_MAX : 2147483647<br />UINT_MAX : 4294967295<br />SIZE_MAX : 18446744073709551615<br />----------------------:-----------------------------------------<br /> : Count x Datatype size = Total Bytes<br />TEST_UNIFORM_COUNT : 2147483647<br />V_SIZE_DOUBLE_COMPLEX : 2147483647 x 16 = 32.0 GB<br />V_SIZE_DOUBLE : 2147483647 x 8 = 16.0 GB<br />V_SIZE_FLOAT_COMPLEX : 2147483647 x 8 = 16.0 GB<br />V_SIZE_FLOAT : 2147483647 x 4 = 8.0 GB<br />V_SIZE_INT : 2147483647 x 4 = 8.0 GB<br />----------------------:-----------------------------------------<br />Results from MPI_Allreduce(int x 2147483647 = 8589934588 or 8.0 GB):<br />Rank 2: PASSED<br />Rank 3: PASSED<br />Rank 0: PASSED<br />Rank 1: PASSED<br />--------------------- Adjust count to fit in memory: 2147483647 x 50.0% = 1073741823<br />Root : payload 34359738336 32.0 GB = 16 dt x 1073741823 count x 2 peers x 1.0 inflation<br />Peer : payload 34359738336 32.0 GB = 16 dt x 1073741823 count x 2 peers x 1.0 inflation<br />Total : payload 34359738336 32.0 GB = 32.0 GB root + 32.0 GB x 0 local peers<br />---------------------<br />Results from MPI_Allreduce(double _Complex x 1073741823 = 17179869168 or 16.0 GB):<br />Rank 0: PASSED<br />Rank 2: PASSED<br />Rank 3: PASSED<br />Rank 1: PASSED<br />---------------------<br />Results from MPI_Iallreduce(int x 2147483647 = 8589934588 or 8.0 GB):<br />Rank 2: PASSED<br />Rank 0: PASSED<br />Rank 3: PASSED<br />Rank 1: PASSED<br />--------------------- Adjust count to fit in memory: 2147483647 x 50.0% = 1073741823<br />Root : payload 34359738336 32.0 GB = 16 dt x 1073741823 count x 2 peers x 1.0 inflation<br />Peer : payload 34359738336 32.0 GB = 16 dt x 1073741823 count x 2 peers x 1.0 inflation<br />Total : payload 34359738336 32.0 GB = 32.0 GB root + 32.0 GB x 0 local peers<br />---------------------<br />Results from MPI_Iallreduce(double _Complex x 1073741823 = 17179869168 or 16.0 GB):<br />Rank 2: PASSED<br />Rank 0: PASSED<br />Rank 3: PASSED<br />Rank 1: PASSED<br />[smpici@host01 BigCount]$</span></p><p>=====================Example for A data integrity issue (DI issue)====</p><p>There is end-to-end data integrity checks to detect data corruption. If any DI issue observed , it will be critical (high priority/ high severity defect)</p><p>DI issue with HCOLL ---let's see an example for DI issue.</p><p>$MY_HOME/openmpi-4.1.1/install/bin/mpirun --np 4 --npernode 1 --host host01,host02,host03,host04 -x LD_LIBRARY_PATH -x BIGCOUNT_MEMORY_PERCENT=6 -x BIGCOUNT_MEMORY_DIFF=10 -x HCOLL_RCACHE=^ucs --mca coll_hcoll_enable 1 --mca coll_hcoll_priority 98 test_allgatherv_uniform_count </p><p><br /></p><p><span style="background-color: #cccccc;">Results from MPI_Allgatherv(double _Complex x 2147483644 = 34359738304 or 32.0 GB): Mode: PACKED MPI_IN_PLACE<br />Rank 2: ERROR: DI in 805306368 of 2147483644 slots ( 37.5 % wrong)<br />Rank 0: ERROR: DI in 805306368 of 2147483644 slots ( 37.5 % wrong)<br />Rank 3: ERROR: DI in 805306368 of 2147483644 slots ( 37.5 % wrong)<br />Rank 1: ERROR: DI in 805306368 of 2147483644 slots ( 37.5 % wrong)</span></p><p><span style="background-color: #cccccc;"><br /></span></p><p>---------------Lets run the same testcase without HCOLL-------------------------------------------</p><p><br /></p><p>$MY_HOME/openmpi-4.1.1/install/bin/mpirun --np 4 --npernode 1 --host host01,host02,host03,host04 -x LD_LIBRARY_PATH -x BIGCOUNT_MEMORY_PERCENT=6 -x BIGCOUNT_MEMORY_DIFF=10 -x HCOLL_RCACHE=^ucs --mca coll_hcoll_enable 0 --mca coll_hcoll_priority 0 test_allgatherv_uniform_count </p><p style="text-align: left;"><span style="background-color: #cccccc;">Results from MPI_Allgatherv(double _Complex x 2147483644 = 34359738304 or 32.0 GB): Mode: PACKED MPI_IN_PLACE<br />Rank 0: PASSED<br />Rank 2: PASSED<br />Rank 3: PASSED<br />Rank 1: PASSED<br /><br />Results from MPI_Iallgatherv(double _Complex x 2147483644 = 34359738304 or 32.0 GB): Mode: PACKED MPI_IN_PLACE<br />Rank 3: PASSED<br />Rank 2: PASSED<br />Rank 0: PASSED<br />Rank 1: PASSED</span></p><p style="text-align: left;"><span style="background-color: #cccccc;">=========================</span></p><p>How to enable and disable HCOLL to find Mellanox defect and use mca coll_base_verbose 10 to get more debug information </p><p><b>CASE 1:</b> Enable HCOLL and run bigcount with allreduce</p><p><span style="font-size: x-small;">[smpici@myhostn01 collective-big-count]$ /nfs_smpi_ci/sachin/ompi_4-0.x/openmpi-4.0.7/install/bin/mpirun --timeout 500 --np 4 --npernode 1 -host myhost01:1,myhost02:1,myhost03:1,myhost04:1 -mca coll_hcoll_np 0 -mca coll_hcoll_enable 1 --mca coll_hcoll_priority 98 -x BIGCOUNT_MEMORY_PERCENT=6 -x BIGCOUNT_MEMORY_DIFF=10 -x BIGCOUNT_ENABLE_NONBLOCKING="0" -x HCOLL_RCACHE=^ucs /nfs_smpi_ci/sachin/bigmpi_ompi/ibm-tests/tests/big-mpi/BigCountUpstream/ompi-tests-public/collective-big-count/test_allreduce_uniform_count</span></p><p style="text-align: left;"><span style="font-size: x-small;">--------------------------------------------------------------------------<br /><br />Total Memory Avail. : 567 GB<br />Percent memory to use : 6 %<br />Tolerate diff. : 10 GB<br />Max memory to use : 34 GB<br />----------------------:-----------------------------------------<br />INT_MAX : 2147483647<br />UINT_MAX : 4294967295<br />SIZE_MAX : 18446744073709551615<br />----------------------:-----------------------------------------<br /> : Count x Datatype size = Total Bytes<br />TEST_UNIFORM_COUNT : 2147483647<br />V_SIZE_DOUBLE_COMPLEX : 2147483647 x 16 = 32.0 GB<br />V_SIZE_DOUBLE : 2147483647 x 8 = 16.0 GB<br />V_SIZE_FLOAT_COMPLEX : 2147483647 x 8 = 16.0 GB<br />V_SIZE_FLOAT : 2147483647 x 4 = 8.0 GB<br />V_SIZE_INT : 2147483647 x 4 = 8.0 GB<br />----------------------:-----------------------------------------<br />---------------------<br />Results from MPI_Allreduce(int x 2147483647 = 8589934588 or 8.0 GB):<br />Rank 2: PASSED<br />Rank 3: PASSED<br />Rank 0: PASSED<br />Rank 1: PASSED<br />--------------------- Adjust count to fit in memory: 2147483647 x 50.0% = 1073741823<br />Root : payload 34359738336 32.0 GB = 16 dt x 1073741823 count x 2 peers x 1.0 inflation<br />Peer : payload 34359738336 32.0 GB = 16 dt x 1073741823 count x 2 peers x 1.0 inflation<br />Total : payload 34359738336 32.0 GB = 32.0 GB root + 32.0 GB x 0 local peers<br />---------------------<br />Results from MPI_Allreduce(double _Complex x 1073741823 = 17179869168 or 16.0 GB):<br />--------------------------------------------------------------------------<br />The user-provided time limit for job execution has been reached:<br /><br /> Timeout: 500 seconds<br /><br />The job will now be aborted. Please check your code and/or<br />adjust/remove the job execution time limit (as specified by --timeout<br />command line option or MPIEXEC_TIMEOUT environment variable).<br />--------------------------------------------------------------------------<br />[smpici@myhostn01 collective-big-count]$</span></p><p><b>CASE 2: </b>Disable HCOLL and run bigcount with allreduce</p><p style="text-align: left;"><span style="font-size: x-small;">[user1@myhostn01 collective-big-count]$ /nfs_smpi_ci/sachin/ompi_4-0.x/openmpi-4.0.7/install/bin/mpirun --np 4 --npernode 1 -host myhost01:1,myhost02:1,myhost03:1,myhost04:1 --mca coll_base_verbose 10 -mca coll_hcoll_np 0 -mca coll_hcoll_enable 0 --mca coll_hcoll_priority 0 -x BIGCOUNT_MEMORY_PERCENT=6 -x BIGCOUNT_MEMORY_DIFF=10 -x BIGCOUNT_ENABLE_NONBLOCKING="0" -x HCOLL_RCACHE=^ucs /nfs_smpi_ci/sachin/bigmpi_ompi/ibm-tests/tests/big-mpi/BigCountUpstream/ompi-tests-public/collective-big-count/test_allreduce_uniform_count<br /><br />[myhostn01:1984671] coll:base:comm_select: component disqualified: self (priority -1 < 0)<br />[myhostn01:1984671] coll:sm:comm_query (0/MPI_COMM_WORLD): intercomm, comm is too small, or not all peers local; disqualifying myself<br />[myhostn01:1984671] coll:base:comm_select: component not available: sm<br />[myhostn01:1984671] coll:base:comm_select: component disqualified: sm (priority -1 < 0)<br />[myhostn01:1984671] coll:base:comm_select: component not available: sync<br />[myhostn01:1984671] coll:base:comm_select: component disqualified: sync (priority -1 < 0)<br />[myhostn01:1984671] coll:base:comm_select: component available: tuned, priority: 30<br />[myhostn01:1984671] coll:base:comm_select: component not available: hcoll<br /><br />----------------------:-----------------------------------------<br />Total Memory Avail. : 567 GB<br />Percent memory to use : 6 %<br />Tolerate diff. : 10 GB<br />Max memory to use : 34 GB<br />----------------------:-----------------------------------------<br />INT_MAX : 2147483647<br />UINT_MAX : 4294967295<br />SIZE_MAX : 18446744073709551615<br />----------------------:-----------------------------------------<br /> : Count x Datatype size = Total Bytes<br />TEST_UNIFORM_COUNT : 2147483647<br />V_SIZE_DOUBLE_COMPLEX : 2147483647 x 16 = 32.0 GB<br />V_SIZE_DOUBLE : 2147483647 x 8 = 16.0 GB<br />V_SIZE_FLOAT_COMPLEX : 2147483647 x 8 = 16.0 GB<br />V_SIZE_FLOAT : 2147483647 x 4 = 8.0 GB<br />V_SIZE_INT : 2147483647 x 4 = 8.0 GB<br />----------------------:-----------------------------------------<br />---------------------<br />Results from MPI_Allreduce(int x 2147483647 = 8589934588 or 8.0 GB):<br />Rank 2: PASSED<br />Rank 0: PASSED<br />Rank 3: PASSED<br />Rank 1: PASSED<br />--------------------- Adjust count to fit in memory: 2147483647 x 50.0% = 1073741823<br />Root : payload 34359738336 32.0 GB = 16 dt x 1073741823 count x 2 peers x 1.0 inflation<br />Peer : payload 34359738336 32.0 GB = 16 dt x 1073741823 count x 2 peers x 1.0 inflation<br />Total : payload 34359738336 32.0 GB = 32.0 GB root + 32.0 GB x 0 local peers<br />---------------------<br />Results from MPI_Allreduce(double _Complex x 1073741823 = 17179869168 or 16.0 GB):<br />Rank 0: PASSED<br />Rank 3: PASSED<br />Rank 2: PASSED<br />Rank 1: PASSED<br />[user1@myhostn01 collective-big-count]$</span></p><p style="text-align: left;"><span style="background-color: #cccccc;">=================================================================</span></p><p style="text-align: justify;">This post briefly shows features for optimal collective communication performance and highlights the general recommendations. The real application performance depends on your application characteristics, runtime configuration, transport protocols, processes per node (ppn) configuration... etc.</p><p style="text-align: justify;"><br /></p><p style="text-align: left;"><span style="font-size: x-small;">Reference:</span><br /><span style="font-size: x-small;">http://mug.mvapich.cse.ohio-state.edu/static/media/mug/presentations/18/bureddy-mug-18.pdf</span><br /><span style="font-size: x-small;">https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/hpc/setup-mpi</span></p><p><br /></p><p><br /></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-3490728427850127092022-01-28T20:30:00.004+05:302022-02-06T20:14:39.127+05:30HPC Clusters in a Multi-Cloud Environment<p style="text-align: justify;">High performance computing (HPC) is the ability to process data and perform complex calculations at high speeds. One of the best-known types of HPC solutions is the supercomputer. A supercomputer contains thousands of compute nodes that work together to complete one or more tasks. This is called parallel processing. HPC solutions have three main components: Compute , Network and Storage. To build a high performance computing architecture, compute servers are networked together into a cluster. Software programs and algorithms are run simultaneously on the servers in the cluster. The cluster is networked to the data storage to capture the output. Together, these components operate seamlessly to complete a diverse set of tasks.</p><p style="text-align: justify;">To operate at maximum performance, each component must keep pace with the others. For example, the storage component must be able to feed and ingest data to and from the compute servers as quickly as it is processed. Likewise, the networking components must be able to support the high-speed transportation of data between compute servers and the data storage. If one component cannot keep up with the rest, the performance of the entire HPC infrastructure suffers.</p><p style="text-align: justify;">Containers give HPC the portability that Hybrid Cloud demands .Containers are ready-to-execute packages of software. Container technology provides hardware abstraction, wherein the container is not tightly coupled with the server. Abstraction between the hardware and software stacks provides ease of access, ease of use, and the agility that bare metal environments lack.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-CYAtwgrjmXQ/YTyB23hrx0I/AAAAAAAAXDc/xBXbvQ4Qm0MIJTX3_4xOa4gQn13lbGEPgCLcBGAsYHQ/s819/HPC_cloud.PNG" style="margin-left: auto; margin-right: auto;"><img alt="Source" border="0" data-original-height="610" data-original-width="819" height="476" src="https://1.bp.blogspot.com/-CYAtwgrjmXQ/YTyB23hrx0I/AAAAAAAAXDc/xBXbvQ4Qm0MIJTX3_4xOa4gQn13lbGEPgCLcBGAsYHQ/w640-h476/HPC_cloud.PNG" title="Source" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: xx-small;"><a href="https://www.theubercloud.com/cloud-hpc#cloud_hpc_technology" target="_blank">Source</a></span></td></tr></tbody></table><p style="text-align: justify;">Software containers and Kubernetes are important tools for building, deploying, running and managing modern enterprise applications at scale and delivering enterprise software faster and more reliably to the end user while using resources more efficiently and reducing costs. Recently, high performance computing (HPC) is moving closer to the enterprise and can therefore benefit from an HPC container and Kubernetes ecosystem, with new requirements to quickly allocate and deallocate computational resources to HPC workloads so that planning of compute capacity no longer required in advance. The HPC community is picking up the concept and applying it to batch jobs and interactive applications.</p><p style="text-align: justify;">In a multi-cloud environment, an enterprise utilizes multiple public cloud services, most often from different cloud providers. For example, an organization might host its web front-end application on AWS and host its Exchange servers on Microsoft Azure. Since all cloud providers are not created equal, organizations adopt a multi-cloud strategy to deliver best of breed IT services, to prevent lock-in to a single cloud provider, or to take advantages of cloud arbitrage and choose providers for specific services based on which provider is offering the lowest price at that time. Although it is similar to a hybrid cloud, multi-cloud specifically indicates more than one public cloud provider service and need not include a private cloud component at all. Enterprises adopt a multi-cloud strategy so as not to ‘keep all their eggs in a single basket’, for geographic or regulatory governance demands, for business continuity, or to take advantage of features specific to a particular provider.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-dPuIK63ycY8/YTIZtb-zcAI/AAAAAAAAXCw/-93QUpbsiDUbLPcW5Za_UmnMTmb5SzZggCLcBGAsYHQ/s1311/Hybrid-multi_cloud.PNG" style="margin-left: auto; margin-right: auto;"><img alt="source" border="0" data-original-height="147" data-original-width="1311" height="72" src="https://1.bp.blogspot.com/-dPuIK63ycY8/YTIZtb-zcAI/AAAAAAAAXCw/-93QUpbsiDUbLPcW5Za_UmnMTmb5SzZggCLcBGAsYHQ/w640-h72/Hybrid-multi_cloud.PNG" title="Source" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.vmware.com/topics/glossary/content/hybrid-cloud-vs-multi-cloud" target="_blank"><span style="font-size: xx-small;">source</span></a></td></tr></tbody></table><br /><p style="text-align: left;"><b>Multi-cloud </b>is the use of multiple cloud computing and storage services in a single network architecture. <span style="text-align: justify;">This refers to the distribution of cloud assets, software, applications, and more across several cloud environments. With a typical multi-cloud architecture utilizing two or more public clouds as well as private clouds, a multi-cloud environment aims to eliminate the reliance on any single cloud provider or instance.</span></p><p style="text-align: left;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgPB8hPluUffiNH6SbkN1CIiQcDuOml_eIvPFG3a8BbwroQkilxB5CMWv77mZdY6NxyiD1N7yltUO54Vh_wtFTbHbaPRlvJdfY2TI4LkWmK1RnGcxsmfTDVZTITLgn03qGpSHp3Pg34EObu73cw3CCtGgjjsjhashrdi8zmgmm_ZY5rd0lafQSaMg9qWQ=s955" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="689" data-original-width="955" height="231" src="https://blogger.googleusercontent.com/img/a/AVvXsEgPB8hPluUffiNH6SbkN1CIiQcDuOml_eIvPFG3a8BbwroQkilxB5CMWv77mZdY6NxyiD1N7yltUO54Vh_wtFTbHbaPRlvJdfY2TI4LkWmK1RnGcxsmfTDVZTITLgn03qGpSHp3Pg34EObu73cw3CCtGgjjsjhashrdi8zmgmm_ZY5rd0lafQSaMg9qWQ=s320" width="320" /></a></div><p></p><p style="text-align: justify;">Multi-cloud is the use of two or more cloud computing services from any number of different cloud vendors. A multi-cloud environment could be all-private, all-public or a combination of both. Companies use multi-cloud environments to distribute computing resources and minimize the risk of downtime and data loss. They can also increase the computing power and storage available to a business. Innovations in the cloud in recent years have resulted in a move from single-user private clouds to multi-tenant public clouds and hybrid clouds — a heterogeneous environment that leverages different infrastructure environments like the private and public cloud.</p><p style="text-align: justify;">A multi-cloud platform combines the best services that each platform offers. This allows companies to customize an infrastructure that is specific to their business goals. A multi-cloud architecture also provides lower risk. If one web service host fails, a business can continue to operate with other platforms in a multi-cloud environment versus storing all data in one place. Examples of public Cloud Providers: </p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><ul style="text-align: left;"><li><span style="text-align: justify;">IBM Cloud [<a href="https://www.ibm.com/cloud">https://www.ibm.com/cloud</a>]</span></li><li><span style="text-align: justify;">AWS [<a href="https://aws.amazon.com/">https://aws.amazon.com/</a>]</span></li><li><span style="text-align: justify;">Microsoft Azure [<a href="https://azure.microsoft.com/en-us/free/">https://azure.microsoft.com/en-us/free/</a>]</span></li><li><span style="text-align: justify;">Google Cloud Platform [<a href="https://cloud.google.com/">https://cloud.google.com/</a>]</span></li><li><span style="text-align: justify;">Openstack (private cloud)</span></li><li><span style="text-align: justify;">Rackspace</span></li><li>VMware Cloud </li></ul></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote></blockquote></blockquote><p style="text-align: justify;"><b>Hybrid-cloud </b>A hybrid cloud architecture is mix of on-premises, private, and public cloud services with orchestration between the cloud platforms. Hybrid cloud management involves unique entities that are managed as one across all environments. Hybrid cloud architecture allows an enterprise to move data and applications between private and public environments based on business and compliance requirements. For example, customer data can live in a private environment. But heavy processing can be sent to the public cloud without ever having customer data leave the private environment. Hybrid cloud computing allows instant transfer of information between environments, allowing enterprises to experience the benefits of both environments.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjiEIoMvyQoojaOH4lHFonZYFdrqxiO6Y9KTrRoyj9RdDiMsnDCEuVALIBqZVj3ns4mlRq7d7lLECItvij4VYCq0OfH-jxmdIUZIqxBVmaLAy_WQFwJRF8BcZE6YdYLHz6qhOErgxWidrTAetcK9Ih0F547__z4Cn4jAuJOmP3Z9XRIGGQEFCklabTqBQ=s934" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="597" data-original-width="934" height="256" src="https://blogger.googleusercontent.com/img/a/AVvXsEjiEIoMvyQoojaOH4lHFonZYFdrqxiO6Y9KTrRoyj9RdDiMsnDCEuVALIBqZVj3ns4mlRq7d7lLECItvij4VYCq0OfH-jxmdIUZIqxBVmaLAy_WQFwJRF8BcZE6YdYLHz6qhOErgxWidrTAetcK9Ih0F547__z4Cn4jAuJOmP3Z9XRIGGQEFCklabTqBQ=w400-h256" width="400" /></a></div><div class="separator" style="clear: both; text-align: center;"><span style="text-align: justify;"><br /></span></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div class="separator" style="clear: both; text-align: left;"><span style="text-align: justify;">Hybrid cloud architecture works well for the following industries:</span></div></blockquote><p style="text-align: justify;">• Finance: Financial firms are able to significantly reduce their space requirements in a hybrid cloud architecture when trade orders are placed on a private cloud and trade analytics live on a public cloud.</p><p style="text-align: justify;">• Healthcare: When hospitals send patient data to insurance providers, hybrid cloud computing ensures HIPAA compliance.</p><p style="text-align: justify;">• Legal: Hybrid cloud security allows encrypted data to live off-site in a public cloud while connected o a law firm’s private cloud. This protects original documents from threat of theft or loss by natural disaster.</p><p style="text-align: justify;">• Retail: Hybrid cloud computing helps companies process resource-intensive sales data and analytics.</p><p style="text-align: justify;">The hybrid cloud strategy could be applied to move workloads dynamically to the most appropriate IT environment based on cost, performance and security. Utilize on-premises resources for existing workloads, and use public or hosted clouds for new workloads. Run internal business systems and data on premises while customer-facing systems run on infrastructure as a service (iaaS), public or hosted clouds.</p><p><span style="font-size: x-small;"><b>Reference:</b></span></p><p style="text-align: left;"><span style="font-size: small;">https://www.hpcwire.com/2019/09/19/kubernetes-containers-and-hpc</span><br /><span style="font-size: small;">https://www.hpcwire.com/2020/03/19/kubernetes-and-hpc-applications-in-hybrid-cloud-environments-part-ii</span><br /><span style="font-size: small;">https://www.hpcwire.com/2021/09/02/kubernetes-based-hpc-clusters-in-azure-and-google-cloud-multi-cloud-environment</span><br /><span style="font-size: small;">https://www-stage.avinetworks.com/</span><br /><span style="font-size: small;">https://www.vmware.com/topics/glossary/content/hybrid-cloud-vs-multi-cloud</span></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-14531221662273487232021-08-22T12:57:00.042+05:302022-10-16T16:58:48.821+05:30Spectrum LSF 10.1 Installation and Applying Patch | FP | interim FIX on Linux Platform<p style="text-align: justify;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-DMnOo6l0PiY/YSIAsoFAJEI/AAAAAAAAW-0/FrKy6KJa324faRlCCpkWEKOCPd8thkGwACLcBGAsYHQ/s483/logo.PNG" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="131" data-original-width="483" height="54" src="https://1.bp.blogspot.com/-DMnOo6l0PiY/YSIAsoFAJEI/AAAAAAAAW-0/FrKy6KJa324faRlCCpkWEKOCPd8thkGwACLcBGAsYHQ/w200-h54/logo.PNG" width="200" /></a></div><div style="text-align: justify;">IBM Spectrum LSF (LSF, originally Platform Load Sharing Facility) is a workload management platform, job scheduler, for distributed high performance computing (HPC) by IBM. In January, 2012, Platform Computing was acquired by IBM. The product is now called IBM® Spectrum LSF.</div><p></p><p style="text-align: justify;">IBM® Spectrum LSF is a complete workload management solution for demanding HPC environments that takes your job requirements, finds the best resources to run the job, and monitors its progress. Jobs always run according to host load and site policies.<br /></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-5Fb6zokt8ag/YSH-OSapFQI/AAAAAAAAW-o/RJQY4_OGjhEb7v0nWsd3zS-Xk2Zm-AK2gCLcBGAsYHQ/s720/LSF-setup.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="578" data-original-width="720" height="321" src="https://1.bp.blogspot.com/-5Fb6zokt8ag/YSH-OSapFQI/AAAAAAAAW-o/RJQY4_OGjhEb7v0nWsd3zS-Xk2Zm-AK2gCLcBGAsYHQ/w400-h321/LSF-setup.PNG" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><span style="font-size: xx-small;"><a href="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=overview-lsf-introduction" target="_blank">LSF cluster (source)</a></span></td></tr></tbody></table><p></p><p></p><ul style="text-align: left;"><li style="text-align: justify;">Cluster is a group of computers (hosts) running LSF that work together as a single unit, combining computing power, workload, and resources. A cluster provides a single-system image for a network of computing resources. Hosts can be grouped into a cluster in a number of ways. A cluster can contain 1) All the hosts in a single administrative group 2) All the hosts on a subnetwork.</li><li style="text-align: justify;">Job is a unit of work that is running in the LSF system or job is a command or set of commands submitted to LSF for execution. LSF schedules, controls, and tracks the job according to configured policies.</li><li style="text-align: justify;">Queue is a cluster-wide container for jobs. All jobs wait in queues until they are scheduled and dispatched to hosts.</li><li>Resources are the objects in your cluster that are available to run work. </li></ul><p></p><p><b>Spectrum LSF 10.1 base Installation and applying FP /PTF/FIX</b></p><p>Plan your installation and install a new production IBM Spectrum LSF cluster on UNIX or Linux hosts. The following diagram illustrates an example directory structure after the LSF installation is complete.</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-xZNuEazMA7w/YSHGbKlKO3I/AAAAAAAAW-g/pGfx5QFTdA8GhRfBOH8pZ5ixxIX2XlutgCLcBGAsYHQ/s762/LSF_installation_dir_structure.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="762" data-original-width="719" height="640" src="https://1.bp.blogspot.com/-xZNuEazMA7w/YSHGbKlKO3I/AAAAAAAAW-g/pGfx5QFTdA8GhRfBOH8pZ5ixxIX2XlutgCLcBGAsYHQ/w604-h640/LSF_installation_dir_structure.jpg" title="LSF Installation directory Structure" width="604" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=linux-example-installation-directory-structure" target="_blank"><span style="font-size: x-small;">Source</span></a></td></tr></tbody></table><br /><p>Plan your installation to determine the required parameters for the install.config file.</p><p><span style="background-color: #eeeeee;">a ) lsf10.1_lsfinstall.tar.Z</span></p><p>The standard installer package. Use this package in a heterogeneous cluster with a mix of systems other than x86-64. Requires approximately 1 GB free space.</p><p><span style="background-color: #eeeeee;">b) lsf10.1_lsfinstall_linux_x86_64.tar.Z </span></p><p><span style="background-color: #eeeeee;"> lsf10.1_lsfinstall_linux_ppc64le.tar.Z</span></p><p>Use this smaller installer package in a homogeneous x86-64 or ppc cluster accordingly . </p><p>------------------------</p><p>Get the LSF distribution packages for all host types you need and put them in the same directory as the extracted LSF installer script. Copy that package to <span style="background-color: #eeeeee;">LSF_TARDIR</span> path mentioned in Step 3.</p><p>For example:</p><p>Linux 2.6 kernel glibc version 2.3, the distribution package is <b><span style="font-size: x-small;">lsf10.1_linux2.6-glibc2.3-x86_64.tar.Z</span></b>.</p><p>Linux kernel glibc version 3.x, the distribution package is<span style="font-size: x-small;"> <b>lsf10.1_lnx310-lib217-ppc64le.tar.Z</b></span></p><p>------------------------</p><p>LSF uses entitlement files to determine which feature set is enabled or disabled based on the edition of the product. Copy entitlement configuration file to <span style="background-color: #eeeeee;">LSF_ENTITLEMENT_FILE </span><span style="background-color: white;">path mentioned in step 3.</span></p><p>The following LSF entitlement configuration files are available for each edition:</p><p><b>LSF Standard Edition ===> lsf_std_entitlement.dat</b></p><p>LSF Express Edition ===> lsf_exp_entitlement.dat</p><p>LSF Advanced Edition ==> lsf_adv_entitlement.dat</p><p>-------------------------</p><p><b>Step 1</b> : Get the LSF installer script package that you selected and extract it.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="background-color: #f3f3f3;"><b># zcat lsf10.1_lsfinstall_linux_x86_64.tar.Z | tar xvf -</b></span></p></blockquote><p><b>Step 2 </b>: Go to extracted directory :</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #f3f3f3;"><b>cd lsf10.1_lsfinstall</b></span></p></blockquote><p><b>Step 3 </b>: Configure install.config as per the plan</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;"><span style="font-size: small;"><b> cat install.config</b></span><br /><span style="font-size: small;"> LSF_TOP="/nfs_shared_dir/LSF_HOME"</span><br /><span style="font-size: small;"> LSF_ADMINS="lsfadmin"</span><br /><span style="font-size: small;"> LSF_CLUSTER_NAME="x86-64_cluster2"</span><br /><span style="font-size: small;"> LSF_MASTER_LIST="myhost1"</span><br /><span style="font-size: small;"> LSF_TARDIR="/nfs_shared_dir/conf_lsf/lsf_distrib/"</span><br /><span style="font-size: small;"> LSF_ENTITLEMENT_FILE="/nfs_shared_dir/conf_lsf/lsf_std_entitlement.dat"</span><br /><span style="font-size: small;"> LSF_ADD_SERVERS="myhost1 myhost2 myhost3 myhost4 myhost5 myhost6 myhost7 myhost8"</span></span><br /><span style="color: #f3f3f3; font-size: small;"> ENABLE_DYNAMIC_HOSTS="Y"</span></p></blockquote><p><b>Step 4</b>: Start LSF 10.1 base installation </p><p> <b>./lsfinstall -f install.config</b></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee; font-size: x-small;">Logging installation sequence in /root/LSF_new/lsf10.1_lsfinstall/Install.log</span><br /><span style="background-color: #eeeeee; font-size: x-small;">International Program License Agreement</span><br /><span style="background-color: #eeeeee; font-size: x-small;">Part 1 - General TermsBY DOWNLOADING, INSTALLING, COPYING, ACCESSING, CLICKING </span><br /><span style="background-color: #eeeeee; font-size: x-small;"> "ACCEPT" BUTTON, OR OTHERWISE USING THE PROGRAM,</span><br /><span style="background-color: #eeeeee; font-size: x-small;">LICENSEE AGREES TO THE TERMS OF THIS AGREEMENT. IF YOU ARE</span><br /><span style="background-color: #eeeeee; font-size: x-small;">ACCEPTING THESE TERMS ON BEHALF OF LICENSEE, YOU REPRESENT</span><br /><span style="background-color: #eeeeee; font-size: x-small;">AND WARRANT THAT YOU HAVE FULL AUTHORITY TO BIND LICENSEE</span><br /><span style="background-color: #eeeeee; font-size: x-small;">TO THESE TERMS. IF YOU DO NOT AGREE TO THESE TERMS</span><br /><span style="background-color: #eeeeee; font-size: x-small;">* DO NOT DOWNLOAD, INSTALL, COPY, ACCESS, CLICK ON AN</span><br /><span style="background-color: #eeeeee; font-size: x-small;">"ACCEPT" BUTTON, OR USE THE PROGRAM; AND</span><br /><span style="background-color: #eeeeee; font-size: x-small;">* PROMPTLY RETURN THE UNUSED MEDIA, DOCUMENTATION, AND</span><br /><span style="background-color: #eeeeee; font-size: x-small;">Press Enter to continue viewing the license agreement, or</span><br /><span style="background-color: #eeeeee; font-size: x-small;">enter "1" to accept the agreement, "2" to decline it, "3"</span><br /><span style="background-color: #eeeeee; font-size: x-small;">to print it, "4" to read non-IBM terms, or "99" to go back</span><br /><span style="background-color: #eeeeee; font-size: x-small;">to the previous screen.</span><br /><span style="background-color: #eeeeee; font-size: x-small;">1</span><br /><span style="background-color: #eeeeee; font-size: x-small;">Checking the LSF TOP directory /<span style="font-size: medium;">nfs_shared_dir</span>/LSF_HOME ...</span><br /><span style="background-color: #eeeeee; font-size: x-small;">... Done checking the LSF TOP directory /<span style="font-size: medium;">nfs_shared_dir</span>i/LSF_HOME ...</span><br /><span style="background-color: #eeeeee; font-size: x-small;">You are installing IBM Spectrum LSF - 10.1 Standard Edition</span><br /><span style="background-color: #eeeeee; font-size: x-small;">Searching LSF 10.1 distribution tar files in /<span style="font-size: medium;">nfs_shared_dir</span>/conf_lsf/lsf_distrib Please wait ...</span><br /><span style="background-color: #eeeeee; font-size: x-small;"> 1) linux3.10-glibc2.17-x86_64</span><br /><span style="background-color: #eeeeee; font-size: x-small;">Press 1 or Enter to install this host type: 1</span><br /><span style="background-color: #eeeeee; font-size: x-small;">Installing linux3.10-glibc2.17-x86_64 ...</span><br /><span style="background-color: #eeeeee; font-size: x-small;">Please wait, extracting lsf10.1_lnx310-lib217-x86_64 may take up to a few minutes ...</span><br /><span style="background-color: #eeeeee; font-size: x-small;">lsfinstall is done.</span><br /><span style="background-color: #eeeeee; font-size: x-small;">After installation, remember to bring your cluster up to date </span><span style="background-color: #eeeeee; font-size: x-small;">by applying the latest updates and bug fixes.</span></p></blockquote><p>NOTE: You can do LSF installation as non-root user. That will be similar but with one extra prompt for multi-node cluster(yes/no)</p><div><div><b>Step 5 </b>: This step required only if installation was done by root .</div></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div><p style="text-align: left;"><b> </b><span style="background-color: #eeeeee;">chown -R lsfadmin:lsfadmin $LSF_TOP</span></p></div></blockquote><div><p><b>Step 6 </b>: check the binary files </p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div><p style="text-align: left;"><span style="background-color: #eeeeee;">cd LSF_TOP/10.1/linux3.10-glibc2.17-x86_64/bin</span></p></div></blockquote><div><p><b>Step 7 </b>: By default, only root can start the LSF daemons. Any user can not submit jobs to your cluster. To make the cluster available to other users, you must manually change the ownership and setuid bit for the lsadmin and badmin binary files to root, and the file permission mode to -rwsr-xr-x (4755) so that the user ID bit for the owner is setuid.</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #eeeeee;"><span> chown root lsadmin</span><br /><span> chown root badmin</span><br /><span> chmod 4755 lsadmin</span><br /><span> chmod 4755 badmin</span><br /><span> ls -alsrt lsadmin</span><br /><span> ls -alsrt badmin</span></span></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p><span><span face="Slack-Lato, appleLogo, sans-serif" style="background-color: #f8f8f8; color: #1d1c1d; font-variant-ligatures: common-ligatures;">chown root </span><span face="Slack-Lato, appleLogo, sans-serif" style="background-color: #f8f8f8; color: #1d1c1d; font-variant-ligatures: common-ligatures;">$LSF_SERVERDIR/eauth</span><span face="Slack-Lato, appleLogo, sans-serif" style="background-color: #f8f8f8; color: #1d1c1d; font-variant-ligatures: common-ligatures;"> </span></span></p><p><span><span face="Slack-Lato, appleLogo, sans-serif" style="background-color: #f8f8f8; color: #1d1c1d; font-variant-ligatures: common-ligatures;">chmod u+s $LSF_SERVERDIR/eauth</span> </span></p></blockquote><p>OR </p><p> <span style="background-color: black;"><span style="color: #eeeeee;"> <b>./hostsetup --top="LSF_HOME" --setuid</b></span> </span></p><div><p><b>Step 8 </b>: C<span face="-apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji"" style="background-color: white; color: #24292e; font-size: 14px;">onfigure /etc/lsf.sudoers </span></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p><span style="background-color: #eeeeee;">[root@myhost1]# cat /etc/lsf.sudoers<br />LSF_STARTUP_USERS="lsfadmin"<br />LSF_STARTUP_PATH="/nfs_shared_dir/</span><span style="background-color: #eeeeee;">LSF_HOME/10.1/linux3.10-glibc2.17-ppc64le/etc"<br />LSF_EAUTH_KEY="testKey1"</span></p></blockquote></blockquote><p style="text-align: justify;">NOTE: This lsf.sudoers file is not installed by default. This file is located in /etc. lsf.sudoers file is used to set the parameter LSF_EAUTH_KEY to configure a key for eauth to encrypt and decrypt user authentication data. All the nodes/hosts should have this file . Customers need to configure LSF_EAUTH_KEY in /etc/lsf.sudoers on each side of multi-cluster. </p></div><div><p><b>Step 9 </b>: check <span face="Slack-Lato, appleLogo, sans-serif" style="background-color: #f8f8f8; color: #1d1c1d; font-size: 15px; font-variant-ligatures: common-ligatures;">$LSF_SERVERDIR/eauth and copy </span>lsf.sudoers to all hosts in the cluster</p><p> ls $LSFTOP/10.1/linux3.10-glibc2.17-x86_64/etc/</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;"><br /><span style="font-size: small;">scp /etc/lsf.sudoers myhost</span><span style="font-size: small;">02:/etc/lsf.sudoers</span><br /><span style="font-size: small;">scp /etc/lsf.sudoers myhost</span><span style="font-size: small;">03:/etc/lsf.sudoers</span><br /><span style="font-size: small;">scp /etc/lsf.sudoers myhost</span><span style="font-size: small;">04:/etc/lsf.sudoers</span><br /><span style="font-size: small;">scp /etc/lsf.sudoers myhost</span><span style="font-size: small;">05:/etc/lsf.sudoers</span><br /><span style="font-size: small;">scp /etc/lsf.sudoers myhost</span><span style="font-size: small;">06:/etc/lsf.sudoers</span><br /><span style="font-size: small;">scp /etc/lsf.sudoers myhost</span><span style="font-size: small;">07:/etc/lsf.sudoers</span><br /><span style="font-size: small;">scp /etc/lsf.sudoers myhost</span><span style="font-size: small;">08:/etc/lsf.sudoers</span></span></p></blockquote><div><p><b>Step 10 </b>: Start LSF as lsfadmin and check base Installation using lsid command.</p><p><b>Step 11 </b>: Check binary type with lsid -V</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><b><span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-size: 12px; font-variant-ligatures: none; white-space: pre-wrap;">$ lsid -V<br />IBM Spectrum LSF 10.1.0.0 build 403338, May 27 2016<br />Copyright International Business Machines Corp. 1992, 2016.<br />US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.</span><span style="font-size: 12px; font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>binary type: linux3.10-glibc2.17-x86_64<br /></b></pre></blockquote><p>NOTE: Download required FP and interim fixes from https://www.ibm.com/support/fixcentral/ </p><div><p><b>Step 12 : </b>Before applying PTF12 and interim patches , bring down the LSF daemons. Use the following commands to shut down the original LSF daemons</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;"> badmin hshutdown all</span><br /><span style="background-color: #eeeeee;"> lsadmin resshutdown all</span><br /><span style="background-color: #eeeeee;"> lsadmin limshutdown all</span></p></blockquote></blockquote><p>Deactivate all queues to make sure that no new jobs can be dispatched during the upgrade:</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="background-color: #eeeeee;">badmin qinact all</span> </p></blockquote></blockquote><div><p><b>Step 13</b>: Then, become the root to apply FP12 and interim patches . </p><p>Set LSF environment : . LSF_TOP/conf/profile.lsf</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div><p style="text-align: left;"><span style="background-color: #eeeeee;">. /nfs_shared_dir/LSF_HOME/conf/profile.lsf</span></p></div></blockquote><div><p><b>Step 14</b>: Apply FP 12 on LSF BASE installation. The patchinstall is available in $LSF_TOP//install directory</p><p> <span style="background-color: #eeeeee;"># cd $LSF_TOP/10.1/install</span></p><p>Perform a check on patches running. It is recommended to check for the patch before its installation</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div><p style="text-align: left;"><span style="background-color: #eeeeee;">$ patchinstall –c</span></p></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div><p style="text-align: left;"><span style="background-color: #eeeeee;"> ./patchinstall /root/PTF12_x86_2versions/lsf10.1_lnx310-lib217-x86_64-600488.tar.Z</span></p></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div style="text-align: left;"><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><span style="font-size: xx-small;">[root@myhost7 install]# ./patchinstall /root/PTF12_x86_2versions/lsf10.1_lnx310-lib217-x86_64-600488.tar.Z<br />Logging patch installation sequence in /nfs_shared_dir/LSF_HOME/10.1/install/patch.log<br />Checking the LSF installation directory /nfs_shared_dir/LSF_HOME ...<br />Done checking the LSF installation directory /nfs_shared_dir/LSF_HOME.<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Checking the patch history directory ...<br />Done checking the patch history directory /nfs_shared_dir/LSF_HOME/patch.<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Checking the backup directory ...<br />Done checking the backup directory /nfs_shared_dir/LSF_HOME/patch/backup.<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Installing package "/root/PTF12_x86_2versions/lsf10.1_lnx310-lib217-x86_64-600488.tar.Z"...<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Checking the package definition for /root/PTF12_x86_2versions/lsf10.1_lnx310-lib217-x86_64-600488.tar.Z ...<br />Done checking the package definition for /root/PTF12_x86_2versions/lsf10.1_lnx310-lib217-x86_64-600488.tar.Z.<br />.<br />.<br />Finished backing up files to "/nfs_shared_dir/LSF_HOME/patch/backup/LSF_linux3.10-glibc2.17-x86_64_600488".<br />Done installing /root/PTF12_x86_2versions/lsf10.1_lnx310-lib217-x86_64-600488.tar.Z.</span></pre></div></blockquote><div><p><b>Step 15</b>: Apply interim fix1</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div><p style="text-align: left;"><span style="background-color: #eeeeee;">./patchinstall /root/LSF_patch1/lsf10.1_lnx310-lib217-x86_64-600505.tar.Z</span></p></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><span style="font-size: xx-small;">Logging patch installation sequence in /nfs_shared_dir/LSF_HOME/10.1/install/patch.log <br />Installing package "/root/LSF_patch1/lsf10.1_lnx310-lib217-x86_64-600505.tar.Z"...<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Checking the package definition for /root/LSF_patch1/lsf10.1_lnx310-lib217-x86_64-600505.tar.Z ...<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Are you sure you want to update your cluster with this patch? (y/n) [y] y<br />Y<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Backing up existing files ...<br />Finished backing up files to "/nfs_shared_dir/LSF_HOME/patch/backup/LSF_linux3.10-glibc2.17-x86_64_600505".<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Done installing /root/LSF_patch1/lsf10.1_lnx310-lib217-x86_64-600505.tar.Z.<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Exiting..</span>.</pre></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote><b>Step 16</b>: Apply interim fix2<div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div><p style="text-align: left;"><span style="background-color: #eeeeee;"> ./patchinstall /root/LSF_patch2/lsf10.1_lnx310-lib217-x86_64-600625.tar.Z</span></p></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><span style="font-size: xx-small;">[root@myhost7 install]# ./patchinstall /root/LSF_patch2/lsf10.1_lnx310-lib217-x86_64-600625.tar.Z<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Installing package "/root/LSF_patch2/lsf10.1_lnx310-lib217-x86_64-600625.tar.Z"...<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Checking the package definition for /root/LSF_patch2/lsf10.1_lnx310-lib217-x86_64-600625.tar.Z ...<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Backing up existing files ...<br />Finished backing up files to "/nfs_shared_dir/LSF_HOME/patch/backup/LSF_linux3.10-glibc2.17-x86_64_600625".<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Done installing /root/LSF_patch2/lsf10.1_lnx310-lib217-x86_64-600625.tar.Z.<span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-variant-ligatures: none; white-space: pre-wrap;"><br /></span></span>Exiting...</span> </pre></blockquote><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><b>Step 17</b>: As a root user , Setbit for new command bctrld</pre><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="background-color: #eeeeee;"> cd LSF_TOP/10.1/linux3.10-glibc2.17-x86_64/bin<br /> chown root bctrld<br /> chmod 4755 bctrld</span> </p></blockquote></blockquote><div><p><b>Step 18 </b>: Check lsf.shared file for multi cluster setup.</p></div></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #eeeeee;">Begin Cluster</span></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #eeeeee;">ClusterName Servers</span></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #eeeeee;">CLUSTER1 (cloudhost)</span></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #eeeeee;">CLUSTER2 (myhost1)</span></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #eeeeee;">CLUSTER3 (remotehost2)</span></blockquote><p style="text-align: left;"> <span style="background-color: #eeeeee;">End Cluster</span></p></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"></blockquote></blockquote><div><div><p><b>Step 19 </b>: Switch back to user lsfadmin. Use the following commands to start LSF using the newer daemons.</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;"> <span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-variant-ligatures: none; white-space: pre-wrap;">lsadmin limstartup all</span><br /><span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-variant-ligatures: none; white-space: pre-wrap;"> lsadmin resstartup all</span><br /><span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-variant-ligatures: none; white-space: pre-wrap;"> badmin hstartup all</span></span></p></blockquote></blockquote><div><p>Use the following command to reactivate all LSF queues after upgrading: badmin qact all</p><p><b>Step 20</b> : Modify Conf files as per requirement add queues, clusters...etc . Then run badmin reconfig or lsadmin reconfig as explained in LSF configuration section below. Restart LSF as "lsfadmin" user .</p><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; word-break: normal;"><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; color: #1d1c1d; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-size: 12px; font-variant-ligatures: none; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; white-space: pre-wrap; word-break: normal;"><b>$ lsid
IBM Spectrum LSF Standard 10.1.0.12, Jun 10 2021
Copyright International Business Machines Corp. 1992, 2016.
US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
My cluster name is CLUSTER2
My master name is myhost1</b>
</pre></pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; word-break: normal;"><span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-size: 12px; font-variant-ligatures: none; white-space: pre-wrap;"><b>$ lsclusters -w
CLUSTER_NAME STATUS MASTER_HOST ADMIN HOSTS SERVERS
CLUSTER1 ok cloudhost lsfadmin 7 7
CLUSTER2 ok myhost1 lsfadmin 8 8
CLUSTER3 ok remotehost2 lsfadmin 8 8</b>
</span></span></pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><span style="color: #1d1c1d;"><span><b>$ bhosts
HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV<br />myhost1 ok - 20 0 0 0 0 0
myhost2 ok - 20 0 0 0 0 0
myhost3 ok - 19 0 0 0 0 0<br /><span style="font-family: Monaco, Menlo, Consolas, Courier New, monospace;"></span>myhost4 ok - 44 4 4 0 0 0<br />myhost5 ok - 44 4 4 0 0 0<br />myhost6 ok - 20 0 0 0 0 0
myhost7 ok - 20 0 0 0 0 0
myhost8 ok - 19 0 0 0 0 0</b></span></span></pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><span style="color: #1d1c1d;"><span>Spectrum LSF Cluster Installation and FP12 upgradation completed successfully as per the details copied above.</span></span></pre><p>You must run hostsetup as root to use --boot="y" option to modify the system scripts to automatically start and stop LSF daemons at system startup or shutdown. . The default is --boot="n".</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><p><span style="background-color: #eeeeee;">1. Log on to each LSF server host as root. Start with the LSF master host.</span></p></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><p><span style="background-color: #eeeeee;">2. Run hostsetup on each LSF server host. For example:</span></p></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><p><span style="background-color: #eeeeee;"># cd $LSF_TOP/10.1/install</span></p></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><p><span style="background-color: #eeeeee;"># ./hostsetup --top="$LSF_TOP" --boot="y"</span></p></div></blockquote></blockquote><div><p>NOTE: For more details on hostsetup usage, enter hostsetup -h.</p><p>In case of multi-cluster environment, reinstalling master cluster would show status=disk after issuing bclusters command. </p><p><br /></p></div></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #eeeeee; font-size: x-small;">[smpici@c656f7n06 ~]$ bclusters<br />[Job Forwarding Information ]<br /><br />LOCAL_QUEUE JOB_FLOW REMOTE CLUSTER STATUS<br /> Queue1 send CLUSTER1 disc<br /> Queue2 send CLUSTER2 disc<br /> Queue3 send CLUSTER3 disc</span></blockquote></blockquote></blockquote><div><p>where status=disc means communication between the two clusters is not established. The disc status might occur because no jobs are waiting to be dispatched, or because the remote master cannot be located.</p><p>Possible solution is to cleanup all the LSF daemons on all clusters. Note : lsfshutdown leaves some of the daemons on Master node. So , you need to manually kill all the LSF daemons on all master nodes.</p><p>Later, bclusters should show the status as shown below:</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #eeeeee; font-size: x-small;">[smpici@c656f7n06 ~]$ bclusters<br />[Job Forwarding Information ]<br /><br />LOCAL_QUEUE JOB_FLOW REMOTE CLUSTER STATUS<br />Queue1 send CLUSTER1 ok<br />Queue2 send CLUSTER2 ok<br />Queue3 send CLUSTER3 ok</span></blockquote></blockquote></blockquote><div><p> </p><p style="text-align: justify;">Singularity is a containerization solution designed [os-level-virtualization] for high-performance computing cluster environments. It allows a user on an HPC resource to run an application using a different operating system than the one provided by the cluster. For example, the application may require Ubuntu but the cluster OS is CentOS. If you are using LSF_EAUTH_KEY in container based environment, then there may be eauth setbit issue. LSF client will invoke "eauth -c" in the container by the job owner; eauth is setuid program, so it can read lsf.sudoers file to get the key. If eauth loses the setuid permission, it cannot read the lsf.sudoers file, and it will use the default key to encrypt user information. When request reaches to a server, it calls "eauth -s" which is running on root at host. It gets the key, and use the key to decrypt the user information, and failed. In other words, only default key can work for singularity environment. </p><p style="text-align: justify;">That can be resolved by disabling LSF_AUTH_QUERY_COMMANDS in configuration file as shown below. Since LSF12, LSF introduced LSF_AUTH_QUERY_COMMANDS in lsf.conf. The default value is set to Y. Basically it adds extra user authentication for batch query. By default, each cluster has own default eauth key. If user uses "boosts remote_cluster", it uses local key to encrypt user data, but talk to remote daemon directly. As the remote daemon uses its own key to decrypt data, it fails. Job operations information is exchanged through mbatchd to mbatchd communication channel. It does not go through kind of auth. Logically, user should only query local mbatchd to see job's status (the remote job status will be sent back to the submission cluster). </p><div><div>Modified files :</div><div><br /></div><div>1) Add LSF_AUTH_QUERY_COMMANDS=N to `lsf.conf file` </div><div>2) Removed : LSF_EAUTH_KEY="testKey1" from `/etc/lsf.sudoers`</div><div><br /></div><div>========================================================================</div><div><br /></div><div>Step 1 : Check cluster names </div></div></div><div><br /></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><div><div style="text-align: left;"><span style="background-color: #eeeeee;">[sachinpb@cluster1_master sachinpb]$ lsclusters -w</span></div></div></div><div><div><div style="text-align: left;"><span style="background-color: #eeeeee;">CLUSTER_NAME STATUS MASTER_HOST ADMIN HOSTS SERVERS</span></div></div></div><div><div><div style="text-align: left;"><span style="background-color: #eeeeee;">cluster1 ok cluster1_master sachinpb 5 5</span></div></div></div><div><div><div style="text-align: left;"><span style="background-color: #eeeeee;">cluster2 ok cluster2_master sachinpb 8 8</span></div></div></div><div><div><div style="text-align: left;"><span style="background-color: #eeeeee;">[sachinpb@cluster1_master sachinpb]$ </span></div></div></div></blockquote><div><div><div><br /></div><div>Step 2: Submit job to remote cluster (Job forwarding queue=Forwarding_queue) </div><div><br /></div></div></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><div><div><span style="background-color: #cccccc;">[sachinpb@cluster1_master sachinpb]$ bsub -n 10 -R "span[ptile=2]" -q Forwarding_queue -R - sleep 1000</span></div></div></div><div><div><div><span style="background-color: #cccccc;">Job <35298> is submitted to queue <Forwarding_queue>.</span></div></div></div><div><div><div><span style="background-color: #cccccc;">[sachinpb@cluster1_master sachinpb]$</span></div></div></div></blockquote><div><div><div><br /></div><div>Step 3 : Check status of job from submission cluster:</div><div><br /></div></div></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><div><div><span style="background-color: #cccccc;">[sachinpb@cluster1_master sachinpb]$ bjobs</span></div></div></div><div><div><div><span style="background-color: #cccccc;">JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME</span></div></div></div><div><div><div><span style="background-color: #cccccc;">35298 sachinpb RUN x86_c656f8 cluster1_master cluster2_master sleep 1000 Oct 11 03:30</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;"> cluster2_master@cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;">[sachinpb@cluster1_master sachinpb]$ bjobs -o 'forward_cluster' 35298</span></div></div></div><div><div><div><span style="background-color: #cccccc;">FORWARD_CLUSTER</span></div></div></div><div><div><div><span style="background-color: #cccccc;">cluster2</span></div></div></div><div><div><div><span style="background-color: #cccccc;">[sachinpb@cluster1_master sachinpb]$ bjobs -o 'dstjobid' 35298</span></div></div></div><div><div><div><span style="background-color: #cccccc;">DSTJOBID</span></div></div></div><div><div><div><span style="background-color: #cccccc;">8589</span></div></div></div><div><div><div><span style="background-color: #cccccc;">[sachinpb@cluster1_master sachinpb]$</span></div></div></div></blockquote><div><div><div><br /></div><div>Step 4 : Get list of compute nodes on remote cluster by issuing bjobs -m command from submission cluster as shown below:</div></div></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div><div><div><span style="background-color: #cccccc;">[sachinpb@cluster1_master sachinpb]$ bjobs -m cluster2 -o 'EXEC_HOST' 8589</span></div></div></div><div><div><div><span style="background-color: #cccccc;">EXEC_HOST</span></div></div></div><div><div><div><span style="background-color: #cccccc;">computeNode04:computeNode04:computeNode06:computeNode06:computeNode05:computeNode05:computeNode03:computeNode03:computeNode07:computeNode07</span></div></div></div><div><div><div><span style="background-color: #cccccc;">[sachinpb@cluster1_master sachinpb]$</span></div></div></div></blockquote><div><p>======================= <b>LSF configuration section</b> ===========================</p><p>After you change any configuration file, use the lsadmin reconfig and badmin reconfig commands to reconfigure your cluster. Log on to the host as root or the LSF administrator (in our case "lsfadmin")</p><p>Run lsadmin reconfig to restart LIM and checks for configuration errors. If no errors are found, you are prompted to either restart the lim daemon on management host candidates only, or to confirm that you want to restart the lim daemon on all hosts. If unrecoverable errors are found, reconfiguration is canceled. Run the badmin reconfig command to reconfigure the mbatchd daemon and checks for configuration errors.</p><p></p><ul style="text-align: left;"><li>lsadmin reconfig to reconfigure the lim daemon</li><li>badmin reconfig to reconfigure the mbatchd daemon without restarting</li><li>badmin mbdrestart to restart the mbatchd daemon</li><li>bctrld restart sbd to restart the sbatchd daemon</li></ul><p></p><p>More details about cluster reconfiguration commands as shown in the table copied below :</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-glsfqkh9IO8/YRXoDAjiynI/AAAAAAAAV6k/26maEWM4LhUtod3l-vwmoZXnDOQw2bn_gCLcBGAsYHQ/s817/lsf_config.PNG" style="margin-left: auto; margin-right: auto;"><img alt="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=cluster-commands-reconfigure-your" border="0" data-original-height="616" data-original-width="817" height="482" src="https://1.bp.blogspot.com/-glsfqkh9IO8/YRXoDAjiynI/AAAAAAAAV6k/26maEWM4LhUtod3l-vwmoZXnDOQw2bn_gCLcBGAsYHQ/w640-h482/lsf_config.PNG" title="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=cluster-commands-reconfigure-your" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><a href="https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=cluster-commands-reconfigure-your" target="_blank"><span style="font-size: x-small;">Source</span></a></td></tr></tbody></table><div><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><span style="font-size: xx-small;"><br /></span></pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;">How to resolve some known eauth related issues - commands like bhosts, bjobs ...etc fail with error "User permission denied".
<b>Problem/solution 1: </b>
Example 1:
[smpici@host1 ~]$ bhosts
User permission denied
Example 2:
mpirun --timeout 30 hello_world
Jan 11 02:42:52 2022 1221079 3 10.1 lsb_pjob_send_requests: lsb_pjob_getAckReturn failed on host <host1>, lsberrno <0>
[host1:1221079] [[64821,0],0] ORTE_ERROR_LOG: The specified application failed to start in file ../../../../../../opensrc/ompi/orte/mca/plm/lsf/plm_lsf_module.c at line 347
--------------------------------------------------------------------------
The LSF process starter (lsb_launch) failed to start the daemons on
the nodes in the allocation.
Returned : -1
lsberrno : (282) Failed while executing tasks
This may mean that one or more of the nodes in the LSF allocation is
not setup properly.
Then, Please check clocks on the node . If clocks show the difference,
then you need configure chrony as shown below on all nodes.
systemctl enable chronyd.service
systemctl stop chronyd.service
systemctl start chronyd.service
systemctl status chronyd.service
The root cause of the problem was that the system clock between the nodes and the launch nodes were out of sync.
After the clocks were sync'ed up, I tried LSF cmds and it worked . In worst-case, If hosts have time un-synchronization,
please configure LSF_EAUTH_TIMEOUT=0 in lsf.conf on each side of multi-cluster</pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><b>Problem/solution 2:</b>
When multi-cluster job forwarding cluster shows unavailable state.
Just check the PORT numbers defined in lsf.conf are same on both the clusters. Otherwise , cluster shows unavailable state.
[lsfadmin@ conf]$ lsclusters -w
CLUSTER_NAME STATUS MASTER_HOST ADMIN HOSTS SERVERS
cluster1 ok c1_master lsfadmin 40 40
cluster2 unavail unknown unknown -
Default port numbers :
[root@c1_master ~]# cat $LSF_HOME/conf/lsf.conf | grep PORT
LSF_LIM_PORT=7869
LSF_RES_PORT=6878
LSB_MBD_PORT=6881
LSB_SBD_PORT=6882
LSB_QUERY_PORT=6891
Useful Tips: How long jobs are available after done or exit
CLEAN_PERIOD_DONE=seconds
Controls the amount of time during which successfully finished jobs are kept in mbatchd core memory.
This applies to DONE and PDONE (post job execution processing) jobs.
If CLEAN_PERIOD_DONE is not defined, the clean period for DONE jobs is defined by CLEAN_PERIOD in lsb.params.
If CLEAN_PERIOD_DONE is defined, its value must be less than CLEAN_PERIOD, otherwise it will be ignored and a warning message will appear.
# bparams -a | grep CLEAN_PERIOD
CLEAN_PERIOD = 3600
CLEAN_PERIOD_DONE = not configured
<div style="text-align: justify;"><b>Problem/solution 3:</b></div><div style="text-align: justify;"></div><div style="text-align: justify;">Regarding job forwarding setup between clusters over public/private IP address. </div><div style="text-align: justify;">I can't ssh from cluster1 Master (f2n01-10.x.x.x) to cluster2 with public IP on cluster2 Master(9.x.x.x).</div><div style="text-align: justify;">Do we need to open these ports by setting the firewall rules ?. From an LSF point of view the ports for </div><div style="text-align: justify;">the lim and mbd need to be opened up. Issue commands lsclusters and bclusters commands. </div><div style="text-align: justify;">The status reported should be OK in both. You should try this from both clusters. </div></pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><b>Problem/solution 4: </b></pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;">Test working of blaunch with LSF job submission. The blaunch command works only under LSF. </pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;">It can be used only to launch tasks on remote hosts that are part of a job allocation. </pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;">It cannot be used as a stand-alone command. The call to blaunch is made under the bsub environment.</pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><div style="text-align: justify;">You cannot run blaunch directly from the command line. </div><div style="text-align: justify;">blaunch is not to be used outside of the job execution environment provided by bsub.</div><div style="text-align: justify;">Most MPI implementations and many distributed applications use the rsh and ssh commands as their task launching</div><div style="text-align: justify;">mechanism. The blaunch command provides a drop-in replacement for the rsh and ssh commands as a transparent </div><div style="text-align: justify;">method for launching parallel applications within LSF.The following are some examples of blaunch usage:</div></pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;">Submit a parallel job:
bsub -n 4 blaunch myjob
Submit a job to an application profile
bsub -n 4 -app pjob blaunch myjob
-----------------Example -----------------------
[sachinpb@xyz]$ cat show_ulimits.c
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>
int main()
{
struct rlimit old_lim;
if(getrlimit(RLIMIT_CORE, &old_lim) == 0)
printf("current limits -> soft limit= %ld \t"
" hard limit= %ld \n", old_lim.rlim_cur, old_lim.rlim_max);
else
fprintf(stderr, "%s\n", strerror(errno));
// abort();
return 0;
}
[sachinpb@xyz]$
-------------------------------------------------
[submission_node]$ gcc -o show_ulimits show_ulimits.c
[submission_node]$ ls
generate_core.c show_ulimits show_ulimits.c test_limit
[submission_node]$ ./show_ulimits
current limits -> soft limit= 0 hard limit= 0
[submission_node]$
--------------------------------------------------------
[submission_node]$ bsub -o /shared-dir/sachin_test_lsf_out_%J -n 4 -R "span[ptile=2]" -q x86_test_q -m "HOSTA HOSTB" blaunch /shared_dir/core_file_test/show_limits
Job <2511> is submitted to queue <x86_test_q>.
[submission_node]$ bjobs 2511
JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
2511 sachinpb DONE x86_test_q bsub_host HOSTA show_limits Oct 29 02:17
HOSTA
HOSTB
HOSTB
[submission_node]$
[submission_node]$ cat /nfs_smpi_ci/sachin_test_lsf_out_2511
Sender: LSF System <sachinpb@HOSTA>
Subject: Job 2511: <blaunch /nfs_smpi_ci/core_file_test/test_limit> in cluster <x86-64_pok-cluster2> Done
Job <blaunch /nfs_smpi_ci/core_file_test/test_limit> was submitted from host <bsub_host> by user <sachinpb> in cluster <x86-64_pok-cluster2> at Thu Oct 29 02:17:45 2020
Job was executed on host(s) <2*HOSTA>, in queue <x86_test_q>, as user <sachinpb> in cluster <x86-64_pok-cluster2> at Thu Oct 29 02:23:52 2020
<2*HOSTB>
Your job looked like:
------------------------------------------------------------
# LSBATCH: User input
blaunch /nfs_smpi_ci/core_file_test/test_limit
------------------------------------------------------------
Successfully completed.
The output (if any) follows:
current limits -> soft limit= -1 hard limit= -1
current limits -> soft limit= -1 hard limit= -1
current limits -> soft limit= -1 hard limit= -1
current limits -> soft limit= -1 hard limit= -1
[submission_node]$
-----------------------------------------------------------------------------------
<b>Problem/solution 5: </b>
When job submission fail with "User permission denied" .
Please check setbits with details copied below
[smpici@c690f2n01 big-mpi]$ bsub
bsub> sleep 100
bsub> User permission denied. Job not submitted.
$LSF_HOME/10.1/linux3.10-glibc2.17-ppc64le/bin
chown root lsadmin
chown root badmin
chmod 4755 lsadmin
chmod 4755 badmin
chown root bctrld
chmod 4755 bctrld
$LSF_HOME/10.1/linux3.10-glibc2.17-ppc64le/etc
chown root eauth
chmod u+s eauth
Just run this script to avoid manual configurations.
./hostsetup --top="LSF_HOME" --setuid </pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;">--------------------------------------------------</pre><pre class="c-mrkdwn__pre" data-stringify-type="pre" style="--saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); border-radius: 4px; border: 1px solid var(--saf-0); box-sizing: inherit; counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; line-height: 1.50001; margin-bottom: 4px; margin-top: 4px; overflow-wrap: break-word; padding: 8px; tab-size: 4; text-align: left; word-break: normal;"><span style="font-size: xx-small;">References:
https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=migrate-install-unix-linux<br />https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=iul-if-you-install-lsf-as-non-root-user</span><span style="color: #1d1c1d; font-family: Monaco, Menlo, Consolas, Courier New, monospace;"><span style="font-size: 12px; font-variant-ligatures: none; white-space: pre-wrap;"></span></span></pre></div></div>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-46455855978946433582021-07-23T10:08:00.018+05:302021-11-25T10:51:17.992+05:30Spectrum scale :High-performance storage GPFS cluster Installation and setup <p style="text-align: justify;">IBM Spectrum Scale(formerly GPFS) is a scale-out high performance global parallel file system (cluster file system) that provides concurrent access to a single file system or set of file systems from multiple nodes. Enterprises and organizations are creating, analyzing and keeping more data than ever before. Islands of data are being created all over the organization and in the cloud creating complexity, difficult to manage systems and increasing costs. Those that can deliver insights faster while managing rapid infrastructure growth are the leaders in their industry. In delivering those insights, an organization’s underlying information architecture must support the hybrid cloud, big data and artificial intelligence (AI) workloads along with traditional applications while ensuring security, reliability, data efficiency and high performance. IBM Spectrum Scale™ meets these challenges as a parallel high-performance solution with global file and object data access for managing data at scale with the distinctive ability to perform archive and analytics in place.</p><p style="text-align: left;">Manually installing the IBM Spectrum Scale software packages on POWER nodes myhost1, myhost2 and myhost3</p><p style="text-align: left;">The following packages are required for IBM Spectrum Scale Standard Edition on Red Hat Enterprise Linux:</p><p style="text-align: left;"></p><ol style="text-align: left;"><li><span style="background-color: #eeeeee;">gpfs.base*.rpm</span></li><li><span style="background-color: #eeeeee;">gpfs.gpl*.noarch.rpm</span></li><li><span style="background-color: #eeeeee;">gpfs.msg.en_US*.noarch.rpm</span></li><li><span style="background-color: #eeeeee;">gpfs.gskit*.rpm</span></li><li><span style="background-color: #eeeeee;">gpfs.license*.rpm</span></li></ol><p style="text-align: left;"></p><p style="text-align: left;">Step 1:Download spectrum scale 5.1.1.1 SE package from fix central and Install RPM packages on all nodes:</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="background-color: #cccccc;"> rpm -ivh gpfs.base*.rpm gpfs.gpl*rpm gpfs.license.std*.rpm gpfs.gskit*rpm gpfs.msg*rpm gpfs.docs*rpm</span></p></blockquote></blockquote></blockquote><p style="text-align: left;"><br /></p><p style="text-align: left;">Step 2 : Verify installed GPFS packages</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p></p></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;"> [root@myhost1 ]# rpm -qa | grep gpfs</span><br /><span style="background-color: #cccccc;"> gpfs.docs-5.1.1-1.noarch</span><br /><span style="background-color: #cccccc;"> gpfs.license.std-5.1.1-1.ppc64le</span><br /><span style="background-color: #cccccc;"> gpfs.bda-integration-1.0.3-1.noarch</span><br /><span style="background-color: #cccccc;"> gpfs.base-5.1.1-1.ppc64le</span><br /><span style="background-color: #cccccc;"> gpfs.gplbin-4.18.0-305.el8.ppc64le-5.1.1-1.ppc64le</span><br /><span style="background-color: #cccccc;"> gpfs.gskit-8.0.55-19.ppc64le</span><br /><span style="background-color: #cccccc;"> gpfs.msg.en_US-5.1.1-1.noarch</span><br /><span style="background-color: #cccccc;"> gpfs.gpl-5.1.1-1.noarch</span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p></p></blockquote></blockquote><p style="text-align: left;">Step 3 : Build GPL (5.1.1.1) module by issuing command mmbuildgpl on all nodes in cluster.</p><p style="text-align: left;">Step 4 : Verify GPFS packages installed on all nodes with GPL module built properly.</p><p style="text-align: left;"> Export the path for GPFS commands. </p><p style="text-align: left;"> export PATH=$PATH:/usr/lpp/mmfs/bin</p><p style="text-align: left;">Step 5 : Use the mmcrcluster command to create a GPFS cluster</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="background-color: #cccccc;">mmcrcluster -N NodeFile -C smpi_gpfs_power8</span></p></blockquote></blockquote><p style="text-align: left;"> where NodeFile has following entries</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;">#cat NodeFile</span><br /><span style="background-color: #cccccc;"> myhost2:quorum</span><br /><span style="background-color: #cccccc;"> myhost1:quorum-manager</span><br /><span style="background-color: #cccccc;"> myhost3:quorum-manager</span></p></blockquote></blockquote></blockquote></blockquote><p style="text-align: left;">Step 6: Use the mmchlicense command to designate licenses as needed. This command controls the type of GPFS license associated with the nodes in the cluster. -- accept indicates that you accept the applicable licensing terms. </p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #cccccc;">mmchlicense server --accept -N serverLicense</span></p></blockquote></blockquote><p style="text-align: left;">Step 7: mmgetstate command. Displays the state of the GPFS™ daemon on one or more nodes.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #cccccc;">mmgetstate -a</span></p></blockquote></blockquote><p style="text-align: left;">Step 8: mmlslicense command displays information about the IBM Spectrum Scale node licensing designation or about disk and cluster capacity.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><span style="background-color: #cccccc;"> mmlslicense -L</span></p></blockquote></blockquote><p style="text-align: left;">Step 9: The mmcrnsd command is used to create cluster-wide names for NSDs used by GPFS. This is the first GPFS step in preparing disks for use by a GPFS file system.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #cccccc;">mmcrnsd -F NSD_Stanza_smpi_gpfs_power -v no</span></p></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> where NSD_Stanza_smpi_gpfs_power has</p></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"></p></blockquote></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="background-color: #cccccc;">#cat NSD_Stanza_smpi_gpfs_power</span><br /><span style="background-color: #cccccc;">%nsd:</span><br /><span style="background-color: #cccccc;"> device=/dev/sda</span><br /><span style="background-color: #cccccc;"> nsd=nsd1</span><br /><span style="background-color: #cccccc;"> servers=myhost2</span><br /><span style="background-color: #cccccc;"> usage=dataAndMetadata</span><br /><span style="background-color: #cccccc;"> failureGroup=-1</span><br /><span style="background-color: #cccccc;"> pool=system</span><br /><br /><span style="background-color: #cccccc;">%nsd:</span><br /><span style="background-color: #cccccc;"> device=/dev/sdb</span><br /><span style="background-color: #cccccc;"> nsd=nsd2</span><br /><span style="background-color: #cccccc;"> servers=myhost1</span><br /><span style="background-color: #cccccc;"> usage=dataAndMetadata</span><br /><span style="background-color: #cccccc;"> failureGroup=-1</span><br /><span style="background-color: #cccccc;"> pool=system</span><br /><br /><span style="background-color: #cccccc;">%nsd:</span><br /><span style="background-color: #cccccc;"> device=/dev/sda</span><br /><span style="background-color: #cccccc;"> nsd=nsd3</span><br /><span style="background-color: #cccccc;"> servers=myhost3</span><br /><span style="background-color: #cccccc;"> usage=dataAndMetadata</span><br /><span style="background-color: #cccccc;"> failureGroup=-1</span><br /><span style="background-color: #cccccc;"> pool=system</span></blockquote></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p></p></blockquote></blockquote></blockquote></blockquote><p style="text-align: left;">Step 10: Use the mmlsnsd command to display the current information for the NSDs belonging to the GPFS cluster.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #cccccc;">mmlsnsd -X</span></p></blockquote></blockquote><p style="text-align: left;">Step 11: Use the mmcrfs command to create a GPFS file system</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #cccccc;">mmcrfs smpi_gpfs -F NSD_Stanza_smpi_gpfs_power</span></p></blockquote></blockquote><p style="text-align: left;">Step 12: The mmmount command mounts the specified GPFS file system on one or more nodes in the cluster.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #cccccc;">mmmount smpi_gpfs -a</span></p></blockquote></blockquote><p style="text-align: left;">Step 13 : Use the mmlsfs command to list the attributes of a file system.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #eeeeee;">mmlsfs all</span></p></blockquote></blockquote><p style="text-align: left;">Step 14: The mmlsmount command reports if a file system is in use at the time the command is issued.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #eeeeee;">mmlsmount all</span></p></blockquote></blockquote><p style="text-align: left;">step 15: How to change the mount point from /gpfs to /my_gpfs </p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"> <span style="background-color: #eeeeee;">mmchfs gpfs -T /my_gpfs</span></p></blockquote></blockquote><p>Step 16: GPFS auto start and auto mount setup</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;">[root@myhost1 ~]# systemctl status gpfs.service<br />● gpfs.service - General Parallel File System<br /> Loaded: loaded (/usr/lib/systemd/system/gpfs.service; disabled; vendor preset: disabled)<br /> Active: active (running) since Tue 2021-07-20 03:27:04 EDT; 3 days ago<br /> Process: 96622 ExecStart=/usr/lpp/mmfs/bin/mmremote startSubsys systemd $STARTSUBSYS_ARGS (code=exited, status=0/SUCCESS)<br /> Main PID: 96656 (runmmfs)<br /> CGroup: /system.slice/gpfs.service<br /> ├─96656 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs<br /> └─97093 /usr/lpp/mmfs/bin/mmfsd<br /></span></p></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;">[root@myhost1 ~]# systemctl is-active gpfs.service<br />active<br />[root@myhost1 ~]# systemctl is-enabled gpfs.service<br />disabled<br />[root@myhost1 ~]# systemctl is-failed gpfs.service<br />active<br />[root@myhost1 ~]# systemctl enable gpfs.service<br />Created symlink from /etc/systemd/system/multi-user.target.wants/gpfs.service to /usr/lib/systemd/system/gpfs.service.<br />[root@myhost1 ~]# systemctl is-enabled gpfs.service<br />enabled<br />[root@myhost1 ~]# ls -alsrt /etc/systemd/system/multi-user.target.wants/gpfs.service<br />0 lrwxrwxrwx 1 root root 36 Jul 23 05:43 /etc/systemd/system/multi-user.target.wants/gpfs.service -> /usr/lib/systemd/system/gpfs.service</span> </p></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;">[root@myhost1 ~]# mmgetstate -a<br /> Node number Node name GPFS state<br />-------------------------------------------<br /> 1 myhost2 active<br /> 2 myhost1 active<br /> 3 myhost3 active</span> </p></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;">[root@myhost1 ~]# mmchfs smpi_gpfs -A yes<br />mmchfs: Propagating the cluster configuration data to all<br /> affected nodes. This is an asynchronous process.</span> </p></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;">[root@myhost1 ~]# mmlsfs smpi_gpfs -A<br />flag value description<br />------------------- ------------------------ -----------------------------------<br /> -A yes Automatic mount option</span> </p></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #cccccc;">[root@myhost1 ~]# mmchconfig autoload=yes<br />mmchconfig: Command successfully completed<br />mmchconfig: Propagating the cluster configuration data to all<br /> affected nodes. This is an asynchronous process.<br />[root@myhost1 ~]#</span></p></blockquote></blockquote><p style="text-align: left;">Step 17: Troubleshoot when GPFS node went to inactive state or when disk goes down .</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmlscluster<br />GPFS cluster information<br />========================<br />GPFS cluster name: my_spectrumScale_cluster<br />GPFS cluster id: 9784093264651231821<br />GPFS UID domain: my_spectrumScale_cluster<br />Remote shell command: /usr/bin/ssh<br />Remote file copy command: /usr/bin/scp<br />Repository type: CCR<br /><br />Node Daemon node name IP address Admin node name Designation<br />---------------------------------------------------------------------<br />1 myhost2 10.x.y.1 myhost2 quorum<br />2 myhost1 10.x.y.2 myhost1 quorum-manager<br />3 myhost3 10.x.y.3 myhost3 quorum-manager</span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmgetstate -a</span><br /><span style="background-color: #eeeeee;">Node number Node name GPFS state</span><br /><span style="background-color: #eeeeee;">-------------------------------------------</span><br /><span style="background-color: #eeeeee;">1 myhost1 active</span><br /><span style="background-color: #eeeeee;">2 myhost2 </span><b style="background-color: red;"> down</b><br /><span style="background-color: #eeeeee;">3 myhost3 active</span></span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmstartup -a<br />Tue Jul 20 03:27:03 EDT 2021: mmstartup: Starting GPFS ...<br />myhost2: The GPFS subsystem is already active.<br />myhost3: The GPFS subsystem is already active.</span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmgetstate -a<br /><br />Node number Node name GPFS state<br />-------------------------------------------<br />1 myhost1 active<br />2 myhost2 active<br />3 myhost3 active<br />[root@myhost1 ~]#</span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmunmount smpi_gpfs -a<br />Tue Jul 20 04:12:04 EDT 2021: mmunmount: Unmounting file systems ...<br />[root@myhost1 ~]# </span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmlsdisk smpi_gpfs</span><br /><span style="background-color: #eeeeee;">disk driver sector failure holds holds storage</span><br /><span style="background-color: #eeeeee;">name type size group metadata data status availability pool</span><br /><span style="background-color: #eeeeee;">------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------</span><br /><span style="background-color: #eeeeee;">nsd1 nsd 512 -1 Yes Yes ready up system</span><br /><span style="background-color: #eeeeee;">nsd2 nsd 512 -1 Yes Yes ready </span><b style="background-color: red;">down</b><span style="background-color: #eeeeee;"> system</span><br /><span style="background-color: #eeeeee;">nsd3 nsd 512 -1 Yes Yes ready up system</span><br /><span style="background-color: #eeeeee;">[root@myhost1 ~]#</span></span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmchdisk smpi_gpfs start -d nsd2<br />mmnsddiscover: Attempting to rediscover the disks. This may take a while ...<br />mmnsddiscover: Finished.<br />myhost1: Rediscovered nsd server access to nsd2.<br />Scanning file system metadata, phase 1 ...<br />100 % complete on Tue Jul 20 04:24:14 2021<br />Scan completed successfully.<br />Scanning file system metadata, phase 2 ...<br />100 % complete on Tue Jul 20 04:24:14 2021<br />Scan completed successfully.<br />Scanning file system metadata, phase 3 ...<br />Scan completed successfully.<br />Scanning file system metadata, phase 4 ...<br />100 % complete on Tue Jul 20 04:24:14 2021<br />Scan completed successfully.<br />Scanning file system metadata, phase 5 ...<br />100 % complete on Tue Jul 20 04:24:14 2021<br />Scan completed successfully.<br />Scanning user file metadata ...<br />100.00 % complete on Tue Jul 20 04:24:25 2021 ( 500736 inodes with total 26921 MB data processed)<br />Scan completed successfully.</span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmmount smpi_gpfs -a<br />Tue Jul 20 04:24:42 EDT 2021: mmmount: Mounting file systems ...<br />[root@myhost1 ~]#</span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmlsdisk smpi_gpfs<br />disk driver sector failure holds holds storage<br />name type size group metadata data status availability pool<br />------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------<br />nsd1 nsd 512 -1 Yes Yes ready up system<br />nsd2 nsd 512 -1 Yes Yes ready up system<br />nsd3 nsd 512 -1 Yes Yes ready up system<br />[root@myhost1 ~]#</span></p></blockquote></blockquote></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: left;"><span style="background-color: #eeeeee;">[root@myhost1 ~]# mmgetstate -a<br />Node number Node name GPFS state<br />-------------------------------------------<br />1 myhost1 active<br />2 myhost2 active<br />3 myhost3 active<br />[root@myhost1 ~]#</span></p></blockquote></blockquote></blockquote><p style="text-align: justify;">Step 18: Steps to permanently uninstall GPFS</p><p style="text-align: justify;">- Unmount all GPFS file systems on all nodes by issuing the mmumount all -a command.</p><p style="text-align: justify;">- Issue the mmdelfs command for each file system in the cluster to remove GPFS file systems.</p><p style="text-align: justify;">- Issue the mmdelnsd command for each NSD in the cluster to remove the NSD volume ID from the device.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: justify;"><span style="background-color: #eeeeee;">mmdelfs smpi_gpfs</span><br /><span style="background-color: #eeeeee;">mmdelnsd nsd1</span><br /><span style="background-color: #eeeeee;">mmdelnsd nsd2</span><br /><span style="background-color: #eeeeee;">mmdelnsd nsd3</span></p></blockquote></blockquote></blockquote><p style="text-align: justify;">- Issue the mmshutdown -a command to shutdown GPFS on all nodes.</p><p style="text-align: justify;">- Uninstall GPFS from each node</p><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: justify;"><span style="background-color: #eeeeee;">rpm -qa | grep gpfs | xargs rpm -e --nodeps</span></p></blockquote></blockquote></blockquote><p style="text-align: justify;">Remove the /var/mmfs and /usr/lpp/mmfs directories.</p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><p style="text-align: justify;"><span style="background-color: #eeeeee;">rm -rf /var/mmfs</span><br /><span style="background-color: #eeeeee;">rm -rf /usr/lpp/mmfs</span></p></blockquote></blockquote></blockquote><p> ------------------------------------------------------------------------------------------</p><p style="text-align: justify;">The Quick Start automatically deploys a highly available IBM Spectrum Scale cluster on the Amazon Web Services (AWS) Cloud. This Quick Start deploys IBM Spectrum Scale into a virtual private cloud (VPC) that spans two Availability Zones in your AWS account. You can build a new VPC for IBM Spectrum Scale, or deploy the software into your existing VPC. The deployment and configuration tasks are automated by AWS CloudFormation templates that you can customize during launch.</p><p style="text-align: justify;">IBM's container-native storage solution for OpenShift is designed for enterprise customers who need global hybrid cloud data access. These storage services meet the strict requirements for mission critical data. IBM Spectrum® Fusion provides a streamlined way for organizations to discover, secure, protect and manage data from the edge, to the core data center, to the public cloud.</p><h4 style="text-align: justify;">Spectrum Fusion</h4><p style="text-align: justify;">IBM launched a containerized derivative of its Spectrum Scale parallel file system called Spectrum Fusion. The rationale is that customers need to store and analyze more data at edge sites, while operating in a hybrid and multi-cloud world that requires data availability across all these locations. The ESS arrays provide Edge storage capacity and a containerized Spectrum Fusion can run in any of the locations mentioned. It’s clear that to build, deploy and manage applications requires advanced capabilities that help provide rapid availability to data across the entire enterprise – from the edge to the data center to the cloud. </p><p style="text-align: justify;">Spectrum Fusion combines Spectrum Scale functionality with unspecified IBM data protection software. It will appear first in a hyperconverged infrastructure (HCI) system that integrates compute, storage and networking. This will be equipped with Red Hat Open Shift to support virtual machine and containerized workloads for cloud, edge and containerized data centres.</p><p style="text-align: justify;">Spectrum Fusion will integrate with Red Hat Advanced Cluster Manager (ACM) for managing multiple Red Hat OpenShift clusters, and it will support tiering. Spectrum Fusion provides customers with a streamlined way to discover data from across the enterprise as it has a global index of the data it stores. It will manage a single copy of data only – i.e. there is no need to create duplicate data when moving application workloads across the enterprise. Spectrum Fusion will integrate with IBM’s Cloud Satellite, a managed distribution cloud that deploys and runs apps across the on-premises, edge and cloud environments. </p><p style="text-align: left;"><span style="font-size: x-small;">References:</span><br /><span style="font-size: x-small;">https://www.ibm.com/in-en/products/spectrum-scale</span><br /><span style="font-size: x-small;">https://aws.amazon.com/quickstart/architecture/ibm-spectrum-scale</span><br /><span style="font-size: x-small;">https://www.ibm.com/in-en/products/spectrum-fusion</span><br /><span style="font-size: x-small;">https://www.ibm.com/docs/en/spectrum-scale/5.0.4?topic=installing-spectrum-scale-linux-nodes-deploying-protocols</span></p>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-40886898500324721382020-12-07T16:16:00.027+05:302022-09-06T12:49:09.381+05:30Overview Of MPI Reduction Operations in HPC cluster<div style="text-align: justify;"><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-0KrrMFR2yyA/X89No_IrAdI/AAAAAAAAVUs/PkOdrTK61QQUyOwnFdWphARRKos-tiO5gCLcBGAsYHQ/s373/message_passing_model.PNG" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="261" data-original-width="373" src="https://1.bp.blogspot.com/-0KrrMFR2yyA/X89No_IrAdI/AAAAAAAAVUs/PkOdrTK61QQUyOwnFdWphARRKos-tiO5gCLcBGAsYHQ/s320/message_passing_model.PNG" width="320" /></a></div>Message Passing Interface[ MPI ] is a de facto standard framework for distributed computing in many HPC applications. MPI collective operations involve a group of processes communicating by message passing in an isolated context, known as a communicator. Each process is identified by its rank, an integer number ranging from 0 to P − 1, where P is the size of the communicator. All processes place the same call (SPMD fashion i.e Single Program Multiple Data) depending on the process.<br /></div><div style="text-align: justify;"><br /></div><div style="text-align: justify;">MPI Reductions are among the most useful MPI operations and form an important class of computational operations.
. The operation can be
either user-specified or from the list of pre-defined operations. Usually, the
predefined operations are largely sufficient for any application. </div><div style="text-align: justify;"> </div><div style="text-align: justify;">Consider a
system where you have N processes. The goal of the game is to compute the dot
product of two N-vectors in parallel. Now the dot product of two vectors u and v
Example operation : u⋅v=u1v1+u2v2+...+uNvN . As you can imagine, this is highly
parallelizable. If you have N processes, each process i can compute the
intermediate value ui×vi. Then, the program needs to find a way to sum all of
these values. This is where the reduction comes into play. We can ask MPI to sum
all those value and store them either on only one process (for instance process
0) or to redistribute the value to every process.</div><div style="text-align: justify;"> </div><div style="text-align: justify;">MPI reduction operations fall
into three categories: </div><div style="text-align: justify;"> </div><div style="text-align: justify;">1) Global Reduction Operations: </div><ul style="text-align: left;"><li>MPI REDUCE, </li><li>MPI IREDUCE, </li><li>MPI ALLREDUCE and </li><li>MPI IALLREDUCE. </li></ul><div style="text-align: justify;">2) Combined Reduction and Scatter Operations: </div><ul style="text-align: left;"><li>MPI REDUCE SCATTER, </li><li>MPI IREDUCE SCATTER, </li><li>MPI REDUCE SCATTER BLOCK and </li><li>MPI
IREDUCE SCATTER BLOCK. </li></ul><div style="text-align: justify;"> </div><div style="text-align: justify;">3) Scan Operations: </div><ul style="text-align: left;"><li>MPI SCAN, </li><li>MPI ISCAN, </li><li>MPI EXSCAN, and </li><li>MPI IEXSCAN. </li></ul><div style="text-align: justify;"> </div><div style="text-align: justify;">The primary idea of these operations is to collectively compute on
a set of input data elements to generate a combined output. MPI REDUCE is a
collective function where each process provides some input data (e.g., an array
of double-precision floating-point numbers). This input data is combined through
an MPI operation, as specified by the“op” parameter. Most applications use MPI
predefined operations such as summations or maximum value identification,
although some applications also utilize reductions based on user-defined
function handlers. The MPI operator “op” is always assumed to be associative.
All predefined operations are also assumed to be commutative. Applications,
however, may define their own operations that are associative but not
commutative. The “canonical” evaluation order of a reduction is determined by
the ranks of the processes in the group. However, an MPI implementation can take
advantage of associativity, or associativity and commutativity of the
operations, in order to change the order of evaluation. Doing so may change the
result of the reduction for operations that are not strictly associative and
commutative, such as floating-point addition </div><div style="text-align: justify;"> </div><div style="text-align: justify;">The following predefined operations
are supplied for MPI_REDUCE and related functions MPI_ALLREDUCE,
MPI_REDUCE_SCATTER, and MPI_SCAN. </div><div style="text-align: justify;"> </div><div style="text-align: justify;">These operations are invoked by placing the
following in op </div><ul style="text-align: left;"><li><u><b>[ Name] Meaning </b></u></li><li>[ MPI_MAX] maximum </li><li>[ MPI_MIN] minimum </li><li>[ MPI_SUM]
sum </li><li>[ MPI_PROD] product </li><li>[ MPI_LAND] logical and </li><li>[ MPI_BAND] bit-wise and </li><li>[
MPI_LOR] logical or </li><li>[ MPI_BOR] bit-wise or </li><li>[ MPI_LXOR] logical xor </li><li>[ MPI_BXOR]
bit-wise xor </li><li>[ MPI_MAXLOC] max value and location </li><li>[ MPI_MINLOC] min value and
location </li></ul><p> </p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-Ac7YZAbGDdA/X89JtmqNQLI/AAAAAAAAVUU/zfXqdKiw0kAnMqm10OXGRa6tXDnK5i3sgCLcBGAsYHQ/s985/Example1_MPI_reduce.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="426" data-original-width="985" height="389" src="https://1.bp.blogspot.com/-Ac7YZAbGDdA/X89JtmqNQLI/AAAAAAAAVUU/zfXqdKiw0kAnMqm10OXGRa6tXDnK5i3sgCLcBGAsYHQ/w685-h389/Example1_MPI_reduce.PNG" width="685" /></a></div><br /><div style="text-align: center;"><a href="https://1.bp.blogspot.com/-Mf5yausWGfs/X8397iQXN6I/AAAAAAAAVTI/OylC21G2b8g9vV0k-UV_DuCl_LI-z4njQCLcBGAsYHQ/s974/MPI_reduce_example.PNG"><img border="0" data-original-height="306" data-original-width="974" height="146" src="https://1.bp.blogspot.com/-Mf5yausWGfs/X8397iQXN6I/AAAAAAAAVTI/OylC21G2b8g9vV0k-UV_DuCl_LI-z4njQCLcBGAsYHQ/w508-h146/MPI_reduce_example.PNG" width="508" /></a></div><br /><p></p><p><span style="font-size: x-small;"><b> Example 1:</b></span> Get the memory on each node and perform MPI_SUM operation to calculate average Memory on the cluster.</p><p><span></span></p><a name='more'></a><span style="font-family: arial;"><span style="font-size: small;"><b>cat reduce_sum.c</b></span><span style="font-size: small;"><br /></span></span><div style="margin-left: 120px; text-align: left;"><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>#include <stdio.h></span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>#include <stdlib.h></span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>#include <mpi.h></span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>#include <assert.h></span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>#include <time.h></span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>#include <string.h></span></span></span></span></span></div><div style="margin-left: 120px; text-align: left;"><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>//Get meminfo function to calc size of RAM</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>char *get_meminfo()</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>{</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> FILE *in=NULL;</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> char temp[256];</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> char *ram_size=(char*)malloc(256);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> assert(ram_size != NULL);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> in=popen("cat /proc/meminfo | grep MemTotal | cut -c15-26", "r");</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> fgets(temp, 255, in);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> strcpy(ram_size, temp);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> //printf("In function %s", ram_size);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> pclose(in);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> return ram_size;</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>}</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>int main(int argc, char** argv) {</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> MPI_Init(NULL, NULL);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> int world_rank;</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> int world_size;</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> MPI_Comm_size(MPI_COMM_WORLD, &world_size);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> // Get the size of RAM locally</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> char *size_of_RAM=NULL;</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> int local_val;</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> size_of_RAM=get_meminfo();</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> local_val=atoi(size_of_RAM);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> // Print the SIZE of memory on each process</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> printf("Size of Local Memory for process %d , RAM_size_local= %d\n",</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> world_rank, local_val);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> // Reduce all of the local meminfo into the global meminfo</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> int global_val;</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span><span><b> MPI_Reduce(&local_val, &global_val, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);</b></span></span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> // Print the result</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> if (world_rank == 0) {</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> printf("Total global sum = %d, avg = %d\n", global_val,</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> global_val / (world_size));</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> }</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> // Clean up</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> free(size_of_RAM);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> MPI_Barrier(MPI_COMM_WORLD);</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span> MPI_Finalize();</span></span></span></span></span><br /><span style="font-family: trebuchet;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span><span>}</span></span></span></span></span><br /></div><p><span style="font-size: xx-small;"><span style="font-family: trebuchet;"><b>Compile :</b></span></span><br /> mpicc -o reduce_sum reduce_sum.c<br />--------------------------------------------------------------<br />$mpirun -np 6 -host exechost03:1,exechost04:1,exechost05:1,exechost06:1,exechost07:1,exechost08:1 reduce_sum<i> </i></p><p><i>Size of Local Memory for process 3 , RAM_size_local= 32742748<br />Size of Local Memory for process 4 , RAM_size_local= 32742748<br />Size of Local Memory for process 2 , RAM_size_local= 65666080<br />Size of Local Memory for process 1 , RAM_size_local= 65666088<br />Size of Local Memory for process 5 , RAM_size_local= 65633576<br />Size of Local Memory for process 0 , RAM_size_local= 65669284<br />Total global sum = 328120524, avg = 54686754</i><br />$<br /></p><span><!--more--></span><p><b><span style="font-size: x-small;">Example 2:</span> </b>Get the memory on each node and perform MPI_MIN operation.</p><p><span style="font-size: x-small;"><b> <span style="font-family: arial;"><span style="font-size: small;">cat reduce_min.c</span></span></b></span><span style="font-family: arial;"><span style="font-size: small;"><br /></span></span></p><p style="margin-left: 80px; text-align: left;"><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>#include <stdio.h></span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>#include <stdlib.h></span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>#include <mpi.h></span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>#include <assert.h></span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>#include <time.h></span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>#include <string.h></span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>//Get meminfo function to calc size of RAM</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>char *get_meminfo()</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>{</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> FILE *in=NULL;</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> char temp[256];</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> char *ram_size=(char*)malloc(256);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> assert(ram_size != NULL);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> in=popen("cat /proc/meminfo | grep MemTotal | cut -c15-26", "r");</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> fgets(temp, 255, in);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> strcpy(ram_size, temp);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> //printf("In function %s", ram_size);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> pclose(in);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> return ram_size;</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>}</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>int main(int argc, char** argv) {</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> MPI_Init(NULL, NULL);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> int world_rank;</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> int world_size;</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> MPI_Comm_size(MPI_COMM_WORLD, &world_size);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> // Get the size of RAM locally</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> char *size_of_RAM=NULL;</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> int local_val;</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> size_of_RAM=get_meminfo();</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> local_val=atoi(size_of_RAM);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> // Print the SIZE of memory on each process</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> printf("Size of Local Memory for process %d , RAM_size_local= %d\n",</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> world_rank, local_val);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> // Reduce all of the local meminfo into the global meminfo</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> int global_val;</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> <span><b> MPI_Reduce(&local_val, &global_val, 1, MPI_INT, MPI_MIN, 0, MPI_COMM_WORLD);</b></span></span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> // Print the result</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> if (world_rank == 0) {</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> printf("Total global min val = %d\n", global_val);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> }</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> // Clean up</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> free(size_of_RAM);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> MPI_Barrier(MPI_COMM_WORLD);</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span> MPI_Finalize();</span></span></span></span></span><br /><span style="font-family: arial;"><span style="font-size: small;"><span style="font-size: x-small;"><span style="font-family: georgia;"><span>}</span></span></span></span></span><br /></p><p>--------------</p><p><span style="font-size: xx-small;"><b>Compile :</b></span><br /> mpicc -o reduce_min reduce_min.c<br />$mpirun -np 6 -host exechost03:1,exechost04:1,exechost05:1,exechost06:1,exechost07:1,exechost08:1 reduce_min<br /><i>Size of Local Memory for process 3 , RAM_size_local= 32742748<br />Size of Local Memory for process 1 , RAM_size_local= 65666088<br />Size of Local Memory for process 2 , RAM_size_local= 65666080<br />Size of Local Memory for process 4 , RAM_size_local= 32742748<br />Size of Local Memory for process 5 , RAM_size_local= 65633576<br />Size of Local Memory for process 0 , RAM_size_local= 65669284<br />Total global min val = <b>32742748</b></i><br />$<span style="font-size: x-small;"><b> </b></span></p><p><span style="font-size: x-small;"><b>Another example where you can find Minimum Free memory available and Maximum PPN value to calculate max_bytes_per_rank to avoid OOM</b> .<b> </b></span></p><p><span style="font-size: x-small;"><b>cat reduce_ppn_Memory.c</b></span><br /></p><p style="margin-left: 40px; text-align: left;"><span style="font-family: arial;">#include <stdio.h></span><br /><span style="font-family: arial;">#include <stdlib.h></span><br /><span style="font-family: arial;">#include <mpi.h></span><br /><span style="font-family: arial;">#include <assert.h></span><br /><span style="font-family: arial;">#include <time.h></span><br /><span style="font-family: arial;">#include <string.h></span><br /><span style="font-family: arial;">#include <sys/sysinfo.h></span><br /><br /><span style="font-family: arial;">int main(int argc, char** argv) {</span><br /><br /><span style="font-family: arial;"> MPI_Init(NULL, NULL);</span><br /><span style="font-family: arial;"> const double gigabyte = 1024 * 1024 *1024;</span><br /><span style="font-family: arial;"> double local_free_mem, global_min_memory, max_bytes_per_rank;</span><br /><span style="font-family: arial;"> int len, local_ppn, global_max_ppn, world_rank, world_size;</span><br /><span style="font-family: arial;"> char name[MPI_MAX_PROCESSOR_NAME];</span><br /><span style="font-family: arial;"> struct sysinfo si;</span><br /><span style="font-family: arial;"> sysinfo (&si);</span><br /><br /><span style="font-family: arial;"> MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);</span><br /><span style="font-family: arial;"> MPI_Comm_size(MPI_COMM_WORLD, &world_size);</span><br /><br /><span style="font-family: arial;"> MPI_Comm shared_comm;</span><br /><span style="font-family: arial;"> MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &shared_comm);</span><br /><span style="font-family: arial;"> MPI_Get_processor_name(name, &len);</span><br /><span style="font-family: arial;"> MPI_Comm_size(shared_comm, &local_ppn);</span><br /><span style="font-family: arial;"> MPI_Comm_free(&shared_comm);</span><br /><br /><span style="font-family: arial;"> /* Get the size of Free RAM locally */</span><br /><span style="font-family: arial;"> local_free_mem = (si.freeram * si.mem_unit) / gigabyte;</span><br /><br /><span style="font-family: arial;"> /* Print the SIZE of memory on each process */</span><br /><span style="font-family: arial;"> printf("Size of Local Memory for process %d is %f GB and local_ppn is %d on %s \n", world_rank, local_free_mem, local_ppn, name);</span><br /><br /><span style="font-family: arial;"> /* Reduce all of the local meminfo & ppn into the global meminfo & ppn */</span><br /><span style="font-family: arial;"> /* Get Minimum Memory available and Maximum value of PPN on all the nodes */</span><br /><br /><span style="font-family: arial;"> MPI_Reduce(&local_free_mem, &global_min_memory, 1, MPI_DOUBLE, MPI_MIN, 0, MPI_COMM_WORLD);</span><br /><span style="font-family: arial;"> MPI_Reduce(&local_ppn, &global_max_ppn, 1, MPI_INT, MPI_MAX, 0, MPI_COMM_WORLD);</span><br /><br /><span style="font-family: arial;"> /* Print the result Minimum Free memory available and Maximum PPN value */</span><br /><br /><span style="font-family: arial;"> if (world_rank == 0) {</span><br /><span style="font-family: arial;"> printf("After MPI reduction, Minimum Free memory available is %f GB\n", global_min_memory);</span><br /><span style="font-family: arial;"> printf("After MPI reduction, Maximum PPN value is %d\n", global_max_ppn);</span><br /><br /><span style="font-family: arial;"> /* max_bytes_per_rank calculation based on 50% of minimum free memory available */</span><br /><span style="font-family: arial;"> /* divided by max value of PPN on the nodes in cluster. This will help to avoid OOM Failure */</span><br /><span style="font-family: arial;"> /* in heterogeneous cluster where some of the nodes installed with lesser memory (Example 32GB RAM) */</span><br /><br /><span style="font-family: arial;"> max_bytes_per_rank = (global_min_memory * 0.5) / global_max_ppn;</span><br /><span style="font-family: arial;"> printf("Maximum bytes per rank is %f GB \n", max_bytes_per_rank);</span><br /><span style="font-family: arial;"> }</span><br /><br /><span style="font-family: arial;"> MPI_Barrier(MPI_COMM_WORLD);</span><br /><span style="font-family: arial;"> MPI_Finalize();</span><br /><span style="font-family: arial;">}</span><br /></p><p>---------------------------------------------------------------------------------------------<span style="font-size: xx-small;"><b> </b></span></p><p><span style="font-size: xx-small;"><b>Compile:</b></span><br /><i>[sachinpb@exechost04]$ mpicc -o reduce_ppn_Memory reduce_ppn_Memory.c<br />[sachinpb@exechost04]$</i><br /><b><br /><span style="font-size: xx-small;">Run executable with 5 procs on 4 nodes :</span></b><i> </i></p><p><i>[sachinpb@exechost04]$ mpirun -np 5 -host exechost04:1,exechost05:1,exechost06:1,exechost07:2 reduce_ppn_Memory<br />Size of Local Memory for process 0 is 59.349953 GB and local_ppn is 1 on exechost04<br />Size of Local Memory for process 2 is 29.662674 GB and local_ppn is 1 on exechost06<br />Size of Local Memory for process 1 is 59.596630 GB and local_ppn is 1 on exechost05<br />Size of Local Memory for process 3 is 29.628567 GB and local_ppn is 2 on exechost07<br />Size of Local Memory for process 4 is 29.628567 GB and local_ppn is 2 on exechost07<br /><br />After MPI reduction, Minimum Free memory available is 29.628567 GB<br />After MPI reduction, Maximum PPN value is 2<br />Maximum bytes per rank is 7.407142 GB</i><br />------------------------------------------------------------------------------------------------------<br /><span style="font-size: xx-small;"><b>Run executable with 13 procs on 8 nodes :</b></span><i> </i></p><p><i>[sachinpb@exechost04]$ mpirun -np 13 -host exechost03:1,exechost04:4,exechost05:1,exechost06:1,exechost07:1,exechost08:3,exechost09:1,exechost10:1 reduce_ppn_Memory<br />Size of Local Memory for process 7 is 29.717995 GB and local_ppn is 1 on exechost07<br />Size of </i><i><i>Local </i>Memory for process 6 is 29.657806 GB and local_ppn is 1 on exechost06<br />Size of </i><i><i>Local </i>Memory for process 5 is 59.594044 GB and local_ppn is 1 on exechost05<br />Size of </i><i><i>Local </i>Memory for process 11 is 60.412170 GB and local_ppn is 1 on exechost09<br />Size of </i><i><i>Local </i>Memory for process 12 is 46.994701 GB and local_ppn is 1 on exechost10<br />Size of </i><i><i>Local </i>Memory for process 4 is 59.661568 GB and local_ppn is 1 on exechost03<br />Size of </i><i><i>Local </i>Memory for process 1 is 59.044613 GB and local_ppn is 4 on exechost04<br />Size of </i><i><i>Local </i>Memory for process 0 is 59.044613 GB and local_ppn is 4 on exechost04<br />Size of </i><i><i>Local </i>Memory for process 3 is 59.044613 GB and local_ppn is 4 on exechost04<br /> Size of </i><i><i>Local </i>Memory for process 2 is 59.044613 GB and local_ppn is 4 on exechost04<br />Size of </i><i><i>Local </i>Memory for process 9 is 60.254044 GB and local_ppn is 3 on exechost08<br /> Size of </i><i><i>Local </i>Memory for process 10 is 60.254044 GB and local_ppn is 3 on exechost08<br /> Size of </i><i><i>Local </i>Memory for process 8 is 60.254044 GB and local_ppn is 3 on exechost08<br /><br />After MPI reduction, Minimum Free memory available is 29.657806 GB<br />After MPI reduction, Maximum PPN value is 4<br />Maximum bytes per rank is 3.707226 GB</i><br /><br /><span></span></p><!--more--><p></p><div style="text-align: justify;"><b>MPI_Allreduce</b><br /><br />Many parallel applications will require accessing the reduced results across all processes rather than the root process. In a similar complementary style of MPI_Allgather to MPI_Gather, MPI_Allreduce will reduce the values and distribute the results to all processes. Allreduce is a complex collective operation that performs a combination of vectors owned by processes into a result vector which is distributed back to the processes. In MPI Allreduce operation can be accessed by calling MPI Allreduce()<br />function .The function prototype is the following:</div><div style="text-align: justify;"> <div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-tQDU8EOUjHA/X84GAWom4jI/AAAAAAAAVTs/Wx-RFOiyQr0sjEYDGfoXnDxiYvt4fthtgCLcBGAsYHQ/s383/All_reduce_prototype.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="237" data-original-width="383" src="https://1.bp.blogspot.com/-tQDU8EOUjHA/X84GAWom4jI/AAAAAAAAVTs/Wx-RFOiyQr0sjEYDGfoXnDxiYvt4fthtgCLcBGAsYHQ/s320/All_reduce_prototype.PNG" width="320" /></a></div><br /></div><div style="text-align: justify;">The example above shows MPI_Reduce in which the reduction operation takes place on only one process (in this case process 0). In our case, the reception buffer (result) is only valid for process 0. The other processes will not have a valid value stored in result. Sometimes, you might want to have the result of the reduction stored on all processes, in which case MPI_Reduce is not suited. In such a case, you can use MPI_Allreduce to store the result on every process. So, if we had used MPI_Allreduce instead of MPI_Reduce in the example, all processes would have a valid value in result and could be using this value after the communication. MPI_Allreduce is the equivalent of doing MPI_Reduce followed by an MPI_Bcast.<br /><br /><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-90zVwRAfeNI/X84GiFvNyVI/AAAAAAAAVT4/h_99R1WRofsYqkcGIiIDKu-ie7hoMBQMACLcBGAsYHQ/s654/mpi_allreduce.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="277" data-original-width="654" height="185" src="https://1.bp.blogspot.com/-90zVwRAfeNI/X84GiFvNyVI/AAAAAAAAVT4/h_99R1WRofsYqkcGIiIDKu-ie7hoMBQMACLcBGAsYHQ/w436-h185/mpi_allreduce.PNG" width="436" /></a></div><br />One of the more common applications of the reduction operation
is the
<i>inner product</i>
computation. Typically, you have two vectors <span class="MathJax" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>x</mi><mo>,</mo><mi>y</mi></math>" id="MathJax-Element-16-Frame" role="presentation" style="position: relative;" tabindex="0"><nobr aria-hidden="true"><span class="math" id="MathJax-Span-143" style="display: inline-block; width: 1.781em;"><span style="display: inline-block; font-size: 116%; height: 0px; position: relative; width: 1.509em;"><span style="clip: rect(1.659em, 1001.51em, 2.63em, -1000em); left: 0em; position: absolute; top: -2.263em;"><span class="mrow" id="MathJax-Span-144"><span class="mi" id="MathJax-Span-145" style="font-family: MathJax_Math; font-style: italic;">x</span><span class="mo" id="MathJax-Span-146" style="font-family: MathJax_Main;">,</span><span class="mi" id="MathJax-Span-147" style="font-family: MathJax_Math; font-style: italic; padding-left: 0.167em;">y<span style="display: inline-block; height: 1px; overflow: hidden; width: 0.006em;"></span></span></span><span style="display: inline-block; height: 2.263em; width: 0px;"></span></span></span><span style="border-left: 0px solid; display: inline-block; height: 0.876em; overflow: hidden; vertical-align: -0.3em; width: 0px;"></span></span></nobr></span>
that have the same distribution, that is,
where all processes store equal parts of <span class="MathJax" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>x</mi></math>" id="MathJax-Element-17-Frame" role="presentation" style="position: relative;" tabindex="0"><nobr aria-hidden="true"><span class="math" id="MathJax-Span-148" style="display: inline-block; width: 0.703em;"><span style="display: inline-block; font-size: 116%; height: 0px; position: relative; width: 0.593em;"><span style="clip: rect(1.605em, 1000.54em, 2.382em, -1000em); left: 0em; position: absolute; top: -2.209em;"><span class="mrow" id="MathJax-Span-149"><span class="mi" id="MathJax-Span-150" style="font-family: MathJax_Math; font-style: italic;">x</span></span><span style="display: inline-block; height: 2.209em; width: 0px;"></span></span></span><span style="border-left: 0px solid; display: inline-block; height: 0.65em; overflow: hidden; vertical-align: -0.075em; width: 0px;"></span></span></nobr></span> and <span class="MathJax" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>y</mi></math>" id="MathJax-Element-18-Frame" role="presentation" style="position: relative;" tabindex="0"><nobr aria-hidden="true"><span class="math" id="MathJax-Span-151" style="display: inline-block; width: 0.595em;"><span style="display: inline-block; font-size: 116%; height: 0px; position: relative; width: 0.485em;"><span style="clip: rect(1.605em, 1000.49em, 2.576em, -1000em); left: 0em; position: absolute; top: -2.209em;"><span class="mrow" id="MathJax-Span-152"><span class="mi" id="MathJax-Span-153" style="font-family: MathJax_Math; font-style: italic;">y<span style="display: inline-block; height: 1px; overflow: hidden; width: 0.006em;"></span></span></span><span style="display: inline-block; height: 2.209em; width: 0px;"></span></span></span><span style="border-left: 0px solid; display: inline-block; height: 0.876em; overflow: hidden; vertical-align: -0.3em; width: 0px;"></span></span></nobr></span>. For Example :<span class="MathJax" data-mathml="<math xmlns="http://www.w3.org/1998/Math/MathML"><mi>y</mi></math>" id="MathJax-Element-18-Frame" role="presentation" style="position: relative;" tabindex="0"><nobr aria-hidden="true"><span class="math" id="MathJax-Span-151" style="display: inline-block; width: 0.595em;"></span></nobr></span>
<pre>local_inprod = 0;
for (i=0; i<localsize; i++)
local_inprod += x[i]*y[i];
MPI_Allreduce( &local_inprod, &global_inprod, 1,MPI_DOUBLE ... )
</pre>
<p name="switchToTextMode">
</p><p>
</p><p style="text-align: justify;">Recently one more application for the Allreduce operation has emerged. Distributed training of Deep Neural Networks(DNN) uses Allreduce operation to synchronize neural network parameters between separate training processes after each step of gradient descent optimization. This new application involves data sets of medium and big sizes which depends on a particular neural network model bringing new requirements to the Allreduce performance. </p><p style="text-align: justify;"><b>MPI_SCAN</b> is used to perform a prefix reduction on data distributed across the group. Parallel prefix, also known as scanning operation calculates the reduction of all parts of the local data stored in the process. The operation returns, in the receive buffer of the process with rank i, the reduction of the values in the send buffers of processes with ranks 0,...,i (inclusive). The type of operations supported, their semantics, and the constraints on send and receive buffers are as for MPI_REDUCE. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-Qp0tEIE_m5o/X89JTwd6hpI/AAAAAAAAVUM/1eNgVooC2BgpFhhZqUcg38aR_I7CPCuIgCLcBGAsYHQ/s1012/example_MPI_scan.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="442" data-original-width="1012" height="296" src="https://1.bp.blogspot.com/-Qp0tEIE_m5o/X89JTwd6hpI/AAAAAAAAVUM/1eNgVooC2BgpFhhZqUcg38aR_I7CPCuIgCLcBGAsYHQ/w540-h296/example_MPI_scan.PNG" width="540" /></a></div><p>The diagram below shows the schematic diagram of reduction operation and parallel prefix operation</p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-wm6o9o8oT00/X89LDFv_0KI/AAAAAAAAVUg/91_nWolDeJQTScOaxderDGs626PqC9a1wCLcBGAsYHQ/s1042/example_reduction_scan.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="533" data-original-width="1042" height="283" src="https://1.bp.blogspot.com/-wm6o9o8oT00/X89LDFv_0KI/AAAAAAAAVUg/91_nWolDeJQTScOaxderDGs626PqC9a1wCLcBGAsYHQ/w484-h283/example_reduction_scan.PNG" width="484" /></a></div><p style="text-align: justify;"><b>MPI_Exscan </b>is an exclusive scan: it performs a prefix reduction across all MPI processes in the given communicator, excluding the calling MPI process. <b> </b>The operation returns, in the recvbuf of the process with rank i, the reduction (calculated according to the function op) of the values in the sendbufs of processes with ranks 0, ..., i-1. Compare this with the functionality of MPI_Scan, which calculates over the range 0, ..., i (inclusive). The type of operations supported, their semantics, and the constraints on send and receive buffers are as for MPI_Reduce. The value in recvbuf on process 0 is undefined and unreliable as recvbuf is not significant for process 0. The value of recvbuf on process 1 is always the value in sendbuf on process 0. </p><p style="text-align: justify;"></p><div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-m2qljpABG8Q/X8-w_M9yNpI/AAAAAAAAVU4/kG6T_brTUYITME2lcJk1RRvP32xIl_IcACLcBGAsYHQ/s831/MPI_exscan.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="603" data-original-width="831" height="339" src="https://1.bp.blogspot.com/-m2qljpABG8Q/X8-w_M9yNpI/AAAAAAAAVU4/kG6T_brTUYITME2lcJk1RRvP32xIl_IcACLcBGAsYHQ/w517-h339/MPI_exscan.PNG" width="517" /></a></div><br /><span style="font-size: x-small;">-----------</span><p></p><p style="margin-left: 80px; text-align: justify;"><span style="font-size: x-small;">int main(int argc, char* argv[])<br />{<br /> MPI_Init(&argc, &argv);<br /> <br /> // Get my rank<br /> int my_rank;<br /> MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);<br /> <br /> // Get the sum of all ranks up to the one before mine and print it<br /> int total;<br /> MPI_Exscan(&my_rank, &total, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);<br /> <br /> // The result on MPI process 0 is undefined, do not print it<br /> if(my_rank == 0)<br /> {<br /> printf("[MPI process 0] Total = undefined.\n");<br /> }<br /> else<br /> {<br /> printf("[MPI process %d] Total = %d.\n", my_rank, total);<br /> }<br /> <br /> MPI_Finalize();<br /> <br /> return EXIT_SUCCESS;<br />} </span></p><p style="text-align: justify;"><span style="font-size: x-small;">-----------</span><br /></p><p style="text-align: justify;">MPI_IN_PLACE tells MPI_Allreduce to use a single buffer as both the sending and receiving buffer.The "in place'' operations are provided to reduce unnecessary memory motion by both the MPI implementation and by the user. By allowing the 'in place'[MPI_IN_PLACE ] option, the receive buffer in many of the collective calls becomes a send-and-receive buffer(i.e send and receive buffer are one same buffer). Collective communication applies to intracommunicators. If the operation is rooted (e.g., broadcast, gather, scatter), then the transfer is unidirectional. Non-rooted operations, such as all-to-all, will often occur as part of an exchange, where it makes sense to communicate in both directions at once (i.e transfer is bidirectional). Note that the ``in place'' option does not apply to intercommunicators since in the intercommunicator case there is no communication from a process to itself. <br /></p><p style="text-align: justify;">An intra-communicator refers to a single group, an inter-communicator refers to a pair of groups. The intra-communicator is simply the group of all processes which share that communicator. Collective communications can be performed with an intra-communicator. They cannot be performed on an inter-communicator.The "in place" option for
intracommunicators is specified by passing the value MPI_IN_PLACE to the
argument sendbuf at the root. In such a case, the input data is taken
at the root from the receive buffer, where it will be replaced by the
output data [i.e use if-else to pass MPI_IN_PLACE as sendbuf only on the
root [not for other ranks].<br /></p><p style="text-align: left;"><span style="font-size: x-small;"><b>For example :</b></span><br />-------------------------<br /><span style="font-size: x-small;">if ( rank == root ) <br /> rc = MPI_Reduce( MPI_IN_PLACE, buffer, size * count, MPI_INT, MPI_SUM, root, MPI_COMM_WORLD );<br />else <br /> rc = MPI_Reduce( buffer, NULL , size * count, MPI_INT, MPI_SUM, root, MPI_COMM_WORLD );</span><br /><span style="font-size: xx-small;"><br />NOTE:
For gather and most other collectives, MPI_IN_PLACE should passed as
the sendbuf. For scatter/scatterv, MPI_IN_PLACE should be passed as the
recvbuf. </span><br /></p><p style="text-align: justify;">On the other side, inter-communication is a point-to-point communication between processes in different groups. Inter-communicators are more likely to be used by parallel library designers than application developers. If comm is an intercommunicator, then the result of the reduction of the data provided by processes in group A is stored at each process in group B, and vice versa. Both groups should provide count and datatype arguments that specify the same type signature. <br /></p><p>I hope this blog helped in understanding MPI reduction operations and running sample code on your HPC cluster. </p><p>--------------------------------------------------------------------------------------------------------------------</p><div style="text-align: left;"><b><span style="font-size: xx-small;">Reference:</span></b></div><div style="margin-left: 40px; text-align: left;"><span style="font-size: xx-small;">https://www.open-mpi.org/doc/v4.0/man3/MPI_Reduce.3.php</span></div><div style="margin-left: 40px; text-align: left;"><span style="font-size: xx-small;">https://www.programmersought.com/article/96223924595 </span></div><div style="margin-left: 40px; text-align: left;"><span style="font-size: xx-small;">https://www.rookiehpc.com/mpi/docs/</span><br /></div><div style="margin-left: 40px; text-align: left;"><span style="font-size: xx-small;">http://www.rc.usf.edu/tutorials/classes/tutorial/mpi/chapter8.html</span></div>Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-14310534291883034932020-10-20T05:41:00.024+05:302020-10-23T17:23:48.715+05:30Ansible Concepts: Run first Command and Playbook on Linux cluster<div class="separator" style="clear: both; text-align: center;"><a href="https://1.bp.blogspot.com/-yntuv8-zYI0/X44rPkXEETI/AAAAAAAAVP4/IQ2obDrVT1ApR8epaVMry9Kz6H36aGe1ACLcBGAsYHQ/s554/ansible_logo.PNG" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="393" data-original-width="554" height="177" src="https://1.bp.blogspot.com/-yntuv8-zYI0/X44rPkXEETI/AAAAAAAAVP4/IQ2obDrVT1ApR8epaVMry9Kz6H36aGe1ACLcBGAsYHQ/w320-h177/ansible_logo.PNG" width="320" /></a></div><p style="text-align: justify;"><span style="font-family: arial;">Ansible is a configuration management and orchestration tool that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs. The open source product is maintained by Ansible Inc. It was first released in 2012. Red Hat acquired Ansible in 2015. Red Hat Ansible Engine and Red Hat Ansible Tower are commercial products. Ansible can be run directly from the command line without setting up any configuration files. You only need to install Ansible on the control server or node. It communicates and performs the required tasks using SSH. No other installation is required. This is different from other orchestration tools like Chef and Puppet where you have to install software both on the control and client nodes.It uses no agents and no additional custom security infrastructure, so it's easy to deploy - and most importantly, it uses a very simple language (YAML, in the form of Ansible Playbooks)Ansible uses configuration files called playbooks for a series of tasks. The playbooks are written in YAML syntax. That allow you to describe your automation jobs in a way that approaches plain English.</span></p><p style="text-align: justify;"> <br /></p><p></p><p><span style="font-size: medium;"><b>Architecture:</b></span></p><p><span style="font-size: medium;"><b></b></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-size: medium;"><b><a href="https://1.bp.blogspot.com/-9Gz34W0oZuM/X45N4RR05dI/AAAAAAAAVQE/MveqA60c_VU7Ep7EhwwTke--kkBe-2iZQCLcBGAsYHQ/s785/ansible_arch.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="464" data-original-width="785" height="339" src="https://1.bp.blogspot.com/-9Gz34W0oZuM/X45N4RR05dI/AAAAAAAAVQE/MveqA60c_VU7Ep7EhwwTke--kkBe-2iZQCLcBGAsYHQ/w526-h339/ansible_arch.PNG" width="526" /></a></b></span></div><span style="font-size: medium;"><b><br /></b> The Ansible Automation engine consists of:<br /><br /><b>Control node</b><br /><br />Any machine with Ansible installed. You can run commands and playbooks, invoking /usr/bin/ansible or /usr/bin/ansible-playbook, from any control node. You can use any computer that has Python installed on it as a control node - laptops, shared desktops, and servers can all run Ansible. However, you cannot use a Windows machine as a control node. You can have multiple control nodes.</span><p></p><p><br /><span style="font-size: medium;"><img border="0" data-original-height="310" data-original-width="326" height="263" src="https://1.bp.blogspot.com/-IczPhLWCw5Y/X2Xl7G8NXtI/AAAAAAAAVMY/5HFpdYJMjQAurikLGNRLUXtqNXSl9lNrQCLcBGAsYHQ/w352-h263/ansible_architecture.PNG" width="352" /><br /><b>Managed nodes</b><br /><br />The network devices (and/or servers) you manage with Ansible. Managed nodes are also sometimes called “hosts”. Ansible is not installed on managed nodes.<br /><br /><b>Inventory</b><br /><br />A list of managed nodes. An inventory file is also sometimes called a “hostfile”. Your inventory can specify information like IP address for each managed node. An inventory can also organize managed nodes, creating and nesting groups for easier scaling. Inventories can be of two types static and dynamic, dynamic inventory can be covered while you go through Ansible thoroughly.<br /><b><br />Modules</b><br /><br />The units of code Ansible executes. Each module has a particular use, from administering users on a specific type of database to managing VLAN interfaces on a specific type of network device. You can invoke a single module with a task, or invoke several different modules in a playbook.<br /><br /><b>Tasks</b><br /><br />The units of action in Ansible. You can execute a single task once with an ad-hoc command.<br /><br /><b>Playbooks<br /></b><br />Ordered lists of tasks, saved so you can run those tasks in that order repeatedly. Playbooks can include variables as well as tasks. Playbooks are written in YAML and are easy to read, write, share and understand. </span></p><p><span style="font-size: medium;"><b>CMDB(Configuration Management Database.) :</b><br /><br />It is a repository that acts as a data warehouse for IT installations. It holds data relating to a collection of IT assets (commonly referred to as configuration items (CI)), as well as to describe relationships between such assets.<br /><br /><b>Cloud:</b><br />A network of remote servers on which you can store, manage and process your data, these servers are hosted on internet, storing the data remotely rather than local servers, just launch your resources and instances on cloud, connect them to your servers and you’ve the wisdom of operating your task remotely. <br /></span></p><p><span style="font-size: medium;"><b> </b></span><br /><a href="https://1.bp.blogspot.com/-lmYyqwylJ3w/X2XqNBIOwEI/AAAAAAAAVMk/957PweJBN_AR88jsfvXeGc8tDOTsQkNEwCLcBGAsYHQ/s712/ansible_master_slave.PNG" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="512" data-original-width="712" height="460" src="https://1.bp.blogspot.com/-lmYyqwylJ3w/X2XqNBIOwEI/AAAAAAAAVMk/957PweJBN_AR88jsfvXeGc8tDOTsQkNEwCLcBGAsYHQ/w640-h460/ansible_master_slave.PNG" width="640" /></a><br /></p><p style="text-align: justify;"><span style="font-family: arial;">Ansible works by connecting to your nodes and pushing out small programs, called "Ansible modules" to them. These programs are written to be resource models of the desired state of the system. Ansible then executes these modules (over SSH by default), and removes them when finished.<br /><br />Your library of modules can reside on any machine, and there are no servers, daemons, or databases required. Typically you'll work with your favorite terminal program, a text editor, and probably a version control system to keep track of changes to your content. Passwords are supported, but SSH keys with ssh-agent are one of the best ways to use Ansible. </span></p><p style="text-align: justify;"><span style="font-family: arial;">By default, Ansible represents what machines it manages using a very simple INI file that puts all of your managed machines in groups of your own choosing. The Ansible inventory file defines the hosts and groups of hosts upon which commands, modules, and tasks in a playbook operate. It resides under the /etc/ansible directory. If necessary, you can also create project-specific inventory files in alternate locations. </span></p><p><span style="font-family: arial;"><i><b>How to install Ansible on RHEL8 machine:</b></i></span></p><p><span style="font-family: arial;">Install instructions for Ansible Engine on RHEL on IBM Power (little endian).<br /></span><span style="font-size: x-small;"><span style="font-family: arial;">RHEL 8: (POWER8, POWER9) </span><br />subscription-manager repos --enable="ansible-2.9-for-rhel-8-ppc64le-rpms"<br />yum install ansible </span></p><p>Verify installed version of ansible :</p><p><span style="font-size: x-small;">[root@myhost123 example]# ansible --version<br />ansible 2.10.2<br /> config file = /root/sachin/example/ansible.cfg<br /> configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']<br /> ansible python module location = /root/.local/lib/python3.6/site-packages/ansible<br /> executable location = /usr/bin/ansible<br /> python version = 3.6.8 (default, Dec 5 2019, 16:11:43) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]</span><br /></p><p style="text-align: justify;"><span style="font-family: arial;"><span style="font-family: inherit;">As explained in above section, Ansible Engine consists of Inventory, API, Modules and Plugins. A
user writes playbooks i.e. set of tasks, then the playbook scans the
inventory and matches for the listed hosts or IP addresses where the
tasks must be executed. Ansible copies all the modules to the managed
node and using Python API calls and plugins. Ansible completes the given
tasks. Once the tasks are completed/executed all the modules are
destroyed on the Managed Nodes. Ansible on linux executes the modules on
managed hosts using SSH</span><b><i> <br /></i></b></span></p><p><span style="font-family: arial;"><b><i>How to use ANSIBLE for ad-hoc parallel task execution:</i></b><br />Once you have an instance available, you can talk to it right away, without any additional setup:</span></p><p>ansible 'hosts' -m module_name <br /></p><p>Eg: ansible 'localhost' -m shell -a 'id'<br /><br />ansible all -m ping <br />ansible hostname.com -m yum -a "name=httpd state=installed"<br />ansible hostname.com -a "/usr/sbin/reboot"</p><p><i><b>Examples: <br /></b></i></p><p><span style="font-size: x-small;">CASE 1:[root@myhost123 example]# ansible all -m ping<br />myhost123 | SUCCESS => {<br /> "ansible_facts": {<br /> "discovered_interpreter_python": "/usr/bin/python"<br /> },<br /> "changed": false,<br /> "ping": "pong"<br />}</span><br />---------------------<br /><span style="font-size: x-small;">CASE 2: # ansible 'localhost' -m shell -a 'id'<br />localhost | CHANGED | rc=0 >><br />uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023<br /></span><br />-----------------------------<br /><span style="font-size: x-small;">CASE 3: # ansible myhost123 -m yum -a "name=httpd state=installed"<br /><br />myhost123 | CHANGED => {<br /> "ansible_facts": {<br /> "discovered_interpreter_python": "/usr/bin/python"<br /> },<br /> "changed": true,<br /> "msg": "",<br /> "rc": 0,<br /> "results": [<br /> "Installed: mod_http2-1.11.3-3.module+el8.2.0+7758+84b4ca3e.1.ppc64le",<br /> "Installed: httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.ppc64le",<br /> "Installed: httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca404a3.noarch",<br /> "Installed: apr-util-1.6.1-6.el8.ppc64le",<br /> "Installed: apr-util-openssl-1.6.1-6.el8.ppc64le",<br /> "Installed: apr-1.6.3-9.el8.ppc64le",<br /> "Installed: redhat-logos-httpd-81.1-1.el8.noarch",<br /> "Installed: httpd-tools-2.4.37-21.module+el8.2.0+5008+cca404a3.ppc64le",<br /> "Installed: apr-util-bdb-1.6.1-6.el8.ppc64le"<br /> ]<br />}</span><br /></p><p>--------------------------------------------------------------<br /></p><p><span style="font-family: arial;"><i><b>How To Setup Ansible Master-Slave and Install Apache Web Server</b></i><br />Let’s see the capabilities of Ansible in this example of simple web server setup. We will have the following components:</span></p><ol style="text-align: left;"><li><span style="font-family: arial;">Control Node – It is the node that will have Ansible installed and it will control the other nodes.</span></li><li><span style="font-family: arial;">Load Balancer – A nginx based load balancer will be installed on this node.</span></li><li><span style="font-family: arial;">Web Server 1 and Server 2 – These nodes will have Apache installed with a simple hello world web page. The load balancer will alternate traffic between these two nodes.<br /></span></li></ol><p><span style="font-family: arial;">We will first install Ansible on the control node. Then, we will use the control node to set up the load balancer and application nodes.</span></p><p></p><p>--------------------------------------------------------------</p><p><i><b>How to create playbooks ? Example Hello world <br /></b></i></p><p><span style="font-size: x-small;"> [root@myhost123 example]# cat HelloWorld.yml<br />---<br />- name: This is a hello-world example<br /> hosts: all<br /> tasks:<br /> - name: Create a file called '/tmp/output.txt' with the content 'hello world'.<br /> copy:<br /> content: hello world<br /> dest: /tmp/output.txt</span></p><p><span style="font-size: x-small;">...<br />[root@myhost123 example]#<br />---------------------------------------------------------------</span></p><p><span style="font-size: x-small;">Run Playbook:<br /><br /><span style="font-size: xx-small;">[root@myhost123 example]# <span style="font-size: small;">ansible-playbook HelloWorld.yml</span><br />PLAY [This is a hello-world example] ********************************************************************<br />TASK [Gathering Facts] **********************************************************************************<br />ok: [myhost123]<br />TASK [Create a file called '/tmp/output.txt' with the content 'hello world'.] *************************<br />ok: [myhost123]<br />PLAY RECAP **********************************************************************************************<br />myhost123 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0<br />[root@myhost123 example]#</span></span></p><p><span style="font-size: x-small;">Verify output:<br /><br /><span style="font-size: xx-small;">[root@myhost123 example]# cat /tmp/output.txt<br />hello world</span></span><i><b><br /></b></i>----------------------------------------------------------------------------------</p><p><span style="font-family: arial;"> All YAML files (regardless of their association with Ansible or not) can optionally begin with --- and end with ... This is part of the YAML format and indicates the start and end of a document.</span></p><span style="font-size: xx-small;">Reference:</span><br /><span style="font-size: xx-small;">https://docs.ansible.com/index.html</span><br /><span style="font-size: xx-small;">https://linuxhint.com/ansible-tutorial-beginners/ </span><br />Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-34340495800989653072020-05-24T05:15:00.004+05:302020-07-31T16:20:03.879+05:30RHEL8 - Next generation of Linux container Capabilities - podman, buildah, skopeo .....!<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-8Na-VoegDYI/Xsmo6QKWLII/AAAAAAAAU2Y/EeenRTlj4g8v-qHdq6CSvNDU7d0JfcJwQCLcBGAsYHQ/s1600/podman_logo.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="310" data-original-width="701" height="141" src="https://1.bp.blogspot.com/-8Na-VoegDYI/Xsmo6QKWLII/AAAAAAAAU2Y/EeenRTlj4g8v-qHdq6CSvNDU7d0JfcJwQCLcBGAsYHQ/s320/podman_logo.PNG" width="320" /></a></div>
<div style="text-align: justify;">
Container technology is creating a lot of buzz in the recent times. As people move from virtualization to container technology, many enterprises have adopted software container cloud application deployment. Containers leverage some key capabilities available within Linux. Containers depend on key Linux kernel features such as control groups, namespaces, and SELinux in order to manage resources and isolate the applications that are running inside the containers. It’s not just containers that generally work best with Linux, but also the tools used to manage their lifecycles. Today, Kubernetes is the leading container orchestration platform, and it was built on Linux concepts and uses Linux tooling and application programming interfaces (APIs) to manage the containers.</div>
<br />
<div style="text-align: justify;">
Red Hat OpenShift is a leading hybrid cloud, enterprise Kubernetes application platform, trusted by 1,700+ organizations. It is much easier to use, and it even has a web interface for configuration. They developed container tools for single hosts and in clusters, standardizing on Kubernetes. Other alternative- popular managed Kubernetes service are AWS EKS(Amazon Elastic Kubernetes Service)/Fargate, Azure AKS, or Google Cloud Platform’s GKE, Apache Mesos, Docker Swarm, Nomad, OpenStack, Rancher, and Docker Compose.</div>
<br />
For RHEL 8, the Docker package is not included and not supported by Red Hat. The docker package has been replaced by the new suite of tools in the Container Tools module as listed <br />
<br />
<ul style="text-align: left;">
<li> The <b>podman</b> container engine replaced docker engine</li>
<li> The <b>buildah</b> utility replaced docker build</li>
<li> The <b>skopeo</b> utility replaced docker push </li>
</ul>
<br />
Red Hat Quay -A distributed, highly available container registry for entire enterprise. Unlike other container tools implementations, tools described here do not center around the monolithic Docker container engine and docker command. Instead, they provide a set of command-line tools that can operate without a container engine. These include:<br />
<br />
<ul style="text-align: left;">
<li><b>podman</b> - client tool for directly managing pods and container images (run, stop, start, ps, attach, exec, and so on)</li>
<li><b>buildah</b> - client tool for building, pushing and signing container images</li>
<li><b>skopeo</b> - client tool for copying, inspecting, deleting, and signing images</li>
<li><b>runc</b> - Container runtime client for providing container run and build features to podman and buildahwith OCI format containers </li>
<li><b>crictl</b> - For troubleshooting and working directly with CRI-O container engines</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: justify;">
Because these tools are compatible with the Open Container Initiative (OCI), they can be used to manage the same Linux containers that are produced and managed by Docker and other OCI-compatible container engines. However, they are especially suited to run directly on Red Hat Enterprise Linux, in single-node use cases. Each tool in this scenario can be more light-weight and focused on a subset of features. And with no need for a daemon process running to implement a container engine, these tools can run without the overhead of having to work with a daemon process.<br />
<br />
For a multi-node container platform, there is OpenShift. Instead of relying on the single-node, daemonless tools, OpenShift requires a daemon-based container engine like CRI-O Container Engine. . Also, podman stores its data in the same directory structure used by Buildah, Skopeo, and CRI-O, which will allow podman to eventually work with containers being actively managed by CRI-O in OpenShift. <br />
<br />
In a nutshell, you get Podman with RHEL in a single node use case (orchestrate yourself) and CRI-O as part of the highly automated OpenShift 4 software stack as shown in diagram.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-LyrZmb_rFko/XuNQcakoUZI/AAAAAAAAU6Q/JhnY_Wf3jm0JkxI4Q3zGKUV7eyqs2UdoACLcBGAsYHQ/s1600/cloudNative-Buildah-openshift.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="482" data-original-width="1511" height="204" src="https://1.bp.blogspot.com/-LyrZmb_rFko/XuNQcakoUZI/AAAAAAAAU6Q/JhnY_Wf3jm0JkxI4Q3zGKUV7eyqs2UdoACLcBGAsYHQ/s640/cloudNative-Buildah-openshift.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://crunchtools.com/podman-and-cri-o-in-rhel-8-and-openshift-4/">source</a></td></tr>
</tbody></table>
<br />
<b>What is CRI-O? </b><br />
CRI-O is an implementation of the Kubernetes CRI (Container Runtime Interface) to enable using OCI (Open Container Initiative) compatible runtimes. It is a lightweight alternative to using Docker as the runtime for kubernetes. It allows Kubernetes to use any OCI-compliant runtime as the container runtime for running pods. Today it supports runc and Kata Containers as the container runtimes but any OCI-conformant runtime can be plugged in principle. CRI-O supports OCI container images and can pull from any container registry. It is a lightweight alternative to using Docker, Moby or rkt as the runtime for Kubernetes. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Why CRI-O ? </b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-zGCOcCS9f0Y/XsmqA1Q9ZQI/AAAAAAAAU2k/bz_OXlFu8GYcuBe_S3uRYCFKblEYG_jPACPcBGAYYCw/s1600/criO-logo.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="217" data-original-width="573" height="75" src="https://1.bp.blogspot.com/-zGCOcCS9f0Y/XsmqA1Q9ZQI/AAAAAAAAU2k/bz_OXlFu8GYcuBe_S3uRYCFKblEYG_jPACPcBGAYYCw/s200/criO-logo.PNG" width="200" /></a></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
CRI-O is an open source, community-driven container engine. Its primary goal is to replace the Docker service as the container engine for Kubernetes implementations, such as OpenShift Container Platform. The CRI-O container engine provides a stable, more secure, and performant platform for running Open Container Initiative (OCI) compatible runtimes. You can use the CRI-O container engine to launch containers and pods by engaging OCI-compliant runtimes like runc [the default OCI runtime] or Kata Containers.</div>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-UjNqZhCQ6OI/XsmWPefZwaI/AAAAAAAAU10/PLH_J8mFwykYbk0M-1SYqQsY8yUpVWD1ACLcBGAsYHQ/s1600/runtime_CRI-types.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="413" data-original-width="1169" height="226" src="https://1.bp.blogspot.com/-UjNqZhCQ6OI/XsmWPefZwaI/AAAAAAAAU10/PLH_J8mFwykYbk0M-1SYqQsY8yUpVWD1ACLcBGAsYHQ/s640/runtime_CRI-types.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>CRI RUNTIMES</b> <a href="https://events19.linuxfoundation.org/wp-content/uploads/2018/07/OSLS_-Container-runtimes-and-standards.pdf"><span style="font-size: xx-small;">source</span></a></td></tr>
</tbody></table>
<div style="text-align: justify;">
CRI-O is not supported as a stand-alone container engine. You must use CRI-O as a container engine for a Kubernetes installation, such as OpenShift Container Platform. To run containers without Kubernetes or OpenShift Container Platform, use podman. CRI-O’s purpose is to be the container engine that implements the Kubernetes Container Runtime Interface (CRI) for OpenShift Container Platform and Kubernetes, replacing the Docker service. The scope of CRI-O is tied to the Container Runtime Interface (CRI). CRI extracted and standardized exactly what a Kubernetes service (kubelet) needed from its container engine. There is little need for direct command-line contact with CRI-O. A set of container-related command-line tools are available to provide full access to CRI-O for testing and monitoring - crictl, runc, podman, buildah, skopeo. Some Docker features are included in other tools instead of in CRI-O. For example, podman offers exact command-line compatibility with many docker command features and extends those features to managing pods as well. No container engine is needed to run containers or pods with podman. Features for building, pushing, and signing container images, which are also not required in a container engine, are available in the buildah command.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://computingforgeeks.com/docker-vs-cri-o-vs-containerd/" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="494" data-original-width="781" height="404" src="https://1.bp.blogspot.com/-VDuIxmZn-Xw/Xsw_1dJ9yTI/AAAAAAAAU3g/G5ABm9K4fCMtSmPy84_4wcrec8K0XLyLwCLcBGAsYHQ/s640/kubernetes_CRIO_process.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://computingforgeeks.com/docker-vs-cri-o-vs-containerd/">Kubernetes and CRI-O process</a></td></tr>
</tbody></table>
The following are the components of CRI-O :<br />
<ul>
<li>OCI compatible runtime – Default is runC, other OCI compliant are supported as well e.g Kata Containers.</li>
<li>containers/storage – Library used for managing layers and creating root file-systems for the containers in a pod.</li>
<li>containers/image – Library is used for pulling images from registries.</li>
<li>networking (CNI) – Used for setting up networking for the pods. Flannel, Weave and OpenShift-SDN CNI plugins have been tested.</li>
<li>container monitoring (conmon) – Utility within CRI-O that is used to monitor the containers.</li>
<li>security is provided by several core Linux capabilities </li>
</ul>
</div>
<div style="text-align: justify;">
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-dtUPb_ESejo/XyPwH5ZhN8I/AAAAAAAAVFQ/hhxxXHZuZMMzjClWJRIYKgGeS_-n1ZfDwCLcBGAsYHQ/s1600/Runtime_kubernetes.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="471" data-original-width="1039" height="290" src="https://1.bp.blogspot.com/-dtUPb_ESejo/XyPwH5ZhN8I/AAAAAAAAVFQ/hhxxXHZuZMMzjClWJRIYKgGeS_-n1ZfDwCLcBGAsYHQ/s640/Runtime_kubernetes.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b><a href="https://events19.linuxfoundation.org/wp-content/uploads/2017/11/How-Container-Runtime-Matters-in-Kubernetes_-OSS-Kunal-Kushwaha.pdf">Runtime in Kubernetes </a></b></td></tr>
</tbody></table>
where : OCI runtime works as low-level runtime<br />
High-level runtime provides inputs to OCI runtime as per OCI specs<br />
<br />
How do Podman, CRI-O and Kata Containers relate to this ecosystem?<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-nX7REwQECnc/XyPxkgTAebI/AAAAAAAAVFc/DSALl8Rg7SU2N4T8V0a3z9gh9hvQhJi6wCLcBGAsYHQ/s1600/CRIO_runtime.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="425" data-original-width="1036" height="262" src="https://1.bp.blogspot.com/-nX7REwQECnc/XyPxkgTAebI/AAAAAAAAVFc/DSALl8Rg7SU2N4T8V0a3z9gh9hvQhJi6wCLcBGAsYHQ/s640/CRIO_runtime.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://merlijn.sebrechts.be/blog/2020-01-docker-podman-kata-cri-o/">source</a></td></tr>
</tbody></table>
An OCI runtime is relatively simple. You give it the root filesystem of the container and a json file describing core properties of the container, and the runtime spins up the container and connects it to an existing network using a pre-start hook.<br />
<br />
Listed actions below are the job of a high-level container runtime. On top of this, the high-level container runtime implements the CRI so that Kubernetes has an easy way to drive the runtime.<br />
<ul>
<li> Actually creating the network of a container.</li>
<li> Managing container images.</li>
<li> Preparing the environment of a container.</li>
<li> Managing local/persistent storage.</li>
</ul>
runc is the default for most tools such as Docker and Podman.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-DdRMRBKjqEc/XyP2XKeM3NI/AAAAAAAAVFo/FbwGeWTuGAg83HhJk6q5U_MTTzudV4s7wCLcBGAsYHQ/s1600/CRIO-workflow.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="732" data-original-width="835" height="560" src="https://1.bp.blogspot.com/-DdRMRBKjqEc/XyP2XKeM3NI/AAAAAAAAVFo/FbwGeWTuGAg83HhJk6q5U_MTTzudV4s7wCLcBGAsYHQ/s640/CRIO-workflow.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://www.redhat.com/en/blog/introducing-cri-o-10">source</a></td></tr>
</tbody></table>
<b>What CRI-O isn’t:</b><br />
<br />
Building images, for example, is out of scope for CRI-O and that’s left to tools like Docker’s build command, Buildah, or OpenShift’s Source-to-Image (S2I). Once an image is built, CRI-O will happily consume it, but the building of images is left to other tools. <br />
</div>
<div style="text-align: justify;">
<b>What is Podman? </b></div>
<div style="text-align: justify;">
Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System developed by Red Hat where engineers have paid special attention to using the same nomenclature when executing Podman commands. Containers can either be run as root or in rootless mode. It's a replacement for Docker for local development of containerized applications. Podman commands map 1 to 1 to Docker commands, including their arguments. You could alias docker with podman and never notice that there is a completely different tool managing your local containers.The Podman approach is simply to directly interact with the image registry, with the container and image storage, and with the Linux kernel through the runC container runtime process (not a daemon). Podman allows you to do all of the Docker commands without the daemon dependency. <br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-73mj0t4jpXE/XsnBJn9Oc4I/AAAAAAAAU3A/mG6BNw__uocf9qCryi12dcoEMpqknqQEwCLcBGAsYHQ/s1600/podman_arch.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="803" data-original-width="948" height="542" src="https://1.bp.blogspot.com/-73mj0t4jpXE/XsnBJn9Oc4I/AAAAAAAAU3A/mG6BNw__uocf9qCryi12dcoEMpqknqQEwCLcBGAsYHQ/s640/podman_arch.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: large;">Podman workflow</span><b><br /></b></td></tr>
</tbody></table>
<br />
One of the core features of Podman is it's focus on security. There is no daemon involved in using Podman. It uses traditional fork-exec model instead and as well heavily utilizes user namespaces and network namespaces. As a result, Podman is a bit more isolated and in general more secure to use than Docker. You can even be root in a container without granting container or Podman any root privileges on the host -- and user in a container won't be able to do any root-level tasks on the host machine.Running rootless Podman and Buildah can do most things people want to do with containers, but there are times when root is still required. The nicest feature is running Podman and containers as a non-root user. This means you never have give a user root privileges on the host, while in the client/server model (like Docker employs), you must open a socket to a privileged daemon running as root to launch the containers. There you are at the mercy of the security mechanisms implemented in the daemon versus the security mechanisms implemented in the host operating systems—a dangerous proposition.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<i>How containers run with container Engine ?</i></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-hvpVAm-5ZB8/Xsmum_DYTUI/AAAAAAAAU2s/Im_n0FGhwyQm9s98hYs-0h-Uk24MKmW1gCLcBGAsYHQ/s1600/How_container_run_with_Engine.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="892" height="344" src="https://1.bp.blogspot.com/-hvpVAm-5ZB8/Xsmum_DYTUI/AAAAAAAAU2s/Im_n0FGhwyQm9s98hYs-0h-Uk24MKmW1gCLcBGAsYHQ/s640/How_container_run_with_Engine.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://www.redhat.com/en/blog/why-red-hat-investing-cri-o-and-podman">source</a></td></tr>
</tbody></table>
<div style="text-align: justify;">
<b>Podman can now ease the transition to Kubernetes and CRI-O :</b></div>
<div style="text-align: justify;">
<b> </b> </div>
<div style="text-align: justify;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: justify;">
On a basic level, Kubernetes is often viewed as the application that
runs your containers, but Kubernetes really is a huge bundle of
utilities or APIs that explain how a group of microservices running in
containers on a group of servers can coordinate and work together and
share services and resources. Kubernetes only supplies the APIs for
orchestration and scheduling, and resource management. To have a
complete container orchestration platform, you’ll need the OS
underneath, a container registry, container networking, container
storage, logging and monitoring, and a way to integrate continuous
integration/continuous delivery (CI/CD). Red Hat OpenShift, a supported
Kubernetes for cloud-native applications with enterprise security on
multi-cloud environment.</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
A group of seals is called a pod :)- Padman manage pods. The Pod concept was introduced by Kubernetes. Podman pods are similar to the Kubernetes definition. Podman can now capture the YAML description of local pods and containers and then help users transition to a more sophisticated orchestration environment like Kubernetes. Check this developer and user workflow:</div>
<ul style="text-align: left;">
<li>Create containers/pods locally using Podman on the command line.</li>
<li>Verify these containers/pods locally or in a localized container runtime (on a different physical machine).</li>
<li>Snapshot the container and pod descriptions using Podman and help users re-create them in Kubernetes.</li>
<li>Users
add sophistication and orchestration (where Podman cannot) to the
snapshot descriptions and leverage advanced functions of Kubernetes.</li>
</ul>
<div style="text-align: justify;">
<i>How containers run in kubernetes cluster?</i></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
This container stack within Red Hat Enterprise Linux and Red Hat Enterprise Linux CoreOS serves as part of the foundation for OpenShift. As can be seen in the drawing below, the CRI-O stack in OpenShift shares many of its underlying components with Podman. This allows Red Hat engineers to leverage knowledge gained in experiments conducted in Podman for new capabilities in OpenShift.</div>
<div style="text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-oSxMhVJPgCw/XsmvCojf0xI/AAAAAAAAU20/5nHejmTrjbUYbFKaWlM43KIvXlP3d3PWACLcBGAsYHQ/s1600/how-container-run_kebenetes.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="395" data-original-width="964" height="262" src="https://1.bp.blogspot.com/-oSxMhVJPgCw/XsmvCojf0xI/AAAAAAAAU20/5nHejmTrjbUYbFKaWlM43KIvXlP3d3PWACLcBGAsYHQ/s640/how-container-run_kebenetes.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://www.redhat.com/en/blog/why-red-hat-investing-cri-o-and-podman">source</a></td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-jykMrQ6nurA/XsmmJgDqPNI/AAAAAAAAU2M/njYJf6AS_oUaTse2t7uzeyMy6EhkX9TjwCLcBGAsYHQ/s1600/pod_architecture_podman.PNG" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="647" data-original-width="1066" height="388" src="https://1.bp.blogspot.com/-jykMrQ6nurA/XsmmJgDqPNI/AAAAAAAAU2M/njYJf6AS_oUaTse2t7uzeyMy6EhkX9TjwCLcBGAsYHQ/s640/pod_architecture_podman.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><b>Pod-Architecture</b> <a href="https://developers.redhat.com/blog/2019/01/15/podman-managing-containers-pods/"><span style="font-size: xx-small;">source</span></a></td></tr>
</tbody></table>
<div style="text-align: justify;">
<br />
Every Podman pod includes an “infra” container. This container does nothing, but go to sleep. Its purpose is to hold the namespaces associated with the pod and allow podman to connect other containers to the pod. This allows you to start and stop containers within the POD and the pod will stay running, where as if the primary container controlled the pod, this would not be possible. Most of the attributes that make up the Pod are actually assigned to the “infra” container. Port bindings, cgroup-parent values, and kernel namespaces are all assigned to the “infra” container. This is critical to understand, because once the pod is created these attributes are assigned to the “infra” container and cannot be changed. </div>
<div style="text-align: justify;">
<br />
In the above diagram, notice the box above each container, conmon, this is the container monitor. It is a small C Program that’s job is to watch the primary process of the container, and if the container dies, save the exit code. It also holds open the tty of the container, so that it can be attached to later. This is what allows podman to run in detached mode (backgrounded), so podman can exit but conmon continues to run. Each container has its own instance of conmon.<br />
<br />
<br />
<b>Buildah :</b> The buildah command allows you to build container images either from command line or using Dockerfiles. These images can then be pushed to any container registry and can be used by any container engine, including Podman, CRI-O, and Docker. Buildah specializes in building OCI images. Buildah’s commands replicate
all of the commands that are found in a Dockerfile. Buildah’s goal is
also to provide a lower level coreutils interface to build container
images, allowing people to build containers without requiring a
Dockerfile. Buildah’s other goal is to allow you to use other scripting
languages to build container images without requiring a daemon. The buildah command can be used as a separate command, but is incorporated into other tools as well. For example the podman build command used buildah code to build container images. Buildah is also often used to securely build containers while running inside of a locked down container by a tool like Podman, OpenShift/Kubernetes or Docker. Buildah allows you to have a Kubernetes cluster without any Docker daemon for both runtime and builds. So, When to use Buildah and when to use Podman. With Podman you can run, build (it calls Buildah under the covers for this), modify and troubleshoot containers in your Kubernetes cluster. With the two projects together, you have a well rounded solution for your OCI container image and container needs. Buildah and Podman are easily installable via <i>yum install buildah podman</i>. <br />
<br />
A quick and easy way to summarize the difference between the two projects is the buildah run command emulates the RUN command in a Dockerfile while the podman run command emulates the docker run command in functionality. Buildah is an efficient way to create OCI images while Podman allows you to manage and maintain those images and containers in a production environment using familiar container CLI commands. Together they form a strong foundation to support your OCI container image and container needs.<br />
<br />
<b>skopeo</b>: The skopeo command is a tool for copying containers and images between different types of container storage. It can copy containers from one container registry to another. It can copy images to and from a host, as well as to other container environments and registries. Skopeo can inspect images from container image registries, get images and image layers, and use signatures to create and verify images. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<b>Running containers as root or rootless :</b></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Running
the container tools such as podman, skopeo, or buildah as a user with
superuser privilege (root user) is the best way to ensure that your
containers have full access to any feature available on your system.
However, with the feature called "Rootless Containers," generally
available as of RHEL 8.1, you can work with containers as a regular
user.</div>
<br />
<div style="text-align: justify;">
Although container engines, such as Docker, let you run
docker commands as a regular (non-root) user, the docker daemon that
carries out those requests runs as root. So, effectively, regular users
can make requests through their containers that harm the system, without
there being clarity about who made those requests. By setting up
rootless container users, system administrators limit potentially
damaging container activities from regular users, while still allowing
those users to safely run many container features under their own
accounts.</div>
Also, note that Docker is a daemon-based container engine which allows us to deploy applications inside containers as shown in diagram docker-workflow. With the release of RHEL 8 and CentOS 8, docker package has been removed from their default package repositories, docker has been replaced with podman and buildah. If you are comfortable with docker and deploy most the applications inside the docker containers and does not want to switch to podman then there is a way to install and use community version of docker on CentOS 8 and RHEL 8 system by using the official Docker repository for CentOS7/RHEL7, which is a compatible clone. <br />
<div style="text-align: justify;">
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-RWOSQjIugq0/Xsvgen-DUkI/AAAAAAAAU3U/GzUiOVjoiHA_vhzpuoXeCdZ9pMV0BDpIQCLcBGAsYHQ/s1600/docker_arch.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="656" data-original-width="1179" height="356" src="https://1.bp.blogspot.com/-RWOSQjIugq0/Xsvgen-DUkI/AAAAAAAAU3U/GzUiOVjoiHA_vhzpuoXeCdZ9pMV0BDpIQCLcBGAsYHQ/s640/docker_arch.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span style="font-size: large;">Docker workflow</span></td></tr>
</tbody></table>
<br />
<b>NOTE: </b>Technology Preview features provide early access to upcoming product innovations, enabling you to test functionality and provide feedback during the development process. RHEL 8.2 provides access to technology previews of containerized versions of Buildah, a tool for building container images that comply with the Open Container Image (OCI) specification, and Skopeo, a tool that facilitates the movement of container images. Red Hat is adding Udica, a tool that makes it easier to create customized, container-centric SELinux security policies that reduce the risk that a process might “break out” of a container. RHEL 8.2 also introduces enhancements to the Red Hat Universal Base Image, which now supports OpenJDK and .NET 3.0, in addition to making it easier to access source code associated with a given image via a single command. That adds additional management and monitoring capabilities via updates to Red Hat Insights, which is provided to make it easier to define and monitor policies created by the IT organization, as well as reduce any drift from baselines initially defined by the IT team.<br />
----------------------------------------------------------------------------------------------------------------------------------<br />
<u><b>Podman installation on RHEL and small demo to illustrate with DB application:</b></u><br />
<b>Step 1:</b> <b><span style="font-family: "georgia" , "times new roman" , serif;"><i>yum -y install podman</i></span></b><br />
This command will install Podman and also its dependencies: atomic-registries, runC, skopeo-containers, and SELinux policies. Check this as shown below :<br />
<i>[root@IBMPOWER_sachin]# rpm -qa | grep podman<br />podman-1.6.4-18.el7_8.x86_64<br />[root@IBMPOWER_sachin]# rpm -qa | grep skopeo<br />skopeo-0.1.40-7.el7_8.x86_64<br />[root@IBMPOWER_sachin]# rpm -qa | grep runc<br />runc-1.0.0-67.rc10.el7_8.x86_64<br />[root@IBMPOWER_sachin]# </i><br />
<b>Step 2 :</b> Command-line examples to create container and run RHEL container <br />
<i>[root@IBMPOWER_sachin script]# <b><span style="font-family: "georgia" , "times new roman" , serif;">podman run -it rhel sh</span></b><br />Trying to pull registry.access.redhat.com/rhel...<br />Getting image source signatures<br />Copying blob feaa73091cc9 done<br />Copying blob e20f387c7bf5 done<br />Copying config 1a9b6d0a58 done<br />Writing manifest to image destination<br />Storing signatures<br />sh-4.2#</i><br />
<i>[root@IBMPOWER_sachin ~]# <span style="font-family: "georgia" , "times new roman" , serif;"><b>podman images</b></span><br />REPOSITORY TAG IMAGE ID CREATED SIZE<br />registry.access.redhat.com/rhel latest 1a9b6d0a58f8 2 weeks ago 215 MB<br />[root@IBMPOWER_sachin ~]#</i><br />
<b>Step 3 :</b> Install a containerized service for setting up a MariaDB database :<br />
Run a MariaDB persistent container - MariaDB 10.2 with some custom variables and try to let its “data” be persistent.<br />
<i>[root@IBMPOWER_sachin~]#<b><span style="font-family: "georgia" , "times new roman" , serif;"> </span></b></i><br />
<i><b><span style="font-family: "georgia" , "times new roman" , serif;">podman pull registry.access.redhat.com/rhscl/mariadb-102-rhel7</span></b></i><br />
<i>Trying to pull registry.access.redhat.com/rhscl/mariadb-102-rhel7...</i><br />
<i>Getting image source signatures</i><br />
<i>Copying blob 8574a8f8c7e5 done</i><br />
<i>Copying blob f60299098adf done</i><br />
<i>Copying blob 82a8f4ea76cb done</i><br />
<i>Copying blob a3ac36470b00 done</i><br />
<i>Copying config 66a314da15 done</i><br />
<i>Writing manifest to image destination</i><br />
<i>Storing signatures</i><br />
<i>66a314da15d608d89f7b589f6668f9bc0c2fa814ec9c690481a7a057206338bd</i><br />
<i>[root@IBMPOWER_sachin ~]#</i><br />
<i>[root@IBMPOWER_sachin ~]# <b><span style="font-family: "georgia" , "times new roman" , serif;">podman images</span></b></i><br />
<i>REPOSITORY TAG IMAGE ID CREATED SIZE</i><br />
<i>registry.access.redhat.com/rhscl/mariadb-102-rhel7 latest 66a314da15d6 11 days ago 453 MB</i><br />
<i>registry.access.redhat.com/rhel latest 1a9b6d0a58f8 2 weeks ago 215 MB</i><br />
<i>[root@IBMPOWER_sachin ~]#</i><br />
<br />
After you pull an image to your local system and before you run it, it is a good idea to investigate that image. Reasons for investigating an image before you run it include:<br />
<ul>
<li> Understanding what the image does</li>
<li> Checking what software is inside the image </li>
</ul>
Example: Get information about the “user ID running inside the container”, "ExposedPorts" and the “persistent volume location to attach“ ....etc as shown here:<i><b><br />podman inspect registry.access.redhat.com/rhscl/mariadb-102-rhel7 | grep User<br />podman inspect registry.access.redhat.com/rhscl/mariadb-102-rhel7 | grep -A1 ExposedPorts<br />podman inspect registry.access.redhat.com/rhscl/mariadb-102-rhel7 | grep -A1 Volume</b><br /> </i><br />
<b>Step 4 : </b>Set up a folder that will handle MariaDB’s data once we start our container:<br />
<i>[root@IBMPOWER_sachin ~]# <b>mkdir /root/mysql-data</b></i><br />
<i>[root@IBMPOWER_sachin ~]# <b>chown 27:27 /root/mysql-data</b></i><br />
<b>Step 5: </b>Run the container<br />
<i>[root@IBMPOWER_sachin ~]# <b><span style="font-family: "georgia" , "times new roman" , serif;"> </span></b></i><br />
<div style="text-align: justify;">
<i><b><span style="font-family: "georgia" , "times new roman" , serif;">podman run -d -v /root/mysql-data:/var/lib/mysql/data:Z -e MYSQL_USER=user -e MYSQL_PASSWORD=pass -e MYSQL_DATABASE=db -p 3306:3306 registry.access.redhat.com/rhscl/mariadb-102-rhel7</span></b></i></div>
<i>fd2d30f8ec72734a2eee100f89f35574739c7a6a30281be77998de466635b3b0</i><br />
<i>[root@IBMPOWER_sachin ~]# podman container list</i><br />
<i>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</i><br />
<i>fd2d30f8ec72 registry.access.redhat.com/rhscl/mariadb-102-rhel7:latest run-mysqld 9 seconds ago Up 9 seconds ago 0.0.0.0:3306->3306/tcp wizardly_jang</i><br />
<i>[root@IBMPOWER_sachin ~]# </i><br />
<i><b>Step 6:</b> check logs</i><br />
<i>[root@ ]# <b><span style="font-family: "georgia" , "times new roman" , serif;">podman logs fd2d30f8ec72 | head</span></b></i><br />
<i>=> sourcing 20-validate-variables.sh ...</i><br />
<i>=> sourcing 25-validate-replication-variables.sh ...</i><br />
<i>=> sourcing 30-base-config.sh ...</i><br />
<i>---> 11:03:27 Processing basic MySQL configuration files ...</i><br />
<i>=> sourcing 60-replication-config.sh ...</i><br />
<i>=> sourcing 70-s2i-config.sh ...</i><br />
<i>---> 11:03:27 Processing additional arbitrary MySQL configuration provided by s2i ...</i><br />
<i>=> sourcing 40-paas.cnf ...</i><br />
<i>=> sourcing 50-my-tuning.cnf ...</i><br />
<i>---> 11:03:27 Initializing database ...</i><br />
<b>Step 7: </b>That started and initialized its database . Lets create some table and check <br />
<i>[root@IBMPOWER_sachin ~]# <b><span style="font-family: "georgia" , "times new roman" , serif;">podman exec -it fd2d30f8ec72 /bin/bash</span></b></i><br />
<i>bash-4.2$ <b>mysql --user=user --password=pass -h 127.0.0.1 -P 3306 -t</b></i><br />
<i>Welcome to the MariaDB monitor. Commands end with ; or \g.</i><br />
<i>Your MariaDB connection id is 8</i><br />
<i>Server version: 10.2.22-MariaDB MariaDB Server</i>
<i>Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.</i>
<i>Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.</i>
<i>MariaDB [(none)]> show databases;</i><br />
<i>+--------------------+</i><br />
<i>| Database |</i><br />
<i>+--------------------+</i><br />
<i>| db |</i><br />
<i>| information_schema |</i><br />
<i>| test |</i><br />
<i>+--------------------+</i><br />
<i>3 rows in set (0.00 sec)</i>
<i>MariaDB [(none)]> use test;</i><br />
<i>Database changed</i><br />
<i>MariaDB [test]> show tables;</i><br />
<i>Empty set (0.00 sec)</i>
<i>MariaDB [test]> CREATE TABLE hpc_team (username VARCHAR(20), date DATETIME);</i><br />
<i>Query OK, 0 rows affected (0.00 sec)</i>
<i>MariaDB [test]> INSERT INTO hpc_team (username, date) VALUES ('Aboorva', Now());</i><br />
<i>Query OK, 1 row affected (0.00 sec)</i>
<i>MariaDB [test]> INSERT INTO hpc_team (username, date) VALUES ('Nysal', Now());</i><br />
<i>Query OK, 1 row affected (0.00 sec)</i>
<i>MariaDB [test]> INSERT INTO hpc_team (username, date) VALUES ('Sachin', Now());</i><br />
<i>Query OK, 1 row affected (0.00 sec)</i>
<i>MariaDB [test]> select * from hpc_team;</i><br />
<i>+----------+---------------------+</i><br />
<i>| username | date |</i><br />
<i>+----------+---------------------+</i><br />
<i>| Aboorva | 2020-05-26 11:12:41 |</i><br />
<i>| Nysal | 2020-05-26 11:12:55 |</i><br />
<i>| Sachin | 2020-05-26 11:13:08 |</i><br />
<i>+----------+---------------------+</i><br />
<i>3 rows in set (0.00 sec)</i>
<i>MariaDB [test]> quit</i><br />
<i>Bye</i><br />
<i>bash-4.2$</i><br />
<i>bash-4.2$ ls</i><br />
<i>aria_log.00000001 db ib_buffer_pool ib_logfile1 ibtmp1 mysql performance_schema test</i><br />
<i>aria_log_control fd2d30f8ec72.pid ib_logfile0 ibdata1 multi-master.info mysql_upgrade_info tc.log</i><br />
<i>bash-4.2$ cd test/</i><br />
<i>bash-4.2$ ls -alsrt</i><br />
<i>total 108</i><br />
<i> 4 drwxr-xr-x 6 mysql mysql 4096 May 26 11:03 ..</i><br />
<i> 4 -rw-rw---- 1 mysql mysql 483 May 26 11:12 hpc_team.frm</i><br />
<i> 4 drwx------ 2 mysql mysql 4096 May 26 11:12 .</i><br />
<i>96 -rw-rw---- 1 mysql mysql 98304 May 26 11:13 hpc_team.ibd</i><br />
<i>bash-4.2$</i><br />
<b>Step 8:</b> Check DB folder from host machine :<br />
<i>[root@IBMPOWER_sachin mysql-data]# cd test/</i><br />
<i>[root@IBMPOWER_sachin test]# ls -alsrt</i><br />
<i>total 108</i><br />
<i> 4 drwxr-xr-x 6 27 27 4096 May 26 07:03 ..</i><br />
<i> 4 -rw-rw---- 1 27 27 483 May 26 07:12 hpc_team.frm</i><br />
<i> 4 drwx------ 2 27 27 4096 May 26 07:12 .</i><br />
<i>96 -rw-rw---- 1 27 27 98304 May 26 07:13 hpc_team.ibd</i><br />
<i>[root@IBMPOWER_sachin test]#</i><br />
<br />
<b>Step 9: </b>We can set up our systemd unit file for handling the database. We’ll use a unit file as shown below:<i><br /></i><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-size: x-small;">cat <b>/etc/systemd/system/mariadb-service.service</b><br /><b>[Unit]<br />Description=Custom MariaDB Podman Container<br />After=network.target<br />[Service]<br />Type=simple<br />TimeoutStartSec=5m<br />ExecStartPre=-/usr/bin/podman rm "mariadb-service"<br />ExecStart=/usr/bin/podman run --name mariadb-service -v /root/mysql-data:/var/lib/mysql/data:Z -e MYSQL_USER=user -e MYSQL_PASSWORD=pass -e MYSQL_DATABASE=db -p 3306:3306 --net host registry.access.redhat.com/rhscl/mariadb-102-rhel7<br />ExecReload=-/usr/bin/podman stop "mariadb-service"<br />ExecReload=-/usr/bin/podman rm "mariadb-service"<br />ExecStop=-/usr/bin/podman stop "mariadb-service"<br />Restart=always<br />RestartSec=30<br />[Install]<br />WantedBy=multi-user.target</b></span></span><br />
<a name='more'></a><b><span style="font-size: x-small;"><span style="font-family: "georgia" , "times new roman" , serif;">where:</span></span></b><br />
<b><span style="font-size: x-small;">1) /usr/bin/podman run --name mariadb-service says we want to run a container that will be named mariadb-service.</span></b><br />
<b><span style="font-size: x-small;">2) –v /root/mysql-data:/var/lib/mysql/data:Z says we want to map the just-created data directory to the one inside the container. The Z option informs Podman to map correctly the SELinux context for avoiding permissions issues.</span></b><br />
<b><span style="font-size: x-small;">3) –e MYSQL_USER=user -e MYSQL_PASSWORD=pass -e MYSQL_DATABASE=db identifies the additional environment variables to use with our MariaDB container. We’re defining the username, the password, and the database name to use.</span></b><br />
<b><span style="font-size: x-small;">4) --net host maps the container’s network to the RHEL host.</span></b><br />
<b><span style="font-size: x-small;">5) registry.access.redhat.com/rhscl/mariadb-102-rhel7 specifies the container image to use.</span></b><br />
<ul>
</ul>
<b>Step 10:</b> reload the systemd catalog and start the service:<i><br /></i>[root@IBMPOWER_sachin system]#<i> </i><span style="font-family: "georgia" , "times new roman" , serif;"><b>podman ps</b></span><i><br />CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</i>[root@IBMPOWER_sachin system]#<i> </i><span style="font-family: "georgia" , "times new roman" , serif;"><b>systemctl daemon-reload</b></span><i><br /></i>[root@IBMPOWER_sachin system]#<i> </i><span style="font-family: "georgia" , "times new roman" , serif;"><b>systemctl start mariadb-service</b></span><i><br /></i>[root@IBMPOWER_sachin system]#<i> </i><span style="font-family: "georgia" , "times new roman" , serif;"><b>systemctl status mariadb-service</b></span><i><br /><b><span style="font-size: xx-small;">● mariadb-service.service - Custom MariaDB Podman Container<br /> Loaded: loaded (/etc/systemd/system/mariadb-service.service; disabled; vendor preset: disabled)<br /> Active: active (running) since Wed 2020-05-27 06:42:31 EDT; 10s ago<br /> Process: 7405 ExecStartPre=/usr/bin/podman rm mariadb-service (code=exited, status=0/SUCCESS)<br /> Main PID: 7434 (podman)<br /> CGroup: /system.slice/mariadb-service.service<br /> └─7434 /usr/bin/podman run --name mariadb-service -v /root/mysql-data:/var/lib/mysql/data:Z -e MYSQL_USER=user -e MYSQL_PASSWORD=pass -e MYSQL_DATABASE=db -p ...<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527454361344 [Note] InnoDB: Buffer pool(s) load completed at 200527 10:42:33<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527961991360 [Note] Plugin 'FEEDBACK' is disabled.<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527961991360 [Note] Server socket created on IP: '::'.<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527961991360 [Warning] 'user' entry 'root@fd2d30f8ec72' ignored in --skip-name-resolve mode.<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527961991360 [Warning] 'user' entry '@fd2d30f8ec72' ignored in --skip-name-resolve mode.<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527961991360 [Warning] 'proxies_priv' entry '@% root@fd2d30f8ec72' ignored in --skip-name-resolve mode.<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527961991360 [Note] Reading of all Master_info entries succeded<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527961991360 [Note] Added new Master_info '' to hash table<br />May 27 06:42:33 IBMPOWER_sachin podman[7434]: 2020-05-27 10:42:33 140527961991360 [Note] /opt/rh/rh-mariadb102/root/usr/libexec/mysqld: ready for connections.May 27 06:42:33 IBMPOWER_sachin podman[7434]: Version: '10.2.22-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server </span></b></i><br />
<b><span style="font-size: x-small;">[root@IBMPOWER_sachin system]#<span style="font-family: "georgia" , "times new roman" , serif;"> podman ps</span><i><br />CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES<br />364b98ec614b registry.access.redhat.com/rhscl/mariadb-102-rhel7:latest run-mysqld 24 seconds ago Up 24 seconds ago mariadb-service</i></span></b><br />
<b><span style="font-size: x-small;">[root@IBMPOWER_sachin system]#<i> </i><span style="font-family: "georgia" , "times new roman" , serif;">systemctl stop mariadb-service</span><i><br />[root@IBMPOWER_sachin system]# </i><span style="font-family: "georgia" , "times new roman" , serif;"><i> </i>podman ps</span><i><br />CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</i></span></b><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: small;">NOTE: You check the running instance of container in step 10 . There was no container listed. Then you start container with systemd -systemctl tool and check the status [podman ps]. It should list the running instance of the container as per unit file configuration. Then you stop the container and check the status [podman ps] as shown above.</span><br />
<i>------------------------------------------------------------------------------------------------------------------------------------</i></div>
<b>Finally</b>, List of upstream projects Red Hat participates to bring Innovation:<br />
<ul style="text-align: left;">
<li>Open Container Initiative : https://github.com/opencontainers</li>
<li>podman : https://github.com/containers/libpod</li>
<li>buildah : https://github.com/containers/buildah</li>
<li>skopeo : https://github.com/containers/skopeo</li>
<li>cri-o: https://github.com/cri-o/cri-o</li>
<li>runc : https://github.com/opencontainers/runc</li>
<li>containers/storage : https://github.com/containers/storage</li>
<li>containers/image : https://github.com/containers/image</li>
<li>Linux Kernel: https://www.kernel.org/</li>
<li>SELinux : https://selinuxproject.org/page/Main_Page</li>
<li>containers/udica : https://github.com/containers/udica</li>
<li>CRIU : https://www.criu.org/Main_Page</li>
<li>OpenSCAP : https://www.open-scap.org/</li>
<li>OKD : https://www.okd.io/</li>
</ul>
<h3 style="text-align: left;">
<span style="font-family: "georgia" , "times new roman" , serif; font-size: large;"><i>Conclusion : </i></span></h3>
<div style="text-align: justify;">
Investing in these technologies and others is strategic because it gives OpenShift the ability to add new capabilities like rootless containers, rootless builds, custom SELinux policies, User Namespaces, etc. This can lead to better technology for everyone. Red Hat is pursuing a more modular approach to enabling organizations to build and deploy containers. There is no need to include tools for building containerized applications on instances of Linux where containers run. That approach reduces the overall size of the container platform in the server environment by replacing Docker containers with a lighter-weight CRI-O container engine optimized for Kubernetes clusters. RHEL 8 has made CRI-O its default runtime for containers, the pace of that transition is expected to increase as organizations continue to migrate from RHEL 7. The tools for building containers that are themselves deployed as containers should make it easier to incorporate these tools within the context of existing DevOps workflows. Developers should be able to deploy them on any preferred platform for building applications. Those containerized applications are core to a Red Hat strategy that is keenly focused on RHEL being a foundation for Red Hat OpenShift, the application development and deployment platform based on Kubernetes that can be deployed on-premises or in the cloud. With tools to build those containerized applications being readily accessible, Red Hat is bringing its ultimate hybrid cloud computing ambitions into reality</div>
<br />
<br />
<span style="font-size: x-small;"><b>Reference:</b></span><br />
<span style="font-size: xx-small;">https://www.redhat.com/en/blog/working-container-storage-library-and-tools-red-hat-enterprise-linux<br />https://opensource.com/article/19/5/shortcomings-rootless-containers<br />https://mkdev.me/en/posts/dockerless-part-3-moving-development-environment-to-containers-with-podman </span><br />
<span style="font-size: xx-small;">https://events19.linuxfoundation.org/wp-content/uploads/2017/11/How-Container-Runtime-Matters-in-Kubernetes_-OSS-Kunal-Kushwaha.pdf </span><br />
<span style="font-size: xx-small;">https://hpcw.github.io/ </span><br />
<span style="font-size: xx-small;">NOTE: contents updated as per the information available till May 2020. Later there will be more updates. </span></div>
Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com2tag:blogger.com,1999:blog-8306400524945620796.post-9527860729283894892020-03-25T05:49:00.002+05:302020-11-23T10:04:22.359+05:30 IBM Supercomputer Summit identified possible drug compounds for COVID19 vaccine <div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-QgohGynzeXo/XnqtnSll8GI/AAAAAAAAUrQ/zVH-_1myNeASAK8BUk2ZHZ3peuAaXzeRgCLcBGAsYHQ/s1600/drug.PNG" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="426" data-original-width="714" height="190" src="https://1.bp.blogspot.com/-QgohGynzeXo/XnqtnSll8GI/AAAAAAAAUrQ/zVH-_1myNeASAK8BUk2ZHZ3peuAaXzeRgCLcBGAsYHQ/s320/drug.PNG" width="320" /></a></div>
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">America
is coming together to fight COVID-19, and that means unleashing
the full capacity of our world-class supercomputers to rapidly advance
scientific research for treatments and a vaccine. IBM announced an
initiative to increase access to
high-performance computing for groups researching and fighting the novel
coronavirus, also known as COVID-19. Scientists have enlisted the help
of a supercomputer to fight back
against the rapid spread of the novel coronavirus. Researchers from the
Oak Ridge National Laboratory just published the results of a project in
which they tasked the massive IBM supercomputer known as Summit with
finding the most effective existing drugs that could combat COVID-19.
Summit [described as Formula One of supercomputers], can perform
mathematical equations at speeds that “boggle the mind” i.e Capable of
performing over 200 quadrillion calculations per second . That’s not a
typo. This computation speed accelerates the process of discovery. <span style="font-size: xx-small;">More details on system specifications available at <a href="http://www.sachinpbuzz.com/2019/07/ibms-summit-sierra-most-powerful.html?showComment=1580542016462">link</a></span></span></span></span></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"></span></span></div>
<br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">Supercomputers
can solve calculations and run experiments that, if done
on traditional computing systems or by hand, would take months or years.
In traditional computing systems and data centers, each computer
functions and does calculations independently. By contrast,
high-performance computers can work together and pass calculations
between one another to process information more quickly. Such computers
are also especially good for conducting research in areas like
epidemiology and molecular modeling because the systems mirror the
interconnectivity that exists in nature. </span></span></span><br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><br /></span></span></span></div>
<div style="text-align: justify;">
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">IBM
partnered with the White House Office of Science and Technology Policy
and the Department of Energy to create the COVID-19 High Performance
Computing Consortium. The effort, which IBM started just last week, is
expected to harness powerful high-performance computing, or
“supercomputing,” resources that will massively increase the speed and
capacity of coronavirus-related research. The COVID-19 High-Performance
Computing Consortium includes the Seattle area’s powerhouses of cloud
computing, Amazon Web Services and Microsoft, as well as IBM and Google
Cloud. There are also academic partners (MIT and Rensselaer Polytechnic
Institute), federal agency partners (NASA and the National Science
Foundation) and five Department of Energy labs (Argonne, Lawrence
Livermore, Los Alamos, Oak Ridge and Sandia). Among the resources being
brought to bear is the world’s most powerful supercomputer, the Oak
Ridge Summit, which packs a 200-petaflop punch. The system will harness
16 supercomputing systems from IBM, national
laboratories, several universities, Amazon, Google, Microsoft and
others. Computing power will be provided via remote access to
researchers whose projects are approved by the consortium’s leadership
board, which will be comprised of tech industry leaders and White House
and Energy Department officials. The group plans to begin accepting
research proposals through an online portal.</span></span></span></div>
<div style="text-align: justify;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://1.bp.blogspot.com/-UrN2t2hEaWg/XnqsKxjXYJI/AAAAAAAAUrE/IIoLPSz-1bQb4ziiiMXHYsQ6B4Zly2pwQCLcBGAsYHQ/s1600/summit_SC.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="463" data-original-width="749" height="394" src="https://1.bp.blogspot.com/-UrN2t2hEaWg/XnqsKxjXYJI/AAAAAAAAUrE/IIoLPSz-1bQb4ziiiMXHYsQ6B4Zly2pwQCLcBGAsYHQ/s640/summit_SC.PNG" width="640" /></a></div>
<br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">US Department of Energy’s Oak Ridge National Laboratory(ORNL) has deployed the world’s most powerful and smartest supercomputer, the IBM built Summit, in the fight against COVID-19. Researchers from ORNLwere granted emergency computation time on Summit, using it to perform simulations with unprecedented speed. In just 2 days Summit identified and studied 77 small-molecule drug potential compounds to fight against the COVID-19 (new Coronavirus). A task that – using a traditional wet-lab approach – would have taken years. The researchers at Oak Ridge National Laboratory used Summit to perform simulations of more than 8,000 possible compounds to screen for those that have the most opportunity to have an impact on the disease, by binding to the main “spike” protein of the coronavirus, rendering it unable to infect host cells. Starting with over 8,000 compounds, Summit’s incredible power shortened the time of the experiment dramatically, ruling out the vast majority of possible medications before settling on 77 drugs which it ranked based on how effective they would likely be at halting the virus in the human body.</span></span></span><br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><br />The paper, which was published in the journal ChemRxiv, focuses on the method the virus uses to bind to cells. Like other viruses, the novel coronavirus uses a spike protein to inject cells. Using Summit with an algorithm to investigate which drugs could bind to the protein and prevent the virus from doing its duty, the researchers now have a list of 77 drugs that show promise. They ranked the compounds of interest that could have value in experimental studies of the virus<a href="https://chemrxiv.org/articles/Repurposing_Therapeutics_for_the_Wuhan_Coronavirus_nCov-2019_Supercomputer-Based_Docking_to_the_Viral_S_Protein_and_Human_ACE2_Interface/11871402/3">[source]</a></span></span></span><br />
<br />
<b><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">Whats gene mutation and Features, Evaluation with respect to COVID19 ?</span></span></span></b><br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"></span></span></span><br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><a href="https://1.bp.blogspot.com/-yQbSd47ti8w/XnvlX0_Y89I/AAAAAAAAUrs/Vbe7iY2UNtg9it2spAYl0fw59XtxmzLLQCEwYBhgLKs4DAMBZVoAZDtFOMudHG4aqBTveGcsyMblRscFLFC_ChuyEZRFPfeg7vKCDxyBskrfhynf9zs6xVH_korfZXaI5tUhawsU8lzGmB0NzlsmBKYmnw_ylE0UF3svYs3l1EENrJGSYfRcFP7uirK4m5LMJCwlNhTfSaPl4uYmO4Pj19Zaq5b4halBsG9hJPsEBZoHPOrvlDbb7JtQAJJ2CR8Ro2lJBbmi0k9Jr_CNhDL13qKfoAaxEsd-BhxyoDfeitESgEY6-_JUxMfn0mbOxNPc_Wb-G_I8gjvxq_O_x563deeLuGtwHAnsixKSvNu8gih08wYrdupLbL-6i9D_1VMAh4C6YAohX1nXfgtRll5fsvil0mDsX-xz8U-XwjRU6DYkpVP2lV1qvFzVTJzaTQCRQcpApKSyxE3yDNvx8zCk7nNqiTvPM9CS77THwwcUC__9rxrY7u-ywXxqSLxMfhCUv-fsCwPX0adgY44oUPtDeEDiuDEkNj3VgUt5uLPhyWBAV0PZuad6BO8TQgnICHYIWkU_nY2zvbDRVMe2BsdJ_w2tsGzjhl4k7S6aXMOGmiizcXIvwQ4jyCJjqr3_uvUgKm_zAXt1s2RQqrpABzOzJMMTR7_MF/s1600/corona_virus.PNG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a><a href="https://1.bp.blogspot.com/-yQbSd47ti8w/XnvlX0_Y89I/AAAAAAAAUrs/Vbe7iY2UNtg9it2spAYl0fw59XtxmzLLQCEwYBhgLKs4DAMBZVoAZDtFOMudHG4aqBTveGcsyMblRscFLFC_ChuyEZRFPfeg7vKCDxyBskrfhynf9zs6xVH_korfZXaI5tUhawsU8lzGmB0NzlsmBKYmnw_ylE0UF3svYs3l1EENrJGSYfRcFP7uirK4m5LMJCwlNhTfSaPl4uYmO4Pj19Zaq5b4halBsG9hJPsEBZoHPOrvlDbb7JtQAJJ2CR8Ro2lJBbmi0k9Jr_CNhDL13qKfoAaxEsd-BhxyoDfeitESgEY6-_JUxMfn0mbOxNPc_Wb-G_I8gjvxq_O_x563deeLuGtwHAnsixKSvNu8gih08wYrdupLbL-6i9D_1VMAh4C6YAohX1nXfgtRll5fsvil0mDsX-xz8U-XwjRU6DYkpVP2lV1qvFzVTJzaTQCRQcpApKSyxE3yDNvx8zCk7nNqiTvPM9CS77THwwcUC__9rxrY7u-ywXxqSLxMfhCUv-fsCwPX0adgY44oUPtDeEDiuDEkNj3VgUt5uLPhyWBAV0PZuad6BO8TQgnICHYIWkU_nY2zvbDRVMe2BsdJ_w2tsGzjhl4k7S6aXMOGmiizcXIvwQ4jyCJjqr3_uvUgKm_zAXt1s2RQqrpABzOzJMMTR7_MF/s1600/corona_virus.PNG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="751" data-original-width="900" height="267" src="https://1.bp.blogspot.com/-yQbSd47ti8w/XnvlX0_Y89I/AAAAAAAAUrs/Vbe7iY2UNtg9it2spAYl0fw59XtxmzLLQCEwYBhgLKs4DAMBZVoAZDtFOMudHG4aqBTveGcsyMblRscFLFC_ChuyEZRFPfeg7vKCDxyBskrfhynf9zs6xVH_korfZXaI5tUhawsU8lzGmB0NzlsmBKYmnw_ylE0UF3svYs3l1EENrJGSYfRcFP7uirK4m5LMJCwlNhTfSaPl4uYmO4Pj19Zaq5b4halBsG9hJPsEBZoHPOrvlDbb7JtQAJJ2CR8Ro2lJBbmi0k9Jr_CNhDL13qKfoAaxEsd-BhxyoDfeitESgEY6-_JUxMfn0mbOxNPc_Wb-G_I8gjvxq_O_x563deeLuGtwHAnsixKSvNu8gih08wYrdupLbL-6i9D_1VMAh4C6YAohX1nXfgtRll5fsvil0mDsX-xz8U-XwjRU6DYkpVP2lV1qvFzVTJzaTQCRQcpApKSyxE3yDNvx8zCk7nNqiTvPM9CS77THwwcUC__9rxrY7u-ywXxqSLxMfhCUv-fsCwPX0adgY44oUPtDeEDiuDEkNj3VgUt5uLPhyWBAV0PZuad6BO8TQgnICHYIWkU_nY2zvbDRVMe2BsdJ_w2tsGzjhl4k7S6aXMOGmiizcXIvwQ4jyCJjqr3_uvUgKm_zAXt1s2RQqrpABzOzJMMTR7_MF/s320/corona_virus.PNG" width="320" /></a></span></span></span></div>
<br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">Mutation is a mundane aspect of existence for many viruses, and the novel coronavirus is no exception. This new virus COVID19 which is the acronym of "coronavirus disease 2019" seems to be very contagious and has quickly spread globally.The CoVs have become the major pathogens of emerging respiratory disease outbreaks. They are a large family of single-stranded RNA viruses (+ssRNA) that can be isolated in different animal species. For reasons yet to be explained, these viruses can cross species barriers and can cause, in humans, illness ranging from the common cold to more severe diseases such as MERS and SARS. The potential for these viruses to grow to become a pandemic worldwide seems to be a serious public health risk.</span></span></span><br />
<br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">A mutation is an alteration in the nucleotide sequence of the genome of an organism, virus, or extrachromosomal DNA. Mutations result from errors during DNA replication, mitosis, and meiosis or other types of damage to DNA. The RNA viral genome can be double-stranded (as in DNA) or single-stranded. In some of these viruses, replication occurs quickly, and there are no mechanisms to check the genome for accuracy. This error-prone process often results in mutations. You can think of COVID19 and probably next mutated version COVID20 :) . A gene mutation is a permanent alteration in the DNA sequence that makes up a gene, such that the sequence differs from what is found in most people. Replication errors and DNA damage are actually happening in the cells of our bodies all the time.In most cases, however, they don't cause cancer, or even mutations. That's because they are usually detected and fixed by DNA proofreading and repair mechanisms. </span></span></span><br />
<br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">CoVs are positive-stranded RNA viruses with a crown-like appearance under an electron microscope (coronam is the Latin term for crown) due to the presence of spike glycoproteins on the envelope. The subfamily Orthocoronavirinae of the Coronaviridae family (order Nidovirales) classifies into four genera of CoVs: Alphacoronavirus (alphaCoV), Betacoronavirus (betaCoV), Deltacoronavirus (deltaCoV), and Gammacoronavirus (gammaCoV). Furthermore, the betaCoV genus divides into five sub-genera or lineages.[2] Genomic characterization has shown that probably bats and rodents are the gene sources of alphaCoVs and betaCoVs. On the contrary, avian species seem to represent the gene sources of deltaCoVs and gammaCoVs. Members of this large family of viruses can cause respiratory, enteric, hepatic, and neurological diseases in different animal species, including camels, cattle, cats, and bats. To date, seven human CoVs (HCoVs) — capable of infecting humans — have been identified. </span></span></span><br />
<br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">SARS-CoV-2 belongs to the betaCoVs category. It has round or elliptic and often pleomorphic form, and a diameter of approximately 60–140 nm. Like other CoVs, it is sensitive to ultraviolet rays and heat. Furthermore, these viruses can be effectively inactivated by lipid solvents. Its single-stranded RNA genome contains 29891 nucleotides, encoding for 9860 amino acids. Although its origins are not entirely understood, these genomic analyses suggest that SARS-CoV-2 probably evolved from a strain found in bats. The potential amplifying mammalian host, intermediate between bats and humans, is, however, not known. Since the mutation in the original strain could have directly triggered virulence towards humans, it is not certain that this intermediary exists. Because the first cases of the CoVID-19 disease were linked to direct exposure to the Huanan Seafood Wholesale Market of Wuhan, the animal-to-human transmission was presumed as the main mechanism. Nevertheless, subsequent cases were not associated with this exposure mechanism. Therefore, it was concluded that the virus could also be transmitted from human-to-human, and symptomatic people are the most frequent source of COVID-19 spread. The possibility of transmission before symptoms develop seems to be infrequent, although it cannot be excluded. Moreover, there are suggestions that individuals who remain asymptomatic could transmit the virus. This data suggests that the use of isolation is the best way to contain this epidemic. As with other respiratory pathogens, including flu and rhinovirus, the transmission is believed to occur through respiratory droplets from coughing and sneezing.</span></span></span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-5X8Twh-7VwA/XoRrxGYHjQI/AAAAAAAAUs0/9QTxPwcJPp07tbjup33RTCvBcmUS7bTgQCLcBGAsYHQ/s1600/novel-corona-virus.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="327" data-original-width="681" height="306" src="https://1.bp.blogspot.com/-5X8Twh-7VwA/XoRrxGYHjQI/AAAAAAAAUs0/9QTxPwcJPp07tbjup33RTCvBcmUS7bTgQCLcBGAsYHQ/s640/novel-corona-virus.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://www.ijbs.com/v16p1753.htm">source</a></td></tr>
</tbody></table>
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><br />The novel SARS-CoV-2 coronavirus [COVID19] that emerged in the city of Wuhan, China, last year and has since caused a large scale COVID-19 epidemic and spread all over the world is the product of natural evolution. The scientists analyzed the genetic template for spike proteins, armatures on the outside of the virus that it uses to grab and penetrate the outer walls of human and animal cells. More specifically, they focused on two important features of the spike protein: the receptor-binding domain (RBD), a kind of grappling hook that grips onto host cells, and the cleavage site, a molecular can opener that allows the virus to crack open and enter host cells. A recent scientific article suggested that the novel coronavirus responsible for the Covid-19 epidemic has mutated into a more "aggressive" form. The genetic material of the virus is RNA. Unlike with human DNA, when viruses copy their genetic material, it does not proofread its work. Because RNA viruses essentially operate without a spell-check, they often make mistakes. These "mistakes" are mutations, and viruses mutate rapidly compared to other organisms. Mutations that are harmful to the viruses are less likely to survive and are eliminated through natural selection. Sadly, this new virus doesn’t have that deletion. When mutations occur that help a virus spread or survive better, they are unlikely to make a big difference in the course of an outbreak. Still, a common perception is that the continuous acquisition of mutations will cause our future coronavirus vaccines to be ineffective. While virus evolution may confer vaccine resistance, this process often takes many years for the right mutations to accumulate. A virologist at the Charité University had sequenced the virus from a German patient infected with COVID-19 in Italy. The genome looked similar to that of a virus found in a patient in Munich; both shared three mutations not seen in early sequences from China But he thought it was just as likely that a Chinese variant carrying the three mutations had taken independent routes to both countries. Like all viruses, SARS-CoV-2 evolves over time through random mutations, only some of which are caught and corrected by the virus’s error correction machinery. Scientists will also be scouring the genomic diversity for mutations that might change how dangerous the pathogen is or how fast it spreads. There, too, caution is warranted.<br /> <b> </b></span></span></span><br />
<br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><b> </b><b>Why a supercomputer is needed to fight the coronavirus [COVID19]:</b></span></span></span><br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><b> </b><br />Viruses infect cells by binding to them and using a ‘spike’ to inject their genetic material into the host cell. To understand new biological compounds, like viruses, researchers in wet labs grow the micro-organism and see how it reacts in real-life to the introduction of new compounds. This is a slow process without powerful computers that can perform digital simulations to narrow down the range of potential variables. Computer simulations can examine how different variables react with different viruses. Each of these individual variables can comprise billions of unique data points. When these data points are compounded with multiple simulations, this can become a very time-intensive process if a conventional computing system is used.</span></span></span><br />
<br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;">These promising compounds could now play a role in developing new treatments or even a highly-effective vaccine that would keep the virus from taking root inside a person’s body. Right now, our best defense against the virus is social distancing, but a vaccine or treatment to ease symptoms and shorten recovery time would go a long way toward getting us on track for a return to normalcy. Researchers used the supercomputer to screen 8,000 compounds to identify the 77 most likely to bind to the main “spike” protein in the coronavirus and render it incapable of attaching to host cells in the human body. Those 77 compounds can now be experimented on with the aim of developing a coronavirus treatment. The supercomputer made it possible to avoid the lengthy process of experimenting on all 8,000 of those compounds</span></span></span><br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><br />The results from Summit don’t mean that a cure or treatment for the new coronavirus has been found. But scientists hope that the computational findings will inform future studies and provide a focused framework for wet-labs to further investigate the compounds. Only then will we know if any of them have the needed characteristics to attack and kill the virus. Going forward, the researchers plan to run the experiment again with a new, more accurate model of the protein spike that the virus uses. It’s possible that the new model will change which drugs are most effective against the virus and hopefully shorten the road to a treatment option. It will still be many months before we have a vaccine available, but scientists are hard at work on those solutions. IBM said Summit would continue to be used for "providing ground-breaking technology for the betterment of humankind".<br /><b><span style="font-size: xx-small;">Reference:</span></b><br /><span style="font-size: xx-small;">https://www.ibm.com/blogs/nordic-msp/ibm-supercomputer-summit-attacks-coronavirus<br />https://nypost.com/2020/03/23/supercomputer-finds-77-drugs-that-could-halt-coronavirus-spread<br />https://www.mercurynews.com/2020/03/22/ibm-partners-with-white-house-to-direct-supercomputing-power-for-coronavirus-research</span></span></span></span><br />
<span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-family: "georgia" , "times new roman" , serif;"><span style="font-size: xx-small;">https://www.ncbi.nlm.nih.gov/books/NBK554776/ </span></span></span></span></div>
<ul style="text-align: left;">
</ul>
<ol style="text-align: left;">
</ol>
</div>
Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0tag:blogger.com,1999:blog-8306400524945620796.post-79191326634524337652020-01-10T14:45:00.000+05:302020-01-10T17:40:42.904+05:30what's next in computing - Quantum logic - IBM Q<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
Many of the world’s biggest mysteries and potentially greatest opportunities remain beyond the grasp of classical computers. To continue the pace of progress, we need to augment the classical approach with a new platform, one that follows its own set of rules. That is quantum computing. The importance of quantum computing is both understated and widely
over-hyped at the same time. Although it won’t replace conventional
computers, quantum innovation represents<b> a new computing paradigm</b>. As quantum computing technology advances, clients are becoming
increasingly curious about how it might impact their business. The
intersection of industry and technology will be critical for clients to
identify potential applications of quantum computing.The IBM Q Network is a collaboration of Fortune 500 companies, academic
institutions, and research labs working together to advance quantum
computing. IBM works with the sponsors, champions and stakeholders who will be
influencers to drive initial conversations. Quantum sponsors are
frequently found in a CIO or innovation group that focuses on new and
emerging technology. They will be interested in discussing specific
industry use cases where there is a high potential to leverage quantum
for future business advantage. Mercedes-Benz and Daimler are working with International Business Machines Corp.’s quantum-computing division with the goal of deploying the next-generation computing power in certain use cases. The quantum computers are infinitely faster than supercomputers.There are 3 basic types of quantum computers, quantum annealers, analog quantum and universal quantum computers. Quantum computers operate in a very different way to classical computers. They take advantage of the unusual phenomena of quantum mechanics, for example where subatomic particles can appear to exist in more than one state at any time.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
In September 2019, IBM became the first company to have a <b>fleet of quantum computers</b>. IBM's 14th quantum computer is its most powerful so far, a model with 53
of the qubits that form the fundamental data-processing element at the
heart of the system. IBM is competing with companies like Google,
Microsoft, Honeywell, Rigetti Computing, IonQ, Intel and NTT in the race
to make useful quantum computers. Another company, D-Wave, uses a
different approach called <b>annealing </b>that's already got some customers, while AT&T and others are pursuing the even more distant realm of quantum networking. They are housed at the IBM Quantum Computation Center in New York. The Q
System One is the first quantum system to consolidate thousands of
components into a glass-enclosed, air-tight environment built
specifically for business use. <span style="font-size: xx-small;"><a href="https://www.youtube.com/watch?v=QRaEvXF4YBg">Click here for more info</a> </span></div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-LbX3uTT3q-I/Xhg38er9GAI/AAAAAAAAUeM/mMmnR9S55z4BpRhkGw6SeauTdn0wQohBQCLcBGAsYHQ/s1600/IBM_q_123.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1195" data-original-width="1593" height="480" src="https://1.bp.blogspot.com/-LbX3uTT3q-I/Xhg38er9GAI/AAAAAAAAUeM/mMmnR9S55z4BpRhkGw6SeauTdn0wQohBQCLcBGAsYHQ/s640/IBM_q_123.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Multiple IBM Q systems are housed at the IBM Quantum Computing Center in New York</td></tr>
</tbody></table>
<div style="text-align: justify;">
While traditional computers store information as either 0s and 1s,
quantum computers use quantum bits, or qubits, which represent and store
information as both 0s and 1s simultaneously. That means quantum
computers have the potential to sort through a vast number of possible
solutions in a fraction of a second. Qubits are kept at an extremely
cold temperature of <b>1/100th the temperature of outer space</b>. This measure of temperature is called the Kelvin degree, and zero degrees Kelvin is called "<b>absolute zero</b>". IBM keeps its qubits at <b>0.015 degrees Kelvin</b>,
while the brisk air on a freezing cold winter day is at 273 degrees
Kelvin.Qubits are kept this cold to prolong their fragile quantum state.
The
longer qubits can be kept in a quantum state, the more operations can be
performed on them while taking advantage of superposition and
entanglement. So what's up with a quantum computer with 53 qubits? It stems from the hexagonally derived lattice of qubits that's advantageous when it comes to minimizing unwanted interactions. Quantum computing remains a highly experimental field, limited by the difficult physics of the ultra-small and by the need to keep the machines refrigerated to within a hair's breadth of absolute zero to keep outside disturbances from ruining any calculations.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-UJQBt9AT-7k/XhhBIdht-kI/AAAAAAAAUeY/ELBIFO-e-oEoM2go7ACDlCW6aLxMQwEKQCLcBGAsYHQ/s1600/QC_chip.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="501" data-original-width="949" height="336" src="https://1.bp.blogspot.com/-UJQBt9AT-7k/XhhBIdht-kI/AAAAAAAAUeY/ELBIFO-e-oEoM2go7ACDlCW6aLxMQwEKQCLcBGAsYHQ/s640/QC_chip.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://www.cnet.com/news/ibm-new-53-qubit-quantum-computer-is-its-biggest-yet/"><span class="caption">A close-up view of the IBM Q quantum computer. The processor is in the silver-colored cylinder.</span></a></td></tr>
</tbody></table>
</div>
<div style="text-align: justify;">
Rigetti is racing against similar projects at Google, Microsoft, IBM,
and Intel. Every Bay Area startup will tell you it is doing something
momentously difficult, but Rigetti is biting off more than most – it's
working on quantum computing. All venture-backed startups face the
challenge of building a business, but this one has to do it by making
progress on one of tech's thorniest problems.</div>
<div style="text-align: justify;">
<span class="lede">Within the next </span>five years, Google will produce a viable quantum computer. That's the stake the company has just planted. "The field of quantum computing will soon achieve a historic milestone," They call this milestone "quantum supremacy." the world's biggest tech companies are already jockeying for their own
form of commercial supremacy as they anticipate a quantum breakthrough.
Both Google and IBM now say they will offer access to true quantum
computing over the internet (call it quantum cloud computing).after years spent developing quantum technologies, IBM is also trying to
prevent Google, a relative newcomer to the field, from stealing its
quantum mindshare. And it's still unclear whether the claims made by
these two companies will hold up. The future of quantum computing, like
the quantum state itself, remains uncertain. Rigetti is now entering the fray. The company launched its
own cloud platform, called Forest, where developers can write code for
simulated quantum computers, and some partners get to access the
startup's existing quantum hardware.</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-QNXMSaDCCcM/XfMcXSCia9I/AAAAAAAAUX8/2m9xN48LgWEczCipgSBLO0gSBLIgV5T9wCLcBGAsYHQ/s1600/q_compute.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="628" data-original-width="891" height="225" src="https://1.bp.blogspot.com/-QNXMSaDCCcM/XfMcXSCia9I/AAAAAAAAUX8/2m9xN48LgWEczCipgSBLO0gSBLIgV5T9wCLcBGAsYHQ/s320/q_compute.PNG" width="320" /></a></td></tr>
<tr align="justify"><td class="tr-caption" style="text-align: center;"><a href="https://www.wired.com/story/quantum-computing-factory-taking-on-google-ibm/?mbid=BottomRelatedStories">source</a></td></tr>
</tbody></table>
<div style="text-align: justify;">
Quantum can work in unison with current computing infrastructure to solve complex problems that were previously thought impractical or impossible. This can be paradigm-shifting. For example, the difficulty of factoring large numbers into their primes is the basis of modern cryptography. For the size of numbers used in modern public-private key encryption:On a conventional computer, this calculation would take trillions of years. On a future quantum computer, it would take only minutes. As getting more power out of classical computers for a fixed amount of space, time, and resources becomes more challenging, completely new approaches like quantum computing become ever more interesting as we aim to tackle more complicated problems.Quantum computing could be a way to revive the rate of progress [that we have come to depend on in conventional computers], at least in some areas. "If you can successfully apply it to problems it could give you an exponential increase in computing power that you can’t get” through traditional chip designs That’s because IBM see a future beyond traditional computing. For decades, computing power has doubled roughly every two years — a pattern known as Moore’s Law. Those advances have relied on making transistors ever smaller, thereby enabling each computer chip to have more calculation power. IBM has invented new ways to shrink transistors. IBM grew to its current size by leveraging the continued scaling of conventional computing. But ,that approach is finite, and its end is in sight. Now, times are not so good. “Underlying Moore’s Law is scaling, the ability to pack more and more transistors into a smaller and smaller space, At some point … you’re going to reach atomic dimensions and that’s the end of that approach.” The specter of a world in which silicon chips are not improving exponentially dubbed the IBM Q Network. We don’t lack computing power today, but you see Moore’s Law going into saturation.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
A quantum system in a definite state can still behave randomly. This is a counter-intuitive idea of quantum physics. Quantum computers exist now because we have recently figured out how to control what has been in the world this whole time: the quantum phenomena of superposition, entanglement, and interference. These new ingredients in computing expand what is possible to design into algorithms. The word qubit has two meanings, one physical and one conceptual. Physically, it refers to the individual devices that are used to carry out calculations in quantum computers. Conceptually, a qubit is like a bit in a regular computer. It’s the basic unit of data in a quantum circuit.</div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-y5spts28gYY/XhhYSvkwSRI/AAAAAAAAUe0/7QAq-kwRWsAs8oGYqForgKpkmT38VfJ0ACEwYBhgL/s1600/bit_qbit.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="517" data-original-width="935" height="352" src="https://1.bp.blogspot.com/-y5spts28gYY/XhhYSvkwSRI/AAAAAAAAUe0/7QAq-kwRWsAs8oGYqForgKpkmT38VfJ0ACEwYBhgL/s640/bit_qbit.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://blog.sintef.com/digital-en/diving-deep-into-quantum-computing/">Classical bits, that can only be 0 or 1, qubits can exist in a superposition of these states</a></td></tr>
</tbody></table>
<br />
<div style="text-align: justify;">
A <b>superposition</b> is a weighted sum or difference of two or more states. For example, the state of the air when two or more musical tones sound at once. A "weighted sum or difference" means that some parts of the superposition are more or less prominently represented, such as when a violin is played more loudly than the other instruments in a string quartet. Ordinary, or “classical,” superpositions commonly occur in macroscopic phenomena involving waves.Quantum theory predicts that a computer with n qubits can exist in a superposition of all 2^n of its distinct logical states 000...0, through 111...1. This is exponentially more than a classical superposition. Playing n musical tones at once can only produce a superposition of n states.A set of n coins, each of which might be heads or tails, can be described as a probabilistic mixture of 2^n states, but it actually is in only one of them — we just don’t know which. However, quantum computers are capable of holding their data in superpositions of 2^n distinct logical states. For this reason, quantum superposition is more powerful than classical probabilism. Quantum computers capable of holding their data in superposition can solve some problems exponentially faster than any known classical algorithm. A more technical difference is that while probabilities must be positive (or zero), the weights in a superposition can be positive, negative, or even complex numbers.<b> </b></div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<b>Entanglement</b> is a property of most quantum superpositions and does not occur in classical superpositions. Entanglement is a core concept of quantum computing. In an entangled state, the whole system is in a definite state, even though the parts are not. In an entangled state, the whole system is in a definite state, even though the parts are not. Observing one of two entangled particles causes it to behave randomly, but tells the observer how the other particle would act if a similar observation were made on it. Because entanglement involves a correlation between individually random behaviors of the two particles, it cannot be used to send a message. Therefore, the term “instantaneous action at a distance,” sometimes used to describe entanglement, is a misnomer. There is no action (in the sense of something that can be used to exert a controllable influence or send a message), only correlation, which, though uncannily perfect, can only be detected afterward when the two observers compare notes. The ability of quantum computers to exist in entangled states is responsible for much of their extra computing power.<br />
<br />
One important factor is: Physical qubits are much more sensitive to noise than transistors in
regular circuits. The ability to hold a quantum state is called
coherence. The longer the coherence time, the more operations
researchers can perform in a quantum circuit before resetting it, and
the more sophisticated the algorithms that can be run on it. To reduce
errors, quantum computers need qubits that have long coherence times.
And physicists need to be able to control quantum states more tightly,
with simpler electrical or optical systems than are standard today. A
quantum computer will need about 200 or so perfect qubits to perform
chemical simulations that are impossible on classical computers. Because
qubits are so prone to error, though, these systems are likely to
require redundancy, with tens or perhaps hundreds of faulty qubits doing
the work of one ideal qubit that gives the right answer. These
so-far-theoretical ideal qubits are often called “logical qubits” or
“error-corrected qubits.” . So, It doesn’t make sense to increase the number of qubits before you improve your error rates.</div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
A leap from bits to qubits: this two-letter change could mean entirely
new horizons for healthcare. Quantum computing might bring supersonic
drug design, in silico clinical trials with virtual humans simulated
‘live’, full-speed whole genome sequencing and analytics, the movement
of hospitals to the cloud, the achievement of predictive health, or the
security of medical data via quantum uncertainty. Quantum computing could enable exponential speedups for certain classes of problems by exploiting superposition and entanglement in the manipulation of quantum bits (qubits). One such example is using quantum computing in Artificial Intelligence space, we can implement an optimisation algorithm(s) which can use the properties of superposition and help in speeding up the optimisation problem which can eventually lead to better and faster learning algorithm(s).</div>
<br />
<div style="text-align: justify;">
<b>Quantum encryption</b>, as its name suggests, relies on the quantum properties of photons, atoms, and other small units of matter to secure information. In this case, the physicists used a quantum property of photons known as polarization, which more or less describes the orientation of a photon. For the teleconference, they assigned photons with two different polarizations, to represent 1’s and 0’s. In this way, a beam of light becomes a cryptographic key they could use to scramble a digital message. If implemented the way physicists first envisioned it back in the 1980’s, quantum encryption would be unbreakable. The protocol is a bit complicated, but it essentially involves the sender transmitting photons to the recipient to form a key, and both parties sharing part of the key publicly. If someone had tried to intercept it, the recipient’s key would not match the sender’s key in a specific statistical way, set by rules in quantum mechanics. The sender would immediately know the key was compromised. Physicists also see quantum encryption as an important tool for when quantum computers finally become functional. These quantum computers—or more likely, the ones to follow a few decades later—could bust the best encryption algorithms today. But no computer could crack a properly quantum-encrypted message. Key words: properly encrypted. When physicists started to actually build quantum networks, they couldn’t achieve their vision of perfect quantum encryption. It turns out, sending photons thousands of miles across the world through free space, optical fiber, and relay stations, all without corrupting their polarization, is extremely technically challenging. Quantum signals die after about 100 miles of transmission through optical fiber, and no one knows how to amplify a signal yet. The best quantum memories today can only store a key for a matter of minutes before the information disappears. So Pan’s group had to incorporate conventional telecom technology to propagate their quantum signals. At several points in their network, they had to convert quantum information (polarizations) into classical information (voltages and currents) and then back into quantum. This isn’t ideal, because the absolute security of a quantum key relies on its quantum-ness. Anytime the key gets converted into classical information, normal hacking rules apply.<br />
<br />
Quantum computers working with classical systems have the potential to solve complex real-world problems such as simulating chemistry, modelling financial risk and optimizing supply chains. One of the areas they’re applying their technology is chemistry simulations, for example to understand how materials behave and how chemicals interact. One particularly interesting problem is designing the chemical composition of more effective batteries. These could be used in the next generation of electric vehicles. Exxon Mobil plans to use quantum computing to better understand catalytic and molecular interactions that are too difficult to calculate with classical computers. Potential applications include more predictive environmental models and highly accurate quantum chemistry calculations to enable the discovery of new materials for more efficient carbon capture.<br />
<br />
JP Morgan Chase is focusing on use cases for quantum computing in the financial industry, including trading strategies, portfolio optimization, asset pricing and risk analysis.<br />
<br />
Accelerating drug discovery through quantum-computing molecule comparison :Molecular comparison, an important process in early-phase drug design and discovery. Today, it takes pharmaceutical companies up to 10+ years and often billions of dollars to discover a new drug and bring it to market. Improving the front end of the process with quantum computing can dramatically cut costs and time to market, repurpose pre-approved drugs more easily for new applications, and empower computational chemists to make new discoveries faster that could lead to cures for a range of diseases<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://1.bp.blogspot.com/-7t3ron7a-uk/XhhmBRvqgmI/AAAAAAAAUe8/qAIWzOW8pio9HpmnGSFCelShkX_2G2rqgCEwYBhgL/s1600/qc_drug.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="701" data-original-width="953" height="470" src="https://1.bp.blogspot.com/-7t3ron7a-uk/XhhmBRvqgmI/AAAAAAAAUe8/qAIWzOW8pio9HpmnGSFCelShkX_2G2rqgCEwYBhgL/s640/qc_drug.PNG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://blog.sintef.com/digital-en/diving-deep-into-quantum-computing/">Use case for quantum computers is the design of new materials and drugs</a></td></tr>
</tbody></table>
<br />
Revolutionizing the molecule comparison process Quantum computing has the potential to change the very definition of molecular comparison by enabling pharmaceutical and material science companies to develop methods to analyze larger-scale molecules. Today, companies can run hundreds of millions of comparisons on classical computers; however, they are limited only to molecules up to a certain size that a classical computer can actually compute. As quantum computers become more readily available, it will be possible to compare molecules that are much larger, which opens the door for more pharmaceutical advancements and cures for a range of diseases.</div>
<div style="text-align: justify;">
Discovering new battery materials could “unlock a billion-dollar opportunity for Automotive industries. This <b>case </b>could simulate the actual behavior of a battery with a quantum computer, which is currently not possible with existing computer power. Daimler joins other automotive companies experimenting with quantum computing’s potential applications. Ford Motor Co. is researching how the technology could quickly optimize driving routes and improve the structure of batteries for electric vehicles. Volkswagen AG is developing a quantum-computing-based traffic-management system that could be offered as a commercial service. It also is interested in developing more advanced batteries. Today, battery development and testing is a physical process that requires experts to build prototypes first because there is no simulation software. A quantum computer could help Mercedes-Benz find new materials or combinations of materials that could result in better electrochemical performance and longer life cycles of batteries. Some of those innovations could include organic batteries, which could be safer, energy efficient and environmentally friendly.<br />
<br />
</div>
<div style="text-align: justify;">
<b>Full-Scale Fault Tolerance</b><br />
<br />
The third phase is still
decades away. A universal fault-tolerant quantum computer is the grand
challenge of
quantum computing. It is a device that can properly perform universal
quantum operations using unreliable components. Today's quantum
computers are not fault-tolerant. Achieving full-scale fault tolerance
will require makers of quantum technology to overcome additional
technical constraints, including problems related to scale and
stability. But once they arrive, we expect fault-tolerant quantum
computers to affect a broad array of industries. They have the potential
to vastly reduce trial and error and improve automation in the
specialty-chemicals market, enable tail-event defensive trading and
risk-driven high-frequency trading strategies in finance, and even
promote in silico drug discovery, which has major implications for
personalized medicine.<br />
<br />
Now is the right time for business leaders to prepare for quantum. The
conditions are in place to experiment and expand this fundamentally new
technology. Organizations that seek to be at the forefront of this
transformational shift will seize competitive advantage. Rigetti—like Google, IBM, and Intel—preaches the idea that this advance
will bring about a wild new phase of the cloud computing revolution.
Data centers stuffed with quantum processors will be rented out to
companies freed to design chemical processes and drugs more quickly, or
deploy powerful new forms of machine learning.Over the last decade, banks and government institutions in multiple
countries including the US, China, and Switzerland have dabbled in
quantum encryption products, but Christensen suspects that the
technology will be niche for a while longer. Because the technology is
so new, the costs and benefits aren’t clear yet. IBM Q Network is working with 45 clients, including startups, academic institutions and Fortune 500 clients. Large enterprise clients are investing in the emerging technology now so they will be prepared when a commercial-grade quantum computer comes to market, capable of error-correcting and solving large-scale problems. With all this promise, it’s little surprise that the value creation numbers get very big over time.</div>
<br />
-----------------<b> </b><br />
<b>Reference:</b><br />
<br />
<span style="font-size: xx-small;">https://www.ibm.com/thought-leadership/innovation_explanations/article/dario-gil-quantum-computing.html</span><br />
<span style="font-size: xx-small;">Dario Gil, IBM Research : https://www.youtube.com/watch?v=yy6TV9Dntlw</span><br />
<span style="font-size: xx-small;">https://www.youtube.com/watch?v=lypnkNm0B4A</span><br />
<span style="font-size: xx-small;">https://fortune.com/longform/business-quantum-computing/</span><br />
<span style="font-size: xx-small;">https://www.wired.com/2017/03/race-sell-true-quantum-computers-begins-really-exist/?mbid=BottomRelatedStories</span><br />
<span style="font-size: xx-small;">https://www.wired.com/story/quantum-computing-factory-taking-on-google-ibm/?mbid=BottomRelatedStories</span><br />
<span style="font-size: xx-small;">https://www.wired.com/story/why-this-intercontinental-quantum-encrypted-video-hangout-is-a-big-deal/?mbid=BottomRelatedStories</span><br />
<span style="font-size: xx-small;">https://www.accenture.com/ro-en/success-biogen-quantum-computing-advance-drug-discovery </span><br />
<br /></div>
Sachin P Bhttp://www.blogger.com/profile/13393800234271237966noreply@blogger.com0