Thursday, February 23, 2023

High performance computing

 High-Performance Computing (HPC or supercomputer) is omnipresent in today’s society. For example, every time you watch Netflix, the recommendation algorithm leverages HPC resources remotely to offer you personalized suggestions. HPC stands for High-Performance Computing. The ability to carry out large scale computations to solve complex problems, that either need to process a lot of data, or to have a lot of computing power at their disposal. Basically, any computing system that doesn’t fit on a desk can be described as HPC. 

HPC systems are actually networks of processors. The key principle of HPC lies in the possibility to run massively parallel code to benefit from a large acceleration in runtime.  A common HPC capability is around 100,000 cores. Most HPC applications are complex tasks which require the processors to exchange their results. Therefore, HPC systems need very fast memories and a low-latency, high-bandwidth communication systems (>100Gb/s) between the processors as well as between the processors and the associated memories.

We can differentiate two types of HPC systems: the homogeneous machines and the hybrid ones. Homogeneous machines only have CPUs while the hybrids have both GPUs and CPUs. Tasks are mostly run on GPUs while CPUs oversee the computation. 


They have more computing power since GPUs can handle millions of threads simultaneously and are also more energy efficient. GPUs have faster memories, require less data transfer and are capable to exchange with other GPUs, which is the most energy-intensive part of the machine.

source

High Performance Computing used to be strictly defined with high speed network to allow strong interconnections between cores. The rise of AI applications led to an architecture based on more independent clusters but still massively parallel.

HPC systems also include the software stack. That can be divided into three categories. First the user environment encompasses the applications known as workflows. Then the middleware linking applications and their implementation on the hardware. It includes the runtimes and frameworks. Last, the Operating system, at system level with the job scheduler, management software for load balancing and data availability. Its role is to assign tasks to the processors and organize the exchange of data between the processors and the memories to ensure the best performance.


HPC applications
HPC provides many benefits and value when used for commercial and industrial applications. Applications that can be classified in five categories:

- Fundamental research aims to improve scientific theories to better understand natural or other phenomena. HPC enables more advanced simulations leading to breakthrough discoveries.

- Design simulation allows industries to digitally improve the design of their products and test their properties. It enables companies to limit prototyping and testing, making the designing process quicker and less expensive.

- Behavior prediction enables companies to predict the behavior of a quantity which they can’t impact but depend on, such as the weather or the stock market trends. HPC simulations are more accurate and can look farther into the future thanks to their superior computing abilities. It is especially important for predictive maintenance and weather forecasts.

- Optimization is a major HPC use case. It can be found in most professional fields, from portfolio optimization to process optimization, to most manufacturing challenges faced by the industry.

HPC is more and more used for data analysis. Business models, industrial processes and companies are being built on the ability to connect, analyze and leverage data, making supercomputers a necessity in analyzing massive amounts of data.

The 5 fields of HPC Applications.







Another major application for HPC is in the fields of medical and material advancements. For instance, HPC can be deployed to:

Combat cancer: Machine learning algorithms will help supply medical researchers with a comprehensive view of the U.S. cancer population at a granular level of detail.

Identify next-generation materials: Deep learning could help scientists identify materials for better batteries, more resilient building materials and more efficient semiconductors.

Understand patterns of disease: Using a mix of artificial intelligence (AI) techniques, researchers will identify patterns in the function, cooperation, and evolution of human proteins and cellular systems.

HPC needs are skyrocketing. A lot of sectors are beginning to understand the economic advantage that HPC represents and therefore are developing HPC applications. 

Industrial companies in the field of aerospace, automotive, energy or defence are working on developing digital twins of a machine or a prototype to test certain properties. This requires a lot of data and computing power in order to accurately represent the behavior of the real machine. This will, moving forward, render prototypes and physical testing less and less standard.

The HPC dynamics and industrial landscape

source

The limits of a model :
Unfortunately, supercomputers are revealing some limits. First of all, some problems are not currently solvable by a supercomputer. The race to the exascale (a supercomputer able to realize 10^18 floating point operations per second) is not necessarily going to solve this issue. Some problems or simulations might remain unsolvable, or at least, unsolvable in an acceptable length of time. For example, in the case of digital twins or molecular simulation, calculations have to be greatly simplified in order for current computers to be able to make them in an acceptable length of time (for product or drug design).

Moreover, a second very important challenge is the power consumption. The consumption of computing and data centers represents 1% of power consumption in the world and this is bound to significantly increase. It shows that this model is unsustainable in the long term, especially since exascale supercomputers will most surely consume more than current ones. Not only is it technically unsustainable, it is also financially so. Indeed, a supercomputer can cost as much as 10mUSD per year in electricity consumption.

The new chips revolution
CPUs and GPUs are not the only solutions to tackle the two previously stated issues.

Although most efforts are focused on developing higher-performance CPU and GPU-powered supercomputers in order to reach the exascale, new technologies, in particular “beyond Silicon”, are emerging. Innovative chip technologies could act as accelerators like GPUs did in the 2010s and significantly increase the computing power. Moreover, some technologies, such as quantum processors for example, would be able to solve new categories of problems that are currently beyond our reach.

In addition, 70% of the energy consumption in a HPC is accounted for by the processors. Creating new chips, more powerful and more energy efficient would enable us to solve both problems at once. GPUs were the first step towards this goal. Indeed, for some applications, GPUs can replace up to 200 or 300 CPUs. Although one GPU individually consumes a bit more than a CPU (400W against 300W approximately), overall, a hybrid supercomputer will consume less than a homogeneous supercomputer of equal performance.

The model needs to be reinvented to include disruptive technologies. Homogeneous supercomputers should disappear, and it is already underway. In 2016, only 10 out of the supercomputers in the Top500 were hybrid. By 2020, within only four years, it rose to 333 out of 500, including 6 in the top 10.

At Quantonation, are convinced that innovative chips integrated in hetereogeneous supercomputing architectures, as well as optimized softwares and workflows, will be key enablers to face societal challenges by significantly increasing sustainability and computing power. We trust that these teams are ready to face the challenge and be part of the future of compute:

  1. Pasqal’s neutral atoms quantum computer, highly scalable and energy efficient;
  2. Lighton’s Optical Processing Unit, a special purpose ligh-based AI chip fitted for tasks such as Natural Language Processing;
  3. ORCA Computing’s fiber based photonic systems for simulation and fault tolerant quantum computing;
  4. Quandela’s photonic qubit sources that will fuel next generation of photonic devices;
  5. QuBit Pharmaceuticals software suites leveraging HPC and quantum computing resources to accelerate drug discovery ;
  6. Multiverse Computing’s solutions using disruptive mathematics to resolve finance’s most complex problems on a range of classical and quantum technologies.


IBM Cloud HPC IaaS for building HPC environments using IBM’s Virtual Private Cloud (VPC). It enables you to create your own configuration for Compute Instances; High-performance Storage and Networking like Public Gateways, Load Balancers and Routers. Multiple connectivity options are available upto 80Gbps and IBM Cloud offers the highest level of security and encryption with FIPS 140-2 Level 4. Also available is IBM Code Engine, a fully managed serverless platform to run containers, applications or batch jobs.
– Spectrum Computing provides intelligent dynamic hybrid cloud capabilities which enables organizations to use cloud resources according to defned policies. Spectrum LSF and Symphony allows you to burst workloads to the cloud, dynamically provision cloud resources and intelligently move data to manage egress costs. It also enables the ability for auto scaling to take full advantage of consumption-based pricing and pay for cloud resources only when they are needed.
– Spectrum Scale is an enterprise grade High Performance File System (HPFS) that delivers scalable capacity and performance to handle demanding data analytics, content repositories and HPC workloads. Spectrum Scale architecture allows it to handle tens of thousands of clients, billions of fles and petabytes of data written and retrieved as fles or objects with low latency. Optionally, IBM Aspera can be used for high speed data movement using the FASP protocol.



Use Cases
– Financial Services: Monte Carlo simulation, risk modeling, actuarial sciences
– Health and Life Sciences: Genome analysis, drug discovery, bio-sequencing, clinical treatments, molecular modeling
– Automotive: Vehicle drag coeffcient analysis, crash simulation, engine combustion analysis, air flow modeling
– Aerospace: Structural, fluid dynamics, thermal, electromagnetic and turbine flow analysis
– Electronic Design Automation (EDA): Integrated Circuit (IC) and Printed Circuit Board (PCB) design and analysis
– Oil and Gas: Subsurface terrain modeling, reservoir simulation, seismic analysis
– Transportation: Routing logistics, supply chain optimization
– Energy & Utility: Severe storm prediction, climate, weather and wind modelling
– Education/Research: High energy physics, computational chemistry 

The reign and modern challenges of the Message Passing Interface (MPI)

All good, but why do you guys doing numerical linear algebra and parallel computing always use the Message Passing Interface to communicate between the processors?”

MPI begun about 25 years ago and has been since then, undoubtedly, the “King” of HPC. What were the characteristics of MPI that made it the de-facto language of HPC?

MPI-3 has added several interfaces to enable more powerful communication scheduling, for example nonblocking collective operations and neighborhood collective operations. 

Much of the big data community moved from single-nodes to parallel and distributed computing to process larger amounts of data using relatively short-lived programs and scripts. Thus, programmer productivity only played a minor role in MPI/HPC while it was one of the major requirements for big data analytics. While MPI codes are often orders-of-magnitude faster than many big data codes, they also take much longer to develop. And that is most often a good trade-off.

MPI I/O has been introduced nearly two decades ago to improve the handling of large datasets in parallel settings. It is successfully used in many large applications and I/O libraries such as HDF-5. 

MPI predates the time when the use of accelerators became commonplace. However, when these accelerators are used in distributed-memory settings such as computer clusters, then MPI is the common way to program them. The current model, often called MPI+X (e.g., MPI+CUDA), combines traditional MPI with accelerator programming models (e.g., CUDA, OpenACC, OpenMP etc.) in a simple way. In this model, MPI communication is performed by the CPU. Yet, this can be inefficient and inconvenient and recently, we have proposed a programming model called distributed CUDA (dCUDA) to perform communication from within a CUDA compute kernel [3]. This allows to use the powerful GPU warp scheduler for communication latency hiding. In general, integrating accelerators and communication functions is an interesting research topic.

Programming at the transport layer, where every exchange of data has to be implemented with lovingly hand-crafted sends and receives or gets and puts, is an incredibly awkward fit for numerical application developers, who want to think in terms of distributed arrays, data frames, trees, or hash tables. 

Everyone uses MPI” has made it nearly impossible for even made-in-HPC-land tools like Chapel or UPC to make any headway, much less quite different systems like Spark or Flink, meaning that HPC users are largely stuck with using an API which was a big improvement over anything else available 25 years ago, 

Chapel is a modern programming language designed for productive parallel computing at scale. Chapel's design and implementation have been undertaken with portability in mind, permitting Chapel to run on multicore desktops and laptops, commodity clusters, and the cloud, in addition to the high-end supercomputers for which it was originally undertaken.

Reference:

https://medium.com/quantonation/a-beginners-guide-to-high-performance-computing-ae70246a7af

https://github.com/ljdursi/mpi-tutorial/blob/master/presentation/presentation.md

https://github.com/chapel-lang/chapel

https://blog.xrds.acm.org/2017/02/message-passing-interface-mpi-reign-modern-challenges/

================

Test log Analytics with Elasticsearch and Kibana

Software Test Analytics is the process of collecting and analyzing data from software testing activities to improve the quality and efficiency of the testing process. This can include metrics such as test coverage, defect density, and test execution time, as well as data on test automation and test case management. The goal of software test analytics is to identify trends and patterns in the data that can be used to make informed decisions about how to improve the testing process, such as where to focus testing efforts, which tests to automate, and how to optimize test case design.

Test log analysis is the process of collecting, analyzing, and interpreting data from test logs in order to identify patterns, trends, and issues that can help improve the quality and performance of software systems. This can include data such as test results, error messages, performance metrics, and other relevant information. The goal of test log analysis is to help identify and resolve issues that may be impacting the performance or functionality of the system, and to improve the overall quality of the software. Common techniques used in test log analysis include statistical analysis, machine learning, and data visualization.

Insights that can be gained from analyzing test case output logs are listed below: 

1) Identifying which test cases are passing and which are failing, and the reasons for the failures. This can help you to focus your testing efforts on the areas of the application that need the most attention.

2) Understanding the performance of the application under test. This can include metrics such as response time, memory usage, and CPU usage, which can help you to identify and fix performance bottlenecks.

3) Identifying patterns in the test case data that indicate potential issues with the application under test. For example, if a large number of test cases are failing in a particular module, it could indicate a problem with that module that needs to be investigated.

4) Identifying areas of the application that are not being adequately tested. This can help you to create new test cases to cover these areas, and to ensure that the application is thoroughly tested before release.

5) Identifying where automation can improve the testing process. With the help of logs, you can automate test cases which are repetitive, time-consuming or prone to human errors.

6) To get the most value out of test case output logs, it's important to have a systematic and automated way of collecting and analyzing the data. This can include using tools such as log analyzers, data visualization tools, and automated reporting tools.

Open-source frameworks that can be used for test result analytics: 

ELK Stack: ELK stands for Elasticsearch, Logstash, and Kibana. Elasticsearch is a search engine, Logstash is a log aggregator, and Kibana is a data visualization tool. By using the ELK stack, you can collect, store, and analyze large volumes of test result data in real-time.


The ELK stack (Elasticsearch, Logstash, and Kibana) can be integrated with a CI (Continuous Integration) framework in several ways to show Test analytics.

1. Logstash: Logstash can be used to collect and parse log files generated by the CI framework. You can configure Logstash to read log files from the CI server and to parse the data into a format that can be indexed by Elasticsearch.

2. Kibana: Kibana can be used to visualize the data collected by Logstash. You can create a dashboard in Kibana that displays metrics such as build time, test execution time, and pass/fail rate.

3. Elasticsearch: Elasticsearch can be used to store and index the data collected by Logstash. You can use Elasticsearch to search and analyze the data, and to create complex queries and visualizations.

4. Integrate with CI/CD tool: You can integrate ELK stack with your CI/CD tool, for example, Jenkins, Travis or CircleCI. You can configure the CI tool to send log files to Logstash, or directly to Elasticsearch.

-------------------

After Installation and configuring the ELK stack and Jenkins, the next step for analytics on test reports logs would be to start analyzing and visualizing the data. Here are some steps you can take:

1. Verify data collection: Ensure that the data is being collected correctly and that the logs are being indexed in Elasticsearch.

2. Create visualizations: Use Kibana to create visualizations such as line charts, bar charts, and pie charts to represent the data in a meaningful way. These visualizations can be used to represent metrics such as build time, test execution time, and pass/fail rate.

3. Create dashboards: Use Kibana to create dashboards that display multiple visualizations in a single view. These dashboards can be used to monitor the build and test results in real-time and to analyze the data over time.

4. Define alerts: Set up alerts in Kibana to notify you when certain conditions are met, such as a high number of test failures.

5. Analyze the data: Use Elasticsearch to create complex queries and to analyze the data in more detail. This can be used to identify patterns and trends in the data that can help improve the testing process.

6. Improve test coverage: Use the data to identify areas of the application that are not being adequately tested and to focus your testing efforts on those areas.

7. Identify and fix defects: Use the data to identify the root cause of test failures and to fix defects in the application

------------------------

We can ingest data with Python on Elastic search instead of using logstash which is good for realtime data ingestion. Prerequisites: Get elasticsearch packages

python -m pip install elasticsearch
python -m pip install elasticsearch-async

Some commonly used elasticsearch APIs listed below :

  1. es.index: Used to index a document into an index.
  2. es.search: Used to search for documents in an index based on a query.
  3. es.get: Used to retrieve a document from an index by its ID.
  4. es.delete: Used to delete a document from an index by its ID.
  5. es.update: Used to update a document in an index.
  6. es.count: Used to count the number of documents that match a query without returning the actual documents.
  7. es.exists: Used to check if a document exists in an index.
  8. es.bulk: Used to execute multiple index, update, or delete requests in a single HTTP request.
  9. es.create: Used to create a new index.
  10. es.delete_index: Used to delete an existing index.
  11. es.get_mapping: Used to retrieve the mapping of a document type in an index.
  12. es.put_mapping: Used to define the mapping of a document type in an index.
  13. es.cluster.health: Used to retrieve information about the health of the Elasticsearch cluster.
  14. es.cluster.state: Used to retrieve the current state of the Elasticsearch cluster.
  15. es.nodes.info: Used to retrieve information about the nodes in the Elasticsearch cluster.
  16. es.nodes.stats: Used to retrieve statistics about the nodes in the Elasticsearch cluster.
  17. es.termvectors: Used to retrieve information about the terms in a document.
  18. es.mtermvectors: Used to retrieve information about the terms in multiple documents.
  19. es.explain: Used to explain how a particular document matches a query.
  20. es.mget: Used to retrieve multiple documents from an index by their IDs

Example : Python code to use the Elasticsearch Python client to index a test log report into an Elasticsearch index:

Pre-requisite: Elasticsearch Python client installed (pip install elasticsearch)

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

# cat my_test_dataToES.py

#Let me create a sample test report data and push it to ElasticsearchDB

from datetime import datetime
from elasticsearch import Elasticsearch
# create an Elasticsearch client instance
es = Elasticsearch("http://sachin.bengaluru.com:9200")

# create an Elasticsearch index to store the test log report
es.indices.create(index='test_logs_sachin', ignore=400)

# define the test log report as a Python dictionary
test_log = {
    'test_name': 'login_test',
    'status': 'failed',
    'error_message': 'Invalid credentials',
    'timestamp': datetime.now()
}
# index the test log report into the Elasticsearch index
es.index(index='test_logs_sachin', body=test_log)
#

Execute the python script 

# python3 my_test_dataToES.py

#

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Check the kibana dashboard and create ==> Create index pattern


Visualizing testing results can help you to quickly and easily identify patterns and trends in the data, and can provide valuable insights into the performance and quality 

1) Test Execution Progress: Creating a graph or chart that shows the progress of test execution over time can help you to identify trends in test case pass/fail rates, and to identify areas of the application that are not being adequately tested.

2) Test Case Pass/Fail Rates: Creating a graph or chart that shows the pass/fail rate for each test case can help you to quickly identify which test cases are passing and which are failing. This can help you to focus your testing efforts on the areas of the application that need the most attention.

3) Defect Density: Creating a graph or chart that shows the number of defects per unit of code can help you to identify areas of the application that are prone to defects and to identify patterns in the types of defects that are being found.

4) Test Execution Time: Creating a graph or chart that shows the execution time for each test case can help you to identify performance bottlenecks and to optimize test case design.

5) Test Automation: Creating a graph or chart that shows the percentage of test cases that are automated can help you to identify areas of the application that can benefit from test automation.

6) Test coverage : Creating a graph or chart that shows how much of the application is being tested by your test suite can help you identify the areas that are not being covered and focus on increasing test coverage.

-----------------------------------------------

NOTE: 

Arrays: https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html

In Elasticsearch, there is no dedicated array data type. Any field can contain zero or more values by default, however, all values in the array must be of the same data type. For instance:

an array of strings: [ "one", "two" ]

an array of integers: [ 1, 2 ]

an array of arrays: [ 1, [ 2, 3 ]] which is the equivalent of [ 1, 2, 3 ]

an array of objects: [ { "name": "Mary", "age": 12 }, { "name": "John", "age": 10 }]

Arrays of objects: Arrays of objects do not work as you would expect: you cannot query each object independently of the other objects in the array. If you need to be able to do this then you should use the nested data type instead of the object data type. This is explained in more detail in Nested.

Detailed insights from test result logs :

1) Root Cause Analysis: By analyzing the logs of failed test cases, you can identify the root cause of the failure, such as an issue with the application under test, a problem with the test case design, or an environment issue. This can help you to quickly fix the problem and prevent similar issues in the future.

2) Correlation Analysis: By analyzing the logs of multiple test cases, you can identify patterns and correlations between test results, such as the relationship between test execution time and the number of defects found. This can help you to identify areas of the application that are prone to defects and to optimize test case design.

3) Regression Analysis: By analyzing the logs of test cases that have been executed over time, you can identify trends in test case pass/fail rates, and to identify areas of the application that are not being adequately tested. This can help you to focus your testing efforts on the areas of the application that need the most attention.

4) Log Parsing: By parsing the logs, you can extract relevant information such as test case name, status, execution time, error messages, and stack trace. This information can be further analyzed to identify trends and patterns that can help improve the testing process.

5) Anomaly Detection: By analyzing the logs, you can identify anomalies or unexpected behavior in the test results. This can help you to identify potential issues with the application under test and to quickly fix them before they become major problems.

Machine Learning: You can use machine learning techniques such as clustering, classification, or prediction to analyze test results logs. This can help you to identify patterns and insights that would be difficult to discover manually.

Natural Language Processing: By using NLP techniques, you can extract useful information from unstructured test result logs. This information can be used to identify patterns and insights that would be difficult to discover manually.

These techniques can be implemented using machine learning libraries and frameworks such as scikit-learn, TensorFlow, or PyTorch.  It's also important to have a good understanding of the data, cleaning and preprocessing it before training the model.

Monday, February 20, 2023

HPC workloads on Kubernetes cluster with Volcano batch scheduler

Kubernetes is primarily designed for managing containerized workloads, and it can also be used for managing High-Performance Computing (HPC) clusters. However, using Kubernetes for HPC clusters may require additional customization and configuration.


Performance: Kubernetes is not specifically designed for high-performance computing workloads, and it may not provide the same level of performance as specialized HPC schedulers. To improve performance, you may need to customize the Kubernetes scheduling policies and resource management.

Networking: HPC workloads often require high-bandwidth, low-latency networking between compute nodes. Kubernetes networking is designed for containerized workloads, and it may not provide the required level of performance for HPC workloads. You may need to use specialized network plugins or customize the Kubernetes networking configuration to achieve the desired performance.

Storage: HPC workloads often require large amounts of high-performance storage. Kubernetes provides built-in support for persistent volumes, but it may not be optimized for HPC workloads. You may need to use specialized storage solutions or customize the Kubernetes storage configuration.

Resource Management: HPC clusters often require strict resource management to ensure that workloads are efficiently utilizing compute resources. Kubernetes provides a range of resource management features, such as quotas and limits, but you may need to customize the configuration to meet the specific needs of your HPC workloads

----------------------

Kubernetes Batch: Kubernetes Batch is a built-in batch scheduler that is included in the core Kubernetes distribution. It provides basic batch job management functionality, such as job creation, monitoring, and cleanup.

In addition to Kubernetes Batch, there are several other batch schedulers that can be used with Kubernetes for managing batch jobs, including:

  • Apache Airflow: Apache Airflow is a popular open-source platform for creating, scheduling, and monitoring workflows. It can be used to manage batch jobs in Kubernetes using the KubernetesExecutor.
  • Apache Spark: Apache Spark is a distributed computing framework that includes a built-in scheduler for managing batch jobs. Spark can be run on Kubernetes using the Spark operator, which provides support for managing Spark jobs as Kubernetes native resources.
  • HTCondor: HTCondor is a widely-used batch scheduler in the high-performance computing (HPC) community. It can be used with Kubernetes through the HTCondor-Kubernetes integration, which allows HTCondor to manage Kubernetes pods as HTCondor jobs.
  • Slurm: Slurm is another popular batch scheduler in the HPC community. It can be used with Kubernetes through the Slurm-Kubernetes integration, which allows Slurm to manage Kubernetes pods as Slurm jobs.
  • Valcano: Valcano is a new batch scheduler designed specifically for Kubernetes. It provides advanced job scheduling and resource management capabilities, such as backfill scheduling and intelligent resource allocation

---------------------------------------------------------------

Volcano is a Kubernetes native batch system and job scheduler designed to handle deep learning, machine learning, and high-performance computing (HPC) workloads. It was developed by Alibaba Cloud and was released as an open-source project in 2018.

Here are some of the features that make Volcano special for HPC workloads:
  • Resource Management: Volcano has advanced resource management capabilities that allow it to efficiently schedule and manage HPC workloads across a large number of nodes. It can allocate GPUs, CPUs, memory, and other resources required for running HPC workloads in a distributed computing environment.
  • Performance Optimization: Volcano is designed to optimize the performance of HPC workloads by minimizing resource contention and reducing the time it takes to start and complete jobs. It uses advanced scheduling algorithms to allocate resources and optimize job execution times.
  • Custom Schedulers: Volcano allows users to create custom schedulers and customize job scheduling policies to meet their specific requirements. This makes it easier for users to configure the scheduler to meet the specific needs of their HPC workloads.
  • Job Prioritization: Volcano supports job prioritization based on various factors such as job dependencies, job age, and user-defined priorities. This ensures that higher priority jobs are executed first, and that resources are allocated efficiently across the cluster.
  • Workflow Support: Volcano supports complex workflow management, allowing users to define dependencies between jobs and execute them in a specific order. This is particularly useful for HPC workloads that require a series of jobs to be executed in a specific sequence.
  • Overall, Volcano is a powerful job scheduler that is optimized for HPC workloads in Kubernetes environments. Its advanced resource management, performance optimization, and custom scheduling capabilities make it an ideal choice for running complex deep learning, machine learning, and HPC workloads in Kubernetes.

------------------

Integrating the Volcano scheduler into a Kubernetes cluster involves following steps:

Install the Volcano components: The first step is to install the Volcano components on the Kubernetes cluster. This can be done using the Volcano Helm chart, which includes all the necessary components such as the Volcano scheduler, admission controllers, and CRDs (Custom Resource Definitions).

Install Volcano components with Helm chart:

# Add the Volcano Helm repository
helm repo add volcano https://volcano.sh/charts
helm repo update


# Install the Volcano components
helm install volcano volcano/volcano

NOTE: HELM is  package manager for Kubernetes

Configure the scheduler: Once the Volcano components are installed, the next step is to configure the Volcano scheduler. This involves setting the scheduling policies, priority classes, and other parameters that govern how the scheduler allocates resources to jobs.

Configure the scheduler with YAML file:

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: default
spec:
  # Set the scheduling policy for the queue
  schedulingPolicy:
    type: "PriorityPolicy"
    priorityPolicy:
      defaultPriority: 50
  # Set the resource limits for the queue
  resources:
    limits:
      cpu: "16"
      memory: "64Gi"
    requests:
      cpu: "2"
      memory: "8Gi"

Apply the YAML file with the following command:

kubectl apply -f queue.yaml

Define the job templates: After configuring the scheduler, the next step is to define the job templates that will be used to create batch jobs. This involves specifying the Docker image, command, arguments, and resource requirements for each job template.

Define job templates with YAML file:

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl"]
        args: ["-Mbignum=bpi", "-wle", "print bpi(2000)"]
        resources:
          limits:
            cpu: "2"
            memory: "8Gi"
          requests:
            cpu: "1"
            memory: "4Gi"
      # Set the queue name to use for this job
      queue: default

Apply the YAML file with the following command:

kubectl apply -f job.yam

---------------------

Create batch jobs: Once the job templates are defined, batch jobs can be created using the Kubernetes API or the kubectl command-line tool. When a batch job is created, the Volcano scheduler will allocate resources to the job based on the scheduling policies and resource requirements defined in the job template. Create batch jobs with kubectl command:

kubectl create job pi --image=perl -- perl -Mbignum=bpi -wle 'print bpi(2000)'

Monitor and manage batch jobs: Finally, the batch jobs can be monitored and managed using the Kubernetes API or the kubectl command-line tool. This includes viewing the status of running jobs, scaling up or down the number of replicas, and deleting completed jobs.


Monitor and manage batch jobs with kubectl command:

# View the status of all batch jobs

kubectl get jobs.batch.volcano.sh


# View the logs for a specific batch job

kubectl logs job/pi


# Scale up or down the number of replicas for a batch job

kubectl scale job/pi --replicas=10


# Delete a completed batch job

kubectl delete job/pi

-----------------------------

kubectl get jobs.batch.volcano.sh

This command will display a table with information about all the Volcano jobs currently running, including the job name, namespace, queue, status, and creation time.

If you want to see more details about a specific job, you can use the following command

kubectl describe job.batch.volcano.sh job-name

--------------------------------

The rdma/hca_shared resource in Kubernetes refers to the HCA (Host Channel Adapter) Shared Device Plugin. This device plugin enables Kubernetes to detect and utilize InfiniBand and RDMA network interfaces on the host nodes, allowing containers to access RDMA resources through a standard Kubernetes API. By using the HCA Shared Device Plugin in a Kubernetes cluster, applications can take advantage of RDMA technology for high-performance networking and low-latency communication. This is especially useful for applications that require high-throughput data transfer or require low-latency communication, such as big data analytics, high-performance computing, and machine learning workloads. So, rdma/hca_shared configuration in Kubernetes is used to enable RDMA support in the cluster, which can improve the performance of certain types of applications that require fast, low-latency networking.

--------------------Add  resources to a Kubernetes cluster using YAML file-----

To add Nvidia GPU resources to a Kubernetes cluster using YAML file, you will need to modify the spec section of the deployment or pod YAML file to include the necessary configuration options. Here's an example YAML file for adding Nvidia GPU resources to a Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-container
          image: my-image
          resources:
            limits:
              nvidia.com/gpu: 1
---------------

In the example above, the resources section specifies that the container needs one Nvidia GPU resource. You can adjust the number of GPUs by changing the value of nvidia.com/gpu. You can also specify the GPU type by using the nvidia.com/gpu-type resource limit.

You can apply this YAML file to your Kubernetes cluster using the kubectl apply -f command 

The most commonly used Kubernetes commands:

  1. kubectl create: creates a resource from a file or from stdin.
  2. kubectl apply: applies changes to an existing resource.
  3. kubectl get: retrieves information about one or more resources.
  4. kubectl describe: provides detailed information about a specific resource or a set of resources.
  5. kubectl delete: removes one or more resources from the cluster.
  6. kubectl logs: displays logs from a specific pod or container.
  7. kubectl exec: runs a command inside a container in a specific pod.
  8. kubectl port-forward: forwards one or more local ports to a pod.
  9. kubectl rollout: manages rolling updates of a deployment.
You can use the kubectl logs command with the -f option to stream logs in real-time, similar to the tail -f command. Here's the syntax for using kubectl logs with the -f option:

kubectl logs -f <pod-name> <container-name>
kubectl logs -f my-pod my-container

This will start streaming the logs from the my-container container in the my-pod pod in real-time. You can stop the log streaming by pressing Ctrl+C.


---------------------------------------------------------------------------------------------

Version of Volcano scheduler installed on K8s cluster is : 

$ kubectl get deployment -n volcano-system volcano-scheduler -o=jsonpath='{.spec.template.spec.containers[0].image}' | cut -d':' -f2
1.7
$

----------details ----------------
$ kubectl describe deployment -n volcano-system volcano-scheduler
Name:                   volcano-scheduler
Namespace:              volcano-system
CreationTimestamp:      Fri, 03 Feb 2023 00:38:14 -0500
Labels:                 app=volcano-scheduler
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=volcano-scheduler
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=volcano-scheduler
  Service Account:  volcano-scheduler
  Containers:
   volcano-scheduler:
    Image:      sachin-power-hpc-docker-local/volcanosh/vc-scheduler:1.7
----------------
NOTE: This command describes the deployment object for the Volcano Scheduler in the volcano-system namespace and provides detailed information about the deployment, including the image tag and version.

You can get the version of Kubernetes installed in your cluster by running the following command:

$ kubectl version --short | grep Server
Server Version: v1.23.4
$

After creating the job - we can check the job name as shown in Step 2 

Step 1 :  Submit Job - spec defines scheduler=volcano

[spb@k8s-masterNode]#kubectl create -f openmpi-example.yaml
job.batch.volcano.sh/nj-ompi-job created

Step 2: get the name of job submitted by volcano scheduler . So, It's vcjob 

[spb@k8s-masterNode]#kubectl get vcjob -o custom-columns=:.metadata.name
nj-ompi-job
[spb@k8s-masterNode]#
[spb@k8s-masterNode]#kubectl get vcjob -o custom-columns=NAME:.metadata.name
NAME
nj-ompi-job
[spb@k8s-masterNode]#

Step 3 : Just observe - how the vcjob changes it's state  from Pending ---> Running--->Completed

[spb@k8s-masterNode]#kubectl get vcjob
NAME          STATUS    MINAVAILABLE   RUNNINGS   AGE
nj-ompi-job   Pending   2                         5s
[spb@k8s-masterNode]#kubectl get vcjob
NAME          STATUS    MINAVAILABLE   RUNNINGS   AGE
nj-ompi-job   Running   2              2          14s
[spb@k8s-masterNode]#kubectl get vcjob
NAME          STATUS    MINAVAILABLE   RUNNINGS   AGE
nj-ompi-job   Running   2              3          37s
[spb@k8s-masterNode]#
[spb@k8s-masterNode]#kubectl get vcjob
NAME          STATUS      MINAVAILABLE   RUNNINGS   AGE
nj-ompi-job   Completed   2                         90s
[spb@k8s-masterNode]#


Step 4 : Based on the spec defined in yaml file - you can dispaly the vcjob 

[spb@k8s-masterNode]#kubectl get vcjob -o custom-columns='NAME:.metadata.name,MinAvailable:.spec.minAvailable,SCHED:.spec.schedulerName'
NAME          MinAvailable   SCHED
nj-ompi-job   2              volcano
[spb@k8s-masterNode]#

Step 5 : Get the status of VC job 

[spb@k8s-masterNode]#kubectl get vcjob nj-ompi-job -n kube-system -o jsonpath='{.status.conditions[?(@.status=="Completed")].status}'
Completed
[spb@k8s-masterNode]#
[spb@k8s-masterNode]#kubectl get vcjob nj-ompi-job -n kube-system -o jsonpath='{.status.conditions[-1:].status}'
Completed
[spb@k8s-masterNode]#

Get the status of a specific job:
kubectl get job.batch.volcano.sh job-name -o yaml

This command will display the job status in YAML format, which can be useful for debugging errors or issues with the job

Get the logs for a specific pod in the job
kubectl logs job-name-pod-name

Get detailed information about the job and its pods:
kubectl describe job.batch.volcano.sh job-name
kubectl describe pod job-name-pod-name

View the events for a specific job:

kubectl get events --field-selector involvedObject.name=job-name

Example:

$ kubectl get events --field-selector involvedObject.name=smpi-build-0 -n smpici
LAST SEEN   TYPE     REASON      OBJECT                                MESSAGE
20m         Normal   Scheduled   pod/smpi--build-0   Successfully assigned smpici/smpi-build-0 to workerNode
20m         Normal   Pulling     pod/smpi-build-0   Pulling image "sachin-power-hpc-docker-local/master:rhel_8.6"
20m         Normal   Pulled      pod/smpi-build-0   Successfully pulled image "sachin-power-hpc-docker-local/master:rhel_8.6" in 2.374936782s
20m         Normal   Created     pod/smpi--build-0   Created container build
20m         Normal   Started     pod/smpi-build-0   Started container build
$

View the resource usage of a specific pod:
kubectl top pod job-name-pod-name

This command will display the CPU and memory usage of a specific pod in the job, which can help with diagnosing performance issues or resource constraints.

Example : 
$ kubectl get pods -n smpici
NAME                              READY   STATUS    RESTARTS   AGE
smpi-37-build-0   1/1     Running   0          16m
$
$ kubectl top pod smpi-weekly-ibm-smpi-37-build-0 -n smpici
NAME                              CPU(cores)   MEMORY(bytes)
smpi-37-build-0                     2m           208Mi
$

Example : How to delete the pod that  completed successfully

$ kubectl get pods -n smpici  --field-selector=status.phase==Succeeded
NAME                                      READY   STATUS      RESTARTS   AGE
smpi-30-k8s-mpimaster-0   0/1     Completed   0          5d2h
$
$ kubectl delete pods -n smpici  --field-selector=status.phase==Succeeded
pod "smpi-30-k8s-mpimaster-0" deleted
$

-------------------------Vela, IBM’s first AI-optimized, cloud-native supercomputer.----------------------


IBM has announced the release of Vela, a new security technology and AI supercomputer. The technology is designed to assist organizations in detecting and responding to cyber threats in real time.

It is also IBM’s first AI-optimized, cloud-native supercomputer, designed for developing and training large-scale AI models. It combines various security tools and technologies to provide a unified view of a company’s security posture.

Vela is intended to assist organizations in improving their security posture by providing them with the information they need to quickly identify and respond to threats. The technology analyzes massive amounts of security data and provides actionable insights to security teams using artificial intelligence and machine learning algorithms. It integrates various security tools and technologies and employs artificial intelligence and machine learning algorithms to provide security teams with actionable insights.
https://research.ibm.com/blog/AI-supercomputer-Vela-GPU-cluster
source


A comparison of the traditional HPC software stack and Cloud-native AI stack are shown in diagram. But the downside of cloud native stack with virtualization, historically, is that it reduces node performance.


source


This work includes configuring the bare-metal host for virtualization with support for Virtual Machine Extensions (VMX), single-root IO virtualization (SR-IOV), and huge pages. With AI-optimized processor, the IBM AIU [Artificial Intelligence Unit] - The era of cloud-native AI supercomputing has only just begun. The IBM AIU is not a graphics processor. It was specifically designed and optimized to accelerate matrix and vector computations used by deep learning models. The AIU can solve computationally complex problems and perform data analysis at speeds far beyond the capability of a CPU. Deploying AI to classify cats and dogs in photos is a fun academic exercise. But it won’t solve the pressing problems we face today. For AI to tackle the complexities of the real world — things like predicting the next Hurricane /natural calamities or whether we’re heading into a recession — we need enterprise-quality, industrial-scale hardware. IBM AIU takes us one step closer. 

There are many real-world examples of Kubernetes clusters being used to run HPC workloads. Here are a few examples:

  • The National Energy Research Scientific Computing Center (NERSC), which is part of the US Department of Energy, uses Kubernetes to manage its HPC resources. NERSC uses Kubernetes to manage both traditional HPC workloads and machine learning workloads, and has reported significant improvements in resource utilization and efficiency since adopting Kubernetes.
  • Argonne National Laboratory's Theta supercomputer, which is one of the fastest supercomputers in the world, uses Kubernetes to manage its containerized workloads. Theta uses Kubernetes to run a variety of scientific simulations, including simulations of earthquakes and climate models.
  • Oak Ridge National Laboratory's Summit supercomputer, which is the most powerful supercomputer in the world, also uses Kubernetes to manage its containerized workloads.Summit uses Kubernetes to run scientific simulations and other HPC workloads. The Summit supercomputer at Oak Ridge National Laboratory uses IBM Spectrum MPI as its primary MPI implementation for running parallel workloads. It is is capable of running a variety of HPC workloads using MPI, including simulations, data analysis, and machine learning applications.
  • The University of Cambridge's High Performance Computing Service uses Kubernetes to manage its HPC resources. The service uses Kubernetes to run a variety of HPC workloads, including simulations of fluid dynamics and other scientific applications.
One of the most commonly used container runtime engines on Summit is Singularity, which is an open-source container platform designed specifically for HPC workloads. Singularity is optimized for running MPI workloads, and provides a number of features specifically tailored to HPC, including support for Infiniband networking and the ability to run on a wide range of HPC systems. In addition to Singularity, Summit also supports other container runtime engines, including Docker and Shifter. However, Singularity is the preferred container runtime engine for running HPC workloads on Summit, and is used by many researchers and organizations to run MPI workloads.

There are a number of emerging container runtime technologies that are being developed for use in HPC environments. One example is Charliecloud, which is an open-source container runtime that is designed specifically for HPC workloads. Charliecloud provides a lightweight and secure environment for running containerized workloads on HPC systems, and is optimized for performance and flexibility.

Another emerging technology is called udocker, which is a user-space container engine that is designed to provide a simple and lightweight way to run containerized workloads on HPC systems. Udocker is optimized for use in environments where users may not have root access to the system, and provides a number of features specifically tailored to HPC workloads, including support for MPI and Infiniband networking.

Frontier is a new supercomputer that is currently being developed at Oak Ridge National Laboratory. 
One of the key innovations of the Frontier supercomputer is its use of new "next-generation" runtime engines that are being developed specifically for HPC workloads. These new runtime engines are designed to provide improved performance, scalability, and flexibility compared to traditional runtime engines like MPI. One of the most promising new runtime engines being developed for Frontier is called Unified Parallel C (UPC). UPC is a high-level parallel programming language that is designed to simplify the development of parallel algorithms for HPC workloads. It provides a number of features specifically tailored to HPC, including support for shared memory programming and optimized communication primitives.

AWS provides a wide range of compute, storage, and network resources that are optimized for running HPC workloads in the cloud. By leveraging these services, researchers and engineers can easily and cost-effectively run complex simulations and other HPC workloads, while also benefiting from the scalability and flexibility of cloud computing. Some examples:
  • Amazon EC2: This is AWS's flagship compute service, which allows users to rent virtual machines (instances) with a variety of different configurations and capabilities. EC2 provides a wide range of instance types that are optimized for different workloads, including HPC workloads. These instance types offer high-performance CPUs, GPUs, and FPGAs, as well as high-speed network connectivity.
  • Amazon Elastic File System (EFS): This is a fully managed cloud file storage service that is designed for HPC workloads. EFS provides a scalable and highly available file system that can be accessed from multiple instances simultaneously, which is important for parallel computing workloads.
  • Amazon S3: This is a highly scalable and durable object storage service that can be used to store and retrieve large data sets for HPC workloads. S3 provides a simple API that can be used to access data from anywhere, and supports a variety of data formats and access patterns.
  • AWS ParallelCluster: This is an open-source HPC cluster management tool that can be used to deploy and manage HPC clusters on AWS. ParallelCluster provides a simple interface for configuring and launching HPC clusters, and supports a variety of different schedulers and software packages.
  • Amazon FSx for Lustre: This is a fully managed file system service that provides high-performance Lustre file systems for HPC workloads. FSx for Lustre provides scalable performance and high availability, and can be used to store and manage large data sets for parallel computing workloads.
To run weather data analysis using HPC cloud services on AWS, you would need to follow these general steps:
  • Provision the necessary compute resources: Using services such as Amazon EC2, you would need to launch instances that are optimized for your specific workload. For weather data analysis, you may require high-performance CPUs, GPUs, or FPGAs, as well as high-speed network connectivity to move data in and out of the instances.
  • Store the data: You would need to store the weather data in a highly scalable and durable storage system. Amazon S3 is a popular choice for storing large data sets in the cloud.
  • Configure the software stack: Once the compute resources and data storage are set up, you would need to configure the software stack. This would involve installing and configuring the necessary software packages, including any libraries or tools that are required for weather data analysis.
  • Submit the analysis jobs: You would then submit the analysis jobs to the HPC cluster using a batch scheduler such as Slurm or AWS ParallelCluster. The scheduler will manage the allocation of compute resources and ensure that the jobs are executed in a timely and efficient manner.
  • Retrieve the results: Once the analysis jobs are completed, you would retrieve the results from the storage system and perform any post-processing or analysis that is necessary.
Additionally, there are a number of third-party tools and services available that are specifically designed for weather data analysis in the cloud, such as the Unidata Cloud-Ready Data Services, which provides a suite of cloud-based tools and services for working with weather data. There are some configuration details for a typical HPC software stack that might be used for the above example -weather data analysis :
  • Operating System: Many HPC workloads run on Linux-based operating systems, such as CentOS or Ubuntu. These operating systems are generally lightweight and optimized for high-performance computing. Linux distributions, such as CentOS and Ubuntu, are commonly used in HPC environments, particularly in academic and research settings where open source software is often preferred. RHEL (Red Hat Enterprise Linux) and SLES (SUSE Linux Enterprise Server) are actually commonly used operating systems in HPC environments, particularly in the commercial sector. Both RHEL and SLES are enterprise-grade operating systems that offer long-term support and stability, which are important features in HPC environments where system uptime and reliability are critical.
  • Cluster Management Software: There are a variety of cluster management software packages available for HPC workloads, including Slurm, Torque, and LSF. These tools provide a way to manage the allocation of compute resources and schedule jobs on the cluster.
  • MPI Library: For parallel computing workloads, you would typically need to install a Message Passing Interface (MPI) library, such as OpenMPI or MPICH. These libraries allow multiple processes to communicate with each other and coordinate their work on the cluster.
  • Compiler Toolchain: A compiler toolchain is necessary for building and executing code on the cluster. Commonly used compilers include GCC, Clang, and Intel Compiler.
  • Data Processing Libraries: For weather data analysis, you would typically need to install a variety of data processing libraries, such as NetCDF, HDF5, and GRIB. These libraries allow you to read, write, and manipulate weather data in a variety of formats.
  • Visualization Tools: Once the weather data analysis is complete, you may want to visualize the results using tools such as Matplotlib, Paraview, or Visit. These tools allow you to create visualizations and animations that can help you better understand the results of your analysis.
Steps to be followed in AWS to run HPC workloads :
  • Choose an AWS instance: Select an Amazon EC2 instance type that suits your computational needs. EC2 provides a range of instance types with varying CPU, memory, and storage capacities. For a weather data analysis, you may need a high-memory instance type with multiple CPUs and GPUs, depending on the size of your data.
  • Install software dependencies: Install the necessary software dependencies on your EC2 instance, such as the operating system, compilers, libraries, and analysis tools. You can either install these manually or use a configuration management tool like Ansible or Chef to automate the process.
  • Move data to AWS: Transfer your weather data from your on-premises environment to the AWS cloud using AWS Storage services such as Amazon S3 or EFS. You can also use AWS Direct Connect to establish a dedicated network connection between your on-premises environment and your AWS resources.
  • Configure your software: Set up your weather analysis software, including the input/output paths, data formats, and algorithm parameters. You can run your analysis either on a single EC2 instance or on a cluster of instances using HPC job schedulers like Slurm or LSF.
  • Monitor and optimize performance: Monitor the performance of your weather analysis to identify bottlenecks and optimize performance. You can use AWS CloudWatch to monitor CPU, memory, and network utilization, as well as application-level metrics like response times and error rates.
  • Generate reports and visualizations: Once your analysis is complete, you can generate reports and visualizations using tools like Matplotlib, Paraview, or Visit. You can save the results to Amazon S3 or EFS, or use other AWS services like AWS Lambda to trigger alerts or actions based on the results.
Some of the commonly used HPC workloads on AWS cloud :
  • Genomics and bioinformatics: AWS offers a range of genomic and bioinformatics tools and services that can handle large-scale data processing and analysis tasks. These include tools for sequence alignment, variant calling, gene expression analysis, and genome assembly.
  • Computational chemistry and materials science: AWS provides a range of high-performance computing (HPC) resources, such as GPU-enabled instances, that can handle complex simulations and calculations. These resources can be used to run molecular dynamics simulations, quantum chemistry calculations, and other computational chemistry and materials science workflows.
  • Financial modeling and simulation: AWS provides a range of compute and storage resources that can be used for financial modeling and simulation tasks. These resources can be used to run Monte Carlo simulations, backtesting, portfolio optimization, and other financial analysis workflows.
  • Weather and climate modeling: AWS provides a range of weather and climate data services, such as Amazon Forecast, that can be used to generate forecasts and predictions. These services can be used to run weather and climate simulations, analyze large datasets, and generate forecasts for various applications.
  • Machine learning and AI: AWS provides a range of machine learning and AI services, such as Amazon SageMaker and Amazon Rekognition, that can be used to train and deploy machine learning models. These services can be used for natural language processing, image and video analysis, and other AI workflows.
Area of research in HPC is the development of container-based cloud environments that are designed to support scientific workloads. These environments aim to provide a flexible and scalable platform for running scientific applications, while also addressing challenges such as data movement, job scheduling, and resource management. As mentioned earlier ,  Singularity is designed to support the needs of scientific and HPC workloads, and provides features such as secure image management, container mobility, and seamless integration with HPC schedulers. Another area of research is the development of new programming models and frameworks that are designed to simplify the development of HPC applications in container-based environments. For example, the Big Data and Extreme Computing (BDEC) project is developing a framework that provides a high-level abstraction layer for managing and orchestrating distributed scientific workflows. The project has the potential to make it easier and more efficient to manage large-scale scientific workflows in a variety of domains.

Friday, June 10, 2022

Frontier supercomputer powered by AMD is the fastest and first exascale machine

Exascale computing is the next milestone in the development of supercomputers. Able to process information much faster than today’s most powerful supercomputers, exascale computers will give scientists a new tool for addressing some of the biggest challenges facing our world, from climate change to understanding cancer to designing new kinds of materials. 

One way scientists measure computer performance is in floating point operations per second (FLOPS). These involve simple arithmetic like addition and multiplication problems. Their performance in FLOPS has so many zeros - researchers instead use prefixes like Giga, Tera, Exa. where “Exa” means 18 zeros. That means an exascale computer can perform more than 1,000,000,000,000,000,000 FLOPS, or 1 exaFLOP. DOE is deploying the United States’ first exascale computers: Frontier at ORNL and Aurora at Argonne National Laboratory and El Capitan at Lawrence Livermore National Laboratory.

Exascale supercomputers will allow scientists to create more realistic Earth system and climate models. They will help researchers understand the nanoscience behind new materials. Exascale computers will help us build future fusion power plants. They will power new studies of the universe, from particle physics to the formation of stars. And these computers will help ensure the safety and security of the United States by supporting tasks such as the maintenance of  US nuclear deterrent.

For decades, the performance maximization has been the chief concern of both the hardware architects and the software developers.  Due to end of performance scaling by increasing CPUs clock frequencies (i.e Moore's Law),  Industries transit from single-core to multi-core and many-core architectures.  As a result, the hardware acceleration and the use of co-processors together with CPU are becoming a popular choice to gain the performance boost while keeping the power budget low. This includes both the new customized hardware for particular application domain such as Tensor Processing Unit (TPU), Vision Processing Unit (VPU) and Neural Processing Unit (NPU); and the modifications in existing platforms such as Intel Xeon Phi co-processors, general purpose GPUs and Field Programmable Gate Array (FPGA)s. Such accelerators together with main processors and memory, constitute a heterogeneous system. However, this heterogeneity has raised unprecedented difficulties posed to performance and energy optimization of modern heterogeneous HPC platforms. 

The focus of maximizing the performance of HPC in terms of completing the hundreds of trillion Floating Point Operations Per Second (FLOPS) has led the supercomputers to consume an enormously high amount of energy in terms of electricity and for cooling down purposes. As a consequence, current HPC systems are already consuming Megawatts of energy. Energy efficiency is becoming an equally important design concern with performance in ICT.  Current HPC systems are already consuming Megawatts of energy. For example, the world’s powerful supercomputer like Summit consumes around 13 Megawatts of power which is roughly equivalent to the power draw of roughly over 10000 households. Because of such high power consumption, future HPC systems are highly likely to be power constrained. For example, DOE aims to deploy this exascale supercomputer capable of performing 1 million trillion ( or 1018) floating-point operations per second in a power envelope of 20-30 megawatts. Initial target was to deliver a double precision exaflops of compute capability for 20 megawatts of power and other target was  2 exaflops for 29 megawatts of power when it’s running at full power. Taking into consideration the above-mentioned factors, HPE Cray designed Frontier supercomputer powered by AMD for growing accelerated computational needs and power constraints.

The Frontier supercomputer, built at the Department of Energy's Oak Ridge National Laboratory in Tennessee, has now become the world's first known supercomputer to demonstrate a processor speed of 1.1 exaFLOPS (1.1 quintillion floating point operations per second, or FLOPS).  The Frontier supercomputer's exascale performance is enabled by  world's most advanced pieces of technology from HPE and AMD.

Frontier supercomputer powered by AMD is  the first exascale machine meaning it can process more than a quintillion calculations per second with an HPL score of 1.102 Exaflop/s. Based on the latest HPE Cray EX235a architecture and equipped with AMD EPYC 64C 2GHz processors, the system has 8,730,112 total cores and a power efficiency rating of 52.23 gigaflops/watt. It relies on gigabit ethernet for data transfer. 

Exascale is the next level of computing performance. By solving calculations five times faster than today’s top supercomputers—exceeding a quintillion [ or 1018  ] calculations per second—exascale systems will enable scientists to develop new technologies for energy, medicine, and materials. The Oak Ridge Leadership Computing Facility will be home to one of America’s first exascale systems, Frontier, which will help guide researchers to new discoveries at exascale.

It's based on HPE Cray’s new EX architecture and Slingshot interconnect with optimized 3rd Gen AMD EPYC™ CPUs for HPC and AI, and AMD Instinct™ 250X accelerators. It delivers linepack (double precision floating point – FP64) compute performance of 1.1 EFLOPS (ExaFLOPS). 

Source

The Frontier test and development system (TDS) secured the first place in the Green500 list, delivering 62.68 gigaflops/watt power-efficiency from a single cabinet of optimised 3rd Gen AMD EPYC processors and AMD Instinct MI250x accelerators. It could lead to breakthroughs in medicine, astronomy, and more.  


The HPE/AMD system delivers 1.102 Linpack exaflops of computing power in a 21.1-megawatt power envelope, an efficiency of 52.23 gigaflops per watt. Frontier only uses about 29 megawatts at its very peak. During a test, Frontier ran at 1.1 exaflops and could go as high as 2 exaflops.

Source


Node diagram:

Source

These are HPE Cray EX systems has  74 cabinets of this — 9,408 nodes. Each node has one CPU and four GPUs. The GPUs are the [AMD] MI250Xs. The CPUs are an AMD Epyc CPU. It’s all wired together with the high-speed Cray interconnect, called Slingshot. And it’s a water-cooled system.  Recently  good efforts towards using computational fluid dynamics to model the water flow in the cooling system. These are incredibly instrumented machines with liquid cooling dynamically adjust to the workloads.  There’s sensors that are monitoring temperatures where even down to the individual components on the individual node-boards, so they can adjust the cooling levels up and down to make sure that the system stays at a safe temperature. It was estimated to provide over 60 gigaflops-per-watt for the single cabinet run.

This is the datacenter where they formerly had the Titan supercomputer. So they removed that supercomputer and refurbished this datacenter. That needed more power and  more cooling. So they brought in 40 megawatts of power to the datacenter and  have 40 megawatts of cooling available. Frontier really only uses about 29 megawatts of that at its very peak. And this Supercomputer is even a little bit quieter than Summit  because they’re going to liquid-cooled with no fans and no rear doors where  exchanging heat with the room. It’s 100 percent liquid cooled, and the [fan] noise generated from storage systems that are also HPE and are air-cooled.

At OLCF [Oak Ridge Leadership Computing Facility], they  have the Center for Accelerated Application Readiness, we call it CAAR. Its  vehicle for application readiness. That group supports eight apps for the OLCF and 12 apps for the Exascale Computing Project. Frontier was OLCF-5, the next system will be OLCF-6.

The result was confirmed in a benchmarking test called High-Performance Linpack (HPL). As impressive as that sounds, the ultimate limits of Frontier are even more staggering, with the supercomputer theoretically capable of a peak performance of 2 quintillion calculations per second. Among all these massively powerful supercomputers, only Frontier has achieved true exascale performance, at least where it counts, according to TOP500. Some of the most exciting things are the work in artificial intelligence and those workloads. Plan for research teams  to develop better treatments for different diseases, how to improve efficacies of treatments, and these systems are capable of digesting just incredible amounts of data. Thousands of  laboratory reports or pathology reports,  can draw inferences across these reports that no human being could ever do but that a supercomputer can do. They still have Summit here, a previous Top500 number-one system, an IBM/Nvidia machine. It’s highly utilized at this point. They will at least run it for a year and overlap with Frontier  so that we can make sure that Frontier is up and stable and give people time to transition their data and their applications over to the new system.

With the Linpack exaflops milestone achieved by the Frontier supercomputer at Oak Ridge National Laboratory, the United States is turning its attention to the next crop of exascale machines, some 5-10x more performant than Frontier. At least one such system is being planned for the 2025-2030 timeline, and the DOE is soliciting input from the vendor community to inform the design and procurement process. That can solve scientific problems 5 to 10 times faster – or solve more complex problems, such as those with more physics or requirements for higher fidelity – than the current state-of-the-art systems. These future systems will include associated networks and data hierarchies. A capable software stack will meet the requirements of a broad spectrum of applications and workloads, including large-scale computational science campaigns in modeling and simulation, machine intelligence, and integrated data analysis. They expect these systems to operate within a power envelope of 20–60 MW. These systems must be sufficiently resilient to hardware and software failures, in order to minimize requirements for user intervention. This could include the successor to Frontier (aka OLCF-6), the successor to Aurora (aka ALCF-5), the successor to Crossroads (aka ATS-5), the successor to El Capitan (aka ATS-6) as well as a future NERSC system (possibly NERSC-11). Note that of the “predecessor systems,” only Frontier has been installed so far. A key thrust of the DOE supercomputing strategy is the creation of an Advanced Computing Ecosystem (ACE) that enables “integration with other DOE facilities, including light source, data, materials science, and advanced manufacturing. The next generation of supercomputers will need to be capable of being integrated into an ACE environment that supports automated workflows, combining one or more of these facilities to reduce the time from experiment and observation to scientific insight.

The original CORAL contract called for three pre-exascale systems (~100-200 petaflops each) with at least two different architectures to manage risk. Only two systems – Summit at Oak Ridge and Sierra at Livermore – were completed in the intended timeframe, using nearly the same heterogeneous IBM-Nvidia architecture. CORAL-2 took a similar tack, calling for two or three exascale-class systems with at least two distinct architectures. The program is procuring two systems – Frontier and El Capitan – both based on a similar heterogenous HPE AMD+AMD architecture. The redefined Aurora – which is based on the heterogenous HPE Intel+Intel architecture – becomes the “architecturally-diverse” third system (although it technically still belongs to the first CORAL contract).

-----

Reference:

https://www.olcf.ornl.gov/wp-content/uploads/2019/05/frontier_specsheet.pdf

https://www.researchgate.net/figure/Power-consumption-by-top-10-supercomputers-over-time-based-on-the-results-published_fig2_350176475

https://www.youtube.com/watch?v=HvJGsF4t2Tc