High Performance Computing
The Cloud is a great place and enabler of High-Performance Computing (HPC) used in today's digital world or genomics (one critical use case that may be interesting to you).
HPC enables Scientists/ researchers their on-demand clusters of EC2 instances. HPC minimizes the research timeline by many folds and makes innovating fun. For instance, the researcher can perform a series of process steps among several "TO BE TESTED" configurations before zero on a single path-breaking solution used in the medicines and healthcare marketplace.
All of this experimentation in minutes as opposed to time-intensive set-up requiring days and at times months to set up & configure the experiment (Non-Core to a researcher) is being enabled by using high-performance Computing.
High-Performance Computing (HPC) use cases cut across various domains such as genomics (more later), computational chemistry, financial risk modeling, weather prediction, machine learning, deep learning, autonomous driving, cancer research, gene therapies for genetic disorders.
HPC collaboration on large data sets and research using statistical models (one of the tools) in no time. For example, providing a mini-cluster on which the researcher in the science domain can perform their research to develop a viable solution/ product in no time by spinning a cluster on demand Vs. Months of test and capital intensive trials is a game-changer.
The path-breaking trials can now be done in minutes instead of days/ months, creating the much-needed differentiation and speeding up the experimentation process was made possible using HPC.
HPC is an elitist combination of Storage, network, and Compute provided at scale in a researcher's hands on-demand.
HPC is an amalgamation of technologies comprising & not limited to :
CSDN: Compute, Storage, Networking, Data transfer and Orchestration
· AMI/ EC2 instances: AWS has been endeavoring to optimize CPU utilization through computing optimization using Technology.
· Spot Fleet (On-Demand + SPOT instances) combination for cost optimization, performance enhancement
· Cluster Placement Group: This entails housing several connected instances within a single AZ leading to better communication patterns & performance.
· EC2 Enhanced SRIOV: Elastic Network Adapter (ENA) releases the pressure on EC2 instances by enhancing throughput and higher IOPS and lower latency. Elastic Fabric Adapter (EFA) is the next ENA version and provides better performance by releasing the compute instance's strain by bypassing the LINUX OS using Single root IO Virtualization (SRIOV). However, EFA is limited to a Linux cluster.
· Moving data sets is a key issue and so is the storage. As we constantly speak of the adage that data has gravity ie. Big data attracts applications, network and more data. Network storage constituting FSx for Linux cluster, S3 -Transfer acceleration, and EFS have significantly improved the EC2 performance required for HPC solutions. EBS (instance storage) and Instance store (Better IOPS) maximize IOPS and Throughput required by enterprise-grade applications.
· Data Transfer finds a way into HPC requirements through AWS Direct Connect (A faster option than STS VPN), providing 10 & 50 Gbps, 50 and 100 Mbps rate of transfer. For Petascale data transfers, the enterprise looks to AWS for Snowmobile and Snowball (a lesser version of the previous). AWS DataSync through an Agent enables data transfer to EFS, S3, and FSX from an NFS or EFS to EFS configurations.
· Orchestration provides AWS Parallel cluster and the use of AWS Batch (think repetitive background processes).
In short, High-Performance computing is a game-changer enabling experimentation and raising your "innovation game" to a whole new level. Happy computing.
A great Use case: aws.amazon.com/scico or https://aws.amazon.com/government-education/research-and-technical-computing/?wwps-cards.sort-by=item.additionalFields.sortDate&wwps-cards.sort-order=desc
