Understanding AWS CloudWatch

AWS CloudWatch is a tool from Amazon that allows system architects, developers, and administrators to help them perform real-time monitoring of their AWS applications hosted on the cloud. The software is automated and configured to give metrics on latency, request counts, CPU usage, etc.

Codebase

According to our research, CloudWatch code is based on telegraf agent. Because of the similar structure of the launch log and config file. More info here

The need for AWS CloudWatch

The CloudWatch software also supports custom metrics, and users can push their logs directly to CloudWatch for monitoring purposes. The reports and data that the software generates allow administrators to track resource usage, application performance, any constraints, and operational issues. In other words, the tool is critical for foreseeing and resolving technical issues well in time to streamline organization-wide operations.

However, the most significant benefit of CloudWatch is that it is done to integrate automatically with Amazon Web Services (AWS) while being scalable and flexible.

Features of AWS CloudWatch

The Amazon CloudWatch is equipped with salient features, which are:

Data Collection

The AWS CloudWatch allows users to collate and store real-time data logs from different apps, resources, and services. Typically, these include vendor logs, logs published under the AWS services banner, and custom-designed logs. The logs can be quickly published by installing the AWS CloudWatch agent.

The software comes with many built-in metrics, but users can also customize as per their requirements. You can aggregate the collected container logs and metrics within the ecosystem as well.

Monitoring

The AWS CloudWatch dashboards allow system administrators to visualize their resource performance on the cloud by creating reusable visual graphs. The central dashboard offers logs data and graph metrics in a unified view so that you can act on the issue in a faster manner. Gaining knowledge of the context is key to understanding and resolving the root cause of the problem.

AWS CloudWatch also simplifies the analysis of between metrics and logs so that you can move on from diagnosis to resolution.

Automation

The AWS CloudWatch comes equipped with Auto Scaling to automate the application’s resource planning and capacity monitoring. You can set alarms and even trigger automated action.

With CloudWatch Events' help, users can view a real-time stream of events describing any changes to the AWS resources. As a result, your response time reduces, and you can correct the issues faster.

Analysis

With AWS CloudWatch, users can view trends in real-time and retain nearly 15 months of data for analysis purposes. This historical data can prove to be the key to fine-tuning the utilization of resources. The software also provides granular data in real-time, thereby creating better visualization and analysis to optimize application performance.

Security

The AWS CloudWatch software is predesigned to integrate with IAM — AWS Identity and Access Management. As a result, the system administrators can define which resources or users can access the controls and data. Further, the data is encrypted while it is being transferred, also ensuring security and compliance.

Advantages of AWS CloudWatch Agent

The advantages of using AWS CloudWatch agent include:

● The convenience of switching between essential monitoring and detailed monitoring depending on your automated monitoring requirements.

● The same software is capable of monitoring other resources as well.

● You have the ease of creating custom metrics and monitoring using these for your particular applications. These metrics can be made using basic API requests.

● You can automatically troubleshoot, maintain, and store affected log files in case of an error. The views can be represented in graphs for better understanding.

● As one of the best software for checking and monitoring cloud resources, the AWS CloudWatch agent enables quick response and effective monitoring in one single package.

Understanding Telegraf

Telegraf is an agent for collecting, aggregating, processing, as well as writing metrics. It is an open-source server agent that helps collect data from stacks, systems, and IoT sensors.

The need for Telegraf

Telegraf’s primary purpose is data collection and transmission. The software performs this employing different components, including databases, systems, and IoT sensors. The software can connect to various data sources like MySQL, MongoDB, Redis, etc., to collate and transmit metrics. Metrics can then be collected from a varied stack of cloud platforms, orchestrators, and containers. It also enables collecting critical stateful data like temperature levels, pressure levels, etc., by connecting with various IoT devices and sensors.

Coverage

Telegraf simplifies metric collection from your endpoints as it comes with more than 200 plugins defined by SMEs (subject matter experts) from the community. Designing and building plugins are straightforward, meaning that you can develop customized plugins that cater to your particular monitoring requirements. Above all, the Telegraf monitoring software can be used to parse any input data formats directly into metrics, including JSON, InfluxDB Line Protocol, Value, Graphite, Collectd, and Nagios.

Agent

Plugins drive Telegraf for data collection as well as data output making it easily extendable. The software application can collect metrics covering a wide range of input avenues and code them into an equally wide range of outputs. Written in Go, the code is a standalone, compiled binary that can be implemented without external dependencies. In other words, it does not require npm, gem, pip, or such package management tools.

Convenience

The Telegraf plugin architecture comes with the flexibility to support your internal organizational processes. You need not modify your workflows to suit the application’s technology. It can be a centralized platform, or you can place it on the system's edge to fit your process requirements. In other words, it is incredibly convenient to implement.

Advantages of Telegraf

Like AWS CloudWatch, Telegraf comes with several advantages and negligible disadvantages.

● Like AWS CloudWatch, the Telegraf open-source monitoring solution is also a cohesive stack catering to all your monitoring requirements in one package.

● A single Telegraf agent is capable of functioning as multiple exporters with bare minimum handling requirements.

● The open-source platform is readily available to one and all and is continuously updated by the community with the latest developments.

● Telegraf agent comes with a wide array of metrics, and users can customize these to suit their organizational monitoring requirements.

● The software also comes with its set of multiple plugins that are more than enough for most organization-wide requirements. However, in case more is required, then additional plugins can be customized using different languages.

Popular CPU Metrics for AWS CloudWatch and Telegraf Agent

Both AWS CloudWatch and Telegraf come with an extension of metrics that support their monitoring capabilities.

Some popular CPU metrics are:

● cpu_time_active

This metric refers to the amount of time for which the CPU remains active irrespective of the capacity. These monitoring software tools are capable of measuring this metric in hundredths of one second.

● cpu_time_guest

This metric refers to the amount of time for which the CPU operates a virtual CPU for guest OS (operating system). Like most other monitoring metrics, this metric is also measured one-hundredths of a second.

● cpu_time_guest_nice

This metric refers to the total amount of time for which the CPU operates a virtual CPU for a guest OS. Typically, this operating system is low in priority. As a result, it can be interrupted by other more critical processes. Its measurement is done in hundredths of a second.

● cpu_time_nice

This metric refers to the total amount of time for which the CPU is in actual user mode while operating low priority processes. These are processes that can be interrupted by other higher-priority methods quickly. The tools are capable of measuring this metric in hundredths of a second for efficient monitoring.

● cpu_usage_active

As the name suggests, this metric refers to the total percentage of time for which the CPU remains active in any capacity. The percentage is the unit of measurement for this metric.

Other popular CPU metrics include:

● time_user (float)

● time_system (float)

● time_idle (float)

● time_active (float)

● time_nice (float)

● time_iowait (float)

● time_irq (float)

● time_softirq (float)

● time_steal (float)

● time_guest (float)

● time_guest_nice (float)

● usage_user (float, percent)

● usage_system (float, percent)

● usage_idle (float, percent)

Conclusion

In all instances where applications are deployed on Amazon Web Services (AWS), users have pre-configured AWS CloudWatch for automated monitoring and insights.

There are a few things to keep in mind:

• AWS CloudWatch agent will collect metrics like a disk, network, and CPU utilization, and it cannot measure EC2 memory consumption or default disk utilization.

This gap is effectively filled in by Telegraf, which is highly suitable for EC2 memory usage monitoring. Hence, both AWS CloudWatch and Telegraf can be complementary to each other when implemented efficiently.

• AWS CloudWatch can be expensive.
• AWS CloudWatch was set up for Amazon Cloud and isn’t working with Google Cloud, Oracle Cloud, IONOS Cloud, and others.
• Telegraf has higher flexibility in customizing than AWS CloudWatch.

So, RocketCompute.com recommends that better attention be paid to Telegraf. In our team, we have many highly specialized and technical professionals. We can help with implementation, custom plugins building, and metrics selection for monitoring.

***

Vladimir Kobzev, RocketCompute

Intro

Containerization technology is greatly simplified GPU computations for machine learning by wrapping software stack above Kernel level in containers and allowing to juggle with different combinations of frameworks, low level libraries, and hardware drivers. Technologies like nvidia-docker even unlocked new stack combinations (driver plus CUDA toolkit) that were not feasible before. This, however, adds another dimension to the performance optimization problem, and now you not only need to choose optimal hardware for your machine learning task but also variate driver/CUDA combination to fine-tune the performance further.

The goal of this short article is to illustrate that such optimization is not only feasible but also makes sense.

Experiment

The exercise is very similar to what has been done in my first article:

Workload: Mask R-CNN model trained on COCO dataset. I used reference implementation from MLCommons set. It is built on PyTorch 1.0.1 which supports both CUDA v.9.0 and v.10.0 and you can switch between these two versions without changing any line of code. I tweaked the number of training steps to reduce benchmark time to 20–90 minutes depending on GPU.

Infrastructure: I took one cheapest single-GPU instance for each GPU family available on AWS. There were four of them: p2.xlarge, g3s.xlarge, p3.2xlarge and g4dn.xlarge.

Stack: For installing different driver versions I used the approach that I described in the previous article. Framework together with the corresponding CUDA toolkit was packaged in a docker image together with the benchmark code.

For each set of {GPU instance, driver, CUDA} I did at least three runs.

Results

Below figures shows how benchmark runtime in seconds (Y axis) depends on device driver version (X axis) for each GPU. Curves correspond to a certain CUDA toolkit version (red for v9.0 and blue for v10.0). Each dot on a graph equals to average runtime and the vertical interval shows the standard
error around this mean value. Each point aggregates at least 3 independent benchmark runs.

Instances with K80 (Figure 1) and M60 (Figure 2) show the smallest variation from the driver version (8% and 4% correspondingly). For M60 overall variation of the performance comparable to the variation for a particular driver which does not allow to make any conclusion regarding the best performing combination. For the K80, however, CUDA v.10.0 and driver v.418 on average better than any other combination.

Instances with V100 revealed the most interesting results (Figure 3). This GPU shows the highest variation of the benchmark score (15%) which is significantly higher than the variation of most individual software stacks. On average there is a clear optimum (CUDA v.10.0 and driver v.418) and a clear worst performing stack (CUDA v.9.0 and driver v.384). The difference between these two combinations can exceed 15% (the highest gap we saw between individual data points was c.20%)

For T4 (Figure 4), which is the most modern GPU in the set, it is hard to say which combination is the best, despite significant variation of performance (14%). However, there is a clear loser (CUDA v.9.0 and driver v.410). It is
safe to say that recent drivers show comparable performance. This graph missing the result for driver v.384 as this version of the driver is not compatible with this GPU family.

Conclusion

As you can see even a quick straightforward check shows that the right driver/CUDA combination can save up to 15–20% of time/cost and add another layer of optimization after choosing the optimal GPU/instance for the job.

It potentially can cause an even bigger impact for inference workloads by reducing both the amount of compute required and latency of each request.

This exercise was just a quick check of the idea and more sorrow research will follow covering the latest versions of CUDA, drivers, and neural net architectures.

Egor Bykov

RocketCompute

https://rocketcompute.com/

Intro

There are a lot of guides on how to install Nvidia drivers in the wild, but still, when I needed to set-up a system where I can easily switch between driver versions for our research, I spent several days trying to figure out the most seamless way to achieve that.

Here I want to summarize multiple days of my googling, trials, errors, and the final solution I came up with which appeared to be way simpler than I thought.

I am using Ubuntu distro as a reference, the process should be the same for any version above 16.04. For other distros, the main part of the installation process should be similar with some differences in getting the required system packages and handling already installed graphics drivers.

Necessary Background

Below is a summary of my findings that helped me better understand the issue and why certain combinations of Nvidia low level libraries work and others do not.

Each driver version was developed against a certain version of Linux Kernel (which, I believe was mainstream at the time of development). This means that you have almost no chances to install old drivers on Linux distros with more modern kernel and it doesn’t matter which way of installation you choose, most probably such drivers just do not exist (e.g. I was not able to install drivers older than version 440 on Ubuntu 18.04 with kernel 4.15 or higher).

However, due to the backward compatibility of Linux kernels, you can install newer drivers on older kernels (e.g. version 450 or even higher on Ubuntu 16.04)

If you plan to install CUDA libraries on your host as well you do not need to install the driver separately as it goes together with the CUDA package due to certain compatibility requirements. CUDA installation deserves a separate article but can be done similarly.

If you plan to run CUDA dependent code inside a container you do not need to install the CUDA toolkit at all. The only requirement is that the driver package contains the library required to run a CUDA-enabled container (later I will show how to check that).

Installation Process

Download installer from Nvidia site.

According to my observations, there is no difference in what product or product type you are choosing, you will get the same installation file. The only difference is if you choose “Data Center / Tesla” instead of e.g. “GeForce” you will get a drop-down for the CUDA Toolkit version. Don’t be confused you will not get CUDA Toolkit together with the driver. Varying CUDA versions you will get a minimal driver version compatible with chosen CUDA Toolkit version. (e.g. if you pick 10.0 as the CUDA version you will get driver v.410, for CUDA v.11.0 you will get driver v.450).
Be aware that your GPU generation should be compatible with chosen CUDA Toolking version. The older you GPU the more options you have in terms of driver/CUDA versions.

After setting up all parameters you will download a self-installation archive. This archive contains pre-compiled libraries and source files necessary to build kernel modules. To build the driver you will need the following packages:

• gcc (C compiler)
• make (utility used to automate compilation process)
• linux-headers (kernel source files necessary to build kernel modules)

One line installation:

$ sudo apt install gcc make linux-headers-$(uname -r)

“$(uname -r)” will add the kernel version of your system to the package name to grab exactly what is required in your case.

The next steps will depend on your particular set-up.

Headless Server without pre-installed drivers

This is the simplest case, you have a VM server with no windows manager (X server) booted from a pre-built image with no drivers (like AWS AMI without pre-installed drivers) and no X server running. Then you can simply run:

$ sudo ./NVIDIA-Linux-x86_64–450.80.02.run --silent

“--silent” option will turn off the interactive UI and will use default answers to all questions that are usually asked. No output except error messages will be printed.

After that you are good to go, no reboot required. You can verify that installation went well by listing loaded kernel modules having “nvidia” in its name:

$ lsmod | grep nvidia

you should get 3 modules: nvidia, nvidia_drm, and nvidia_modeset. Or you can run:

$ nvidia-smi

which will show the current driver version, and list of all GPUs present in the system.

Note the CUDA version in the output. It is the highest version of the CUDA toolkit compatible with this driver, it does not mean that you have ANY version of the CUDA toolkit installed in your system. If you do not see this in the nvidia-smi output (I saw this several times) it means that the driver was installed without a library responsible for processing requests from CUDA libraries and your CUDA enabled containers will not work. In this case, you need to reinstall the driver (or even try a different version of it).

If you want to investigate the content of the package you can unpack it without installation by running:

$ ./NVIDIA-Linux-x86_64–450.80.02.run --extract-only

or

$ ./NVIDIA-Linux-x86_64–450.80.02.run --target [NewDir]

You can scrutinize other available installation options by running:

$ ./NVIDIA-Linux-x86_64–450.80.02.run --advanced-options

Desktop or Server running windows manager

For bare-metal or VM installation undertaken with GPGPU present during the installation, you need to disable default drivers first. To do this you need to create file /etc/modprobe.d/blacklist-nouveau.conf (you can use any name you like) and put the following lines into it:

blacklist nouveau
options nouveau modeset=0

After that you need to rebuild initramfs which is used to boot the system:

$ sudo update-initramfs -u

After reboot, you can do the steps described for the headless server above to install the driver.

Uninstalling Nvidia driver

If you already have an Nvidia driver installed on your system and want to install a different version of it (or simply get rid of it) you can uninstall the driver by running

$ ./NVIDIA-Linux-x86_64–450.80.02.run --uninstall

Additional Note

If you want to switch back to nouveau driver don’t forget to delete /etc/modprobe.d/blacklist-nouveau.conf (and rebuild initramfs if necessary) after deleting the Nvidia driver.

Sometimes, installing kernel updates will break compatibility with the installed driver. In this case, just uninstall and reinstall it again against the updated kernel (the process is the same).

Closing

I hope this sheds some light on the driver installation process and it will never trouble you again. Let me know if you would like to see a similar article on installing the CUDA toolkit or on how nvidia-docker enables containers with CUDA toolkit to run on the host with an incompatible driver.

Egor Bykov

RocketCompute

https://rocketcompute.com/

Proper monitoring is a foundation for mindful configuration management. It’s a prerequisite for any optimization effort. Monitoring is useless without the ability to act on the insight.

What most monitoring solutions lack is easy access to instant and relevant action upon insights you gather. This post is about a bot-based notification and management layer that we’ve created to play with the ways we can deal with that exact problem and that’s, for now, is available in a beta mode for our selected customers.

We in the RC team resolve to provide three simple things — an ability to come up with your own insights (via continuous monitoring), get our insight to support you in your decision making (through anomaly & bottleneck detection), and give you means to act instantly.

We wanted to provide our customers with means to communicate with their infrastructure. So, the obvious idea was, why not integrate this workflow with Telegram (or any other messenger) — the tool that is always with you and which goal is exactly that — instant communication?

Below are some other features of our bot MVP:

1. Integration with AWS
2. Monitoring charts in chats
3. Notification settings
4. Reasonable security measures
5. Start with the most useful cases (e.g. hibernate if something’s terribly off)

Integration with AWS

1.1 Metrics collection
1.2 Telegram authentication
1.3 Kapacitor’s events detection

Metrics Collection

RocketCompute uses Telegraf based metric collection agent it could be installed via command line or AWS launch template.

Telegram bot authentication

According to telegram API: you may use links like t.me/your_bot?start=XXXX that open your bot with a parameter (could be more than one). We use this feature to authenticate a user in the bot by passing the token.

Charts with anomaly events

Initially, we didn’t send any pictures via the bot. There were just links to the Grafana-based monitoring portal. But after some feedback (basically, it’s too much trouble to decide if you’d like to click without at least a preview) we decided to try adding charts right to the bot message.

What can I say, we dig the result.

Notification settings

First of all, every engineer prefers to have her own customizable instance list that she wants to watch over.

Fortunately, there is a checkbox feature in telegram. Exactly what is needed — users can get a list of active instances.

Security issues

Questions:

• What if someone gets a telegram token and access to the AWS management console through it?
• What if someone stole a Telegram account from an engineer and somehow got AWS credentials through it?
• What if our engineer makes critical changes from the Telegram bot (clients want hibernating instances straight from the bot)?

In the initial version we’ve done the following:

• Account admin approves every Telegram user authentication
• All users must have two-factor authentication for a telegram account
• Only hibernating command available right from the Telegram bot. And it works only for instances in idle state (when the corresponding notification appears)
• No SSH credentials in Telegram bot. Only link to AWS console with relevant parameters

Useful cases

Best of all (from what we can gather so far) is the Hibernate button. It’s not something that is used often but it saves money.

Well, that’s it.

Pretty simple setup, that being combined with a personalized notification and anomaly detection system. It saves money from the very start and gives you the comfort of never losing sight of your infrastructure, which is especially important at times when you run something time consuming and infrastructure-heavy.

P.S.: If you think that this might be interesting for your applications (or if you’d like to have integrations with another messenger) — let us know, we’d be happy to give you a demo and discuss how we might be of help. We are keen to discover and learn about new cases to apply our tech.

P.P.S.: You can try our bot along with our monitoring and detection system via RocketCompute item on the AWS marketplace.

Vladimir Kobzev,

Rocketocompute team

Intro

We at RocketCompute regularly optimize cloud HPC configurations for different workloads (e.g. this case for weather prediction). It is quite common for us to see the price or performance variation as wide as 4x-5x for different HPC cluster configurations. This variation usually heavily depends on network latency within the cluster and unique requirements of a workload (algorithm, software stack, RAM and storage requirements, etc.)

When we started looking at deep learning workloads we discovered that our colleagues most of the time use one of the following rules of thumb while picking cloud configurations for training DL models:

• The cheapest instance in case of the tight budget
• Instance with the most modern GPU available in case training time should be as low as possible

We decided not to take it for granted and do our common routine that we usually do when we are looking for optimal configuration for computationally intensive workloads. (Spoiler: it appears that DL training jobs show the same 4x variation in terms of price/performance on different cloud configurations and variation is task-specific)

If you prefer fewer words and more numbers you can check out a formal white paper that our team put together on the topic here.

Approach

The standard approach that the RocketCompute team employs for these tasks is to put together a benchmark that approximates the target workload and can be quickly run on different configurations. The resulting set of benchmarks duration/cost shows the most efficient and fastest set-ups while key hardware performance metrics (e.g. CPU/GPU/RAM utilization, etc.) can shed some light on why certain instances are better than the others.

For benchmarks, we started with reference implementations of the MLPerf training benchmarks set (you can find detailed info here). Out of 8 different neural network implementations available, we picked up 1 model for the most widely used domains. The resulting set is:

• For image processing — object detection Mask R-CNN model trained on COCO dataset
• For natural language processing — BERT model trained on Wikipedia dump
• For recommendations — DLRM model trained on 1Tb Kaggle AdDisplay Challenge dataset

We then fixed the number of training steps to reduce benchmark time to 15–45 minutes on the instance with one V100 GPU. For the BERT model, we also reduced the dimensionality of the neural net from BERT Large to BERT Base to fit the model into GPUs with less than 8Gb of available video memory.

The experiment was done using all AWS single GPU instances.

Key Results

Below table shows (in relative terms) how long each benchmark run and how much each run costs. All numbers for a particular network architecture are normalized to the fastest (or cheapest) instance for this benchmark. For example, for BERT the cheapest run was on g4dn.xlarge instance while it costs four times more money to run the same benchmark on g4dn.8xlarge. Similarly, the fastest run for BERT was on p3.2xlarge instance while it took p2.xlarge 4.6x more time to finish the same job.

Now let’s look carefully at these numbers:

1. The cheapest instance for BERT and Mask R-CNN appears the most expensive for DLRM (g4dn.xlarge with T4)
2. The most expensive instance for Mask R-CNN appears the cheapest for DLRM (p2.xlarge with a half (!) of a dual GPU K80, the oldest GPU available on AWS)
3. It appears that for DLRM you can choose a very well-balanced instance both in terms of price and training time (g4dn.4xlarge) and it will be very close to fastest (only 4% longer) and cheapest (only 2% more expensive). And the same instance would be around 2.5x slower and pricier for other architectures
4. Generally, the instance with the best training time never gave the best training price and vice versa

Looks odd, isn’t it? Let us drill down into hardware metrics to understand what makes the DLRM benchmark so special.

Let us start with memory usage. Below are the graphs showing how much RAM is utilized during the whole model training process for three instances with the same hardware but different RAM and vCPU counts.

Please ignore the yellow area for now, it shows how much RAM Linux kernel is using to cash IO operations. Focus on the green area, this is how much memory is used by the model. As you can see DLRM benchmark requires 42Gb of RAM minimum to put all required data into memory. According to the original DLRM paper (link) all this space is used to convert categorical features into vector representations to feed them into the neural net. In case when memory is not enough the kernel has to dump part of the data on disk (using memory swap). Given that disk is at least by an order of magnitude slower than RAM, no wonder low RAM instances perform so badly.

Now let us look at GPU utilization charts for DLRM benchmark across the whole process of model training.

You can clearly see that the more powerful GPU we use the less it is being utilized (see max gpu_utilisation at the bottom of each chart). This means that the system (or potentially the training algorithm) is not capable of providing GPU with enough work to keep it busy and it becomes even worse the beefier GPU we use.

Another interesting thing here is the chart for the M60. This is a perfect illustration of how lack of RAM looks like on GPU utilization charts. It can be illustrated even better if we compare GPU utilization charts for g4dn.2xlarge where SWAP space is actively used for DLRM benchmark with g4dn.4xlarge where all the data fits into memory.

Systems have exactly the same GPUs, same CPU family (though different number of vCPUs, but this is discussed later), most probably sliced out of the same physical server. Lack of 10Gb of RAM contributed additional 40 minutes to the benchmark and reduced max GPU utilization.

Finally, let us check CPU utilization charts.

You can see that only one core is used during the training (25% out of 4 vCPUs and 13% out of 8 vCPUs), i.e. all data pre-processing before sending it to GPU is done by a single core. Looks like we found another bottleneck.

Now we have a clear picture of what had happened with DLRM benchmark and why it performed so differently from the other two:

• Significant RAM requirements imposed a harsh handicap for instances with available RAM of less than required. If you look at instances with T4 GPU, the performance is higher for the higher amount of RAM (for the same GPU in the system)
• Single-threaded training script implementation caused an additional bottleneck in data throughput making the number of vCPU in the system irrelevant and giving performance advantage to instances with CPUs showing the best single-core performance (in our case it is g4dn instances with more modern Xeon CPU than others)

Conclusion

As you can see, even the simplest experiment can show that choosing an instance for training a Deep Learning model is not an obvious task even when you vary only a handful of instance features (like the number of vCPU, RAM, GPU architecture). The best instance will vary depending on the model architecture, code implementation, and amount of data you need to process.

We just barely scratched the surface of different aspects of the training infrastructure and how it can drive performance. We haven’t mentioned software stack at all (especially effects from different drivers and CUDA libraries) as well as distributed training on multi-GPU nodes and multi-node GPU clusters. Each of these topics deserves a separate post and, hopefully, will be covered in future parts of this series.

Egor Bykov

RocketCompute

https://rocketcompute.com/