Ryzen AI Halo: quatre ajustos per accelerar la IA local fins a un 15%

So last week AMD sent me the new Ryzen AI Hello, which essentially serves as a reference platform for locally AI on the Ryzen AI Max processors, namely Strix-Alo and eventually the upcoming Corgon Halo. Other creators like Alex Ziskind and Wander from Level 1 text have already covered these devices. So here I want to focus mostly on my setup and now I configure the school tiny box to squeeze as much performance out of it as possible. As you will see later in the benchmarks, using some specific optimizations yields over a 10 to 15% improvement in processing speeds for some models. Before we jump in a quick thank you to AMD for sponsoring this video. If you're building on AMD hardware like the Ryzen AI Halo, check out the AMD AI developer program. It gives you free cloud credits, expert access and premium training to optimize and care your AI workloads, the thing to join is in the description. So let's talk about the Out of the Box experience that AMD has designed for these box. They could have just done a low-eye-forward Ubuntu install. Instead, they went all the way to make their own debut-based distribution, and this comes with a lot of useful stuff out of the box, including quick start guides and pre-installed software like lemonade, Lama CPP, VLLM, and comfy UI.

And frankly, this is just what you want when you buy such a device, a easy, polished, and just works experience. AMD also set up a resource with AI playbooks to help people get started and experiment. Check those out and I'm planning myself to contribute to these playbooks. Let me be I'll go over some of them in an upcoming video. So let me know in the comments if you are interested. Before getting into the details, let's start with the base system configuration. At the heart of this machine is the Ryzen AI Max Plus 395 APU. This has 16 Zen 5 cores and an integrated GPU with 40 RdNA 3.5 compute units, which is the architecture commonly referred to as streaks halo. It also comes with two terabytes of SSD storage and most importantly up to 120 gigabytes of LP DDR5x unified memory running at 8000 megatransfers. This gives you a memory bandwidth of 256 gigabytes per second. Now all of these fits into a very compact chassis running at 120 watt for networking it includes a 10 gigabyte internet WiFi 7 and Bluetooth 5.4.

It is also worth mentioning that the platform supports both Windows and Linux. I prefer Linux and that is what I cover in this video, but if you need Windows, these works just fine on these box. As I said, the out of the box UI experience is great for getting started, but once you're familiar with the platform you might want to experiment with a different setup that allows you to squeeze a bit more out of these box. Number one, if you already have a primary machine, I recommend using the Halo box strictly as a server. That means switching the system to multi-user mode so the desktop environment does not load on boot. The graphical interface consumes memory. By running headless, we free that memory up for model weights and context windows. Next, let's look at the unified memory parameters. By default, this comes with 96 gigabytes allocated to the iGPU. But we can push that much a lot of time. Especially if you're running these as a dedicated AI server. On strict cellul to the boxes I show how to do these by passing some kernel parameters, but on these box you can use the AMD debug tool, specifically the AMD TTM Commonline tool to view and set the memory allocated to the integrated GPU. In my setup I set the allocation to 1204 gigabytes and these leaves four gigabytes for the base operating system, which is more than enough,

while maximizing the memory than I make available to the GPU to improve performance. You also want to install the TUND, D-MON and set the system to the Accelerator Performance profile. These background service adjusts the kernel settings and the CPU power states to prioritize compute speed over power saving. Just keep in mind that this profile limits power savings to significantly improve GPU performance. The final configuration step is disabling the IOR MMU or input output memory management unit and this essentially forces the system to fall back to standard direct memory access or DMA. Now in a nutshell, this removes a translation, middleman, on every memory access, which in turn reduces latency and improves memory speed for our AI workloads. Now the Linux distribution on the Ryzen AI Halo uses SystemD boot. So to disable the IOM you need to edit the file, it is the Kernel CMD line and append AMD underscore iOMMU equals to off to the end of that common line. Then regenerate your boot and risk by running Kernel install and finally reboot the system to apply the changes. So you can find the benchmarks on strict cello to the boxes and specifically on the Lama CPP. Bunchmarks, I've made a specific page for the Ryzen AI Halo. Now you see here I have picked mostly the Q1 models. These are some of the best models you can run locally and I have

benchmarked them on the default profile and on the performance profile that I have just discussed. And you can see here, let's say, let's start with the 35 billion parameter model. This is the onslaught dynamic quant Q4K XL. And just to remind you, this model is 3 billion active parameters. We can see the pre-field, so the prom processing at different context sizes. So I go from 0 to 8000 tokens to 60 for 1000 tokens and when we get started with the default profile, we have, as you can see here, a 1,380 tokens. The second, but on the improved performance profile, we go up to 1,500 and 60, 3 tokens per second. So that's a considerable improvement. And you can see that that improvement stays there. Maybe the gap gets a little bit smaller, but it stays there as the context grows. So we see a lot of improvement in problem processing. And actually here, if you scroll down, you are going to see the actual change. And again, in some cases, we have 18% more in terms of from processing speed, and even at a high or long context,

like 64,000 tokens, we still have a 14% improvement. Now in token generation, you can see that there is equally an improvement. It is not that big, but it is still between 3% and 7 or 8% so this is still quite faster. So if we take a look at another model For example, the 122 billion quaint 3.5 model, this has 10 billion active parameters, and it is still in the unslawed dynamic quaint Q4K Excel. Now if you take a look at the performance, again, you can see that especially in prompt processing, we see much better performance in the default configuration, we get 487 tokens per second, And in the improved configuration,

we get 545 tokens per second. And these kind of stays the same, even when the context grows. And if we scroll down and take a look at the numbers, you can see that for prefil, in some cases on this model, we actually get close to 20%. I was tempted to do a clickbait thumbnail where I said improved the rise and the AI halo performance by 20%. But obviously it is not 20% in all cases. And you have to keep in mind that there is some variability in these numbers. So there is a little bit of margin of error, but you can see that the performance improvement is consistent. And what's interesting is that in token generation, that gap actually becomes larger with this model

new start seeing instead of 15 tokens per second you get at a very long context 16.86 so almost two tokens per second more which again is not dramatic but it is it is quite a lot but hopefully this gives you an idea of what you gain by implementing my recommended configuration for these box. So after configuring these as a server typically I get either Lama CPP via the LIM or now the Warstar running these are all engines that allow you to run LIM inference and expose and end point and then I connect from my laptop. I've showed these in other video but very quickly the easiest way for me is to install Lama cockpit with this command over here. So I get into my strict cello and I install Lama cockpit with PPEX. You might need to install PPEX and then I can just do Lama cockpit. And there it is. I have different versions of Lama CPP. These are all containers and they have different versions of Rockham. come typically right now you want to use the latest version which when I'm making this video is 7.2.4 and these version of LAMASPP gets essentially updated twice a day so you get all of the latest features very quickly delivered here and I also have some experimental toolboxes so sometimes

there are some features that LAMASPP is working on and they're not ready yet for production but you might want to experiment with that. For example, Toolbox Wound or Rockham FP4. And this is easy to get up and running. But essentially, I typically select one toolbox and I create or update that particular toolbox. In this case, I already have this here. And then the easiest way is to go to model manager, find a model that you want to use, and you've seen some of the models before. So for example, this one over here, and I've already downloaded the quantizations that I want to use, but otherwise, you can click on one of these and it's going to download it, by default, in your home directory, on the server, in the folder models, you can have it anywhere you want. And then if I want to run this model, I can select that toolbox and I can select the model.

So in my case, I want to run this one over here. that I use actually quite often. And on this model I tend to enable mtp, multitoken predictions, peculiarity of the coding, especially for the use cases that I typically use these models for, which is coding and quite predictable stuff. It really gives up performance improvement, and these enables some tp by default. I have a specific video or need, if you want to know more about that. But then you can select an inference profile and typically I've got this one over here, thinking, coding, which sets some parameters that you can actually see here. I think these parameters come directly from the people that make the coin models and then all the other stuff is selected and you can just start the server and obviously to be able to access it, you want to forward port 8080 to your machine. So once that's up and running on port 8080 you are actually able to access these LMSVP server on your machine and there it is the LMSVP UI so you can obviously interact with the model and as we saw this is very very fast it does thinking and then it gives you the answer, but most importantly you can connect something like Pi code engagement to this and I'm going to show you how I typically do it.

So what you want to do after installing the Pi code engagement is to modify the model's JSON file to essentially make the Lama CPPN point available and this is how I do mine. So I keep it fairly simple I just specify a context window and a model ID. It doesn't really matter because this is the endpoint and whatever model I'm loading, this is going to use it. And once I've got these, I can, if I want, I can start pi from the CLI. But typically I use pi in this was to your code and there are a couple of extensions that I tend to use. There is pendant, which I was using at the beginning. This is very good, but it looked like it was going to be open source and it is not open source anymore. So I'm now trying another one which is fully open source, which is called towering. And actually I really like this because it exposes a lot of things to you in a more obvious way. And you can see here I can select the Lama CPP local model that we configured and it says thinking of simply because I haven't configured it in models. So Jason, but this is going to do thinking and for example, I can ask what is this project about and as you can see, this is now going to go around read different files and it's going to ask me for permissions. I typically use

this permission extension for Pi, but you see kind of read the agent's file and it's giving me the overview of this project. And I find again this model works really well. Sometimes I will use the Q1122 billion parameter model on strict sale. It is slow, but it is much better than this one. But as of later, I'm using more and more deep seek before flush in the Dwarster engine and that's something again you want to probably watch one of my previous videos but very quickly you can go on the Dwarster to the box that I've created and I have a again the video that describes all of it in detail but also this very simple PPEX command so you can install the S4 and then run it I think I've already installed it here so it's going run it and this is the main toolbox that you want to select and create or update and this is going to download the container exactly as it was happening before and you can refresh these or sorry you can check for updates to see if there is any update to that toolbox in this case the reasoned and then you can go to model manager and typically all these machine I recommend this molest quantity, IQ2X success, I matrix quant, it's 80 gigabytes, I already downloaded it and then you can go to server mode obviously that toolbox selected and this is the only model that we have and we can just start this.

This by default starts from port a thousand so we can go and start that, it loads all all the ways into memory. Now, this is obviously slower than some of the other models, but the quality is really high at coding tasks, even at this prompt. And if you go in the agent file, I have the configuration here. And I typically forward that on port 80, 82, you can obviously use whatever you want. And I can go back in here. And maybe we can start a new session, just to show you that these actually works. And a little bit slower, but it will also be able to analyze these and work on this code base.

Now, as this is working, something else that I typically install is AMD GPU top. This allows you to see the details of how the GPU is being utilized, much memory is used and the activity so I don't know I really like looking at the GPU crunching numbers and there it is it is finally coming back with an answer so this is obviously a slower model to RAM than the Quaint 3.635 billion parameter models that only has 3 billion active parameters but also you can see that deep seek before flash did a much more thorough job by taking the Quen model just essentially read Agent's Totem D and stopped whereas this really tried to read as much as possible of this project to get an idea and the answer is considerably more thorough than what we got before but it did consume a lot more tokens and as much slower. If you watch until the end, hopefully you found this content useful, you are a new tube.

If you want to support this channel, remember to like, comment and subscribe or not. You don't have to appease the algorithm if you don't want to. Another way to support my work is to donate using my Bini a coffee link that you find in the description. So to wrap this up, what I wanted to show in this video is fairly simple. The out of the box experience on the Ryzen AI Halo is excellent, and you can push it's performance even further by configuring it as a headless server and enabling a performance profile. Just let me know in the comments if you have any questions or experiments you want me to run on this box. Thank you for watching and I'll see you next time.