TIP_Use_memory_on_video_card_as_swap

Introduction

Everybody considers today's graphic cards to be meant only for gamers. After all, why would one need a fast GPU in a server? Today's graphic cards contain a lot of very fast RAM, typically between 64 and 512 MB. With Linux, it's possible to use it as swap space, or even as RAM disk!

Preparing the kernel

To use the graphic (and possibly any other) card's memory, it needs to be mapped by the kernel. By default only the system RAM is considered as usable RAM, to access memory from the PCI address space one needs a driver which registers them. In 2.6 kernels this driver is shipped: Memory Technology Device (MTD) support. Bring up menuconfig, then:

Device Drivers  --->
  Memory Technology Devices (MTD)  --->
    Memory Technology Device (MTD) support
     [* or M]Direct char device access to MTD devices
     [* or M]Caching block device access to MTD devices
     Self-contained MTD device drivers  --->
      [* or M]Uncached system RAM
      [* or M]Physical system RAM

You can compile them as modules or built-in drivers, it works either way.

Finding the video memory

Now let's search for the video memory in the PCI space. The easiest way is to use lspci:

# lspci -vvv

0000:02:00.0 VGA compatible controller: ATI Technologies Inc R300 AD [Radeon 9500 Pro] (prog-if 00 [VGA])
        Subsystem: PC Partner Limited: Unknown device 7c07
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (2000ns min), cache line size 08
        Interrupt: pin A routed to IRQ 255
        Region 0: Memory at d8000000 (32-bit, prefetchable) [size=128M]
        Region 1: I/O ports at b000 [size=256]
        Region 2: Memory at e9000000 (32-bit, non-prefetchable) [size=64K]
        Expansion ROM at e8000000 [disabled] [size=128K]
        Capabilities: [58] AGP version 3.0
                Status: RQ=256 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64- HTrans- 64bit- FW+ AGP3+ Rate=x4,x8
                Command: RQ=1 ArqSz=0 Cal=0 SBA+ AGP- GART64- 64bit- FW- Rate=<none>
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

What we need is the prefetchable memory region, Region 0. The start of the memory space is 0xd8000000, a 32 bit hex value.

Mapping the memory

To actually access the memory from PCI space, you need to tell the MTD driver where it starts and where it ends. Don't worry, it's easy to calculate the ending address! Just decide how much memory you want to use. However, it's wise to leave some for the video card too, to let its basic functionalities work. Now, if you compiled the SLRAM driver as module, run:

# Region 0 starts at 0xd8000000, leave 4 MB ram for the VGA, so add 4*1024*1024 to it
# which is 0x00400000 in hex, 32 bit. This means: 0xd8400000 will be the starting position.
# Now, to use the other 124 MB, the easiest way is to tell the offset value.
# The third parameter's leading + means that we are telling the offset,
# not the actual region's ending address. 124 MB is 0x7c00000 in hex, 32 bit.

modprobe slram map=VRAM,0xd8400000,+0x7c00000

# Loading with PHRAM module is very similar to SLRAM module, except that it uses a "length"
# parameter by default rather than an "end offset".

# This does the same thing as the SLRAM module above
modprobe phram phram=VRAM,0xd8400000,0x7c00000

# This is also the same (and easier to read)
modprobe phram phram=VRAM,0xd8400000,124Mi

or if you compiled it as built-in, change your kernel command line to contain this:

# Now the first parameter is the device which will be mapped. 
# Other parameters share the same meaning shown above, in the module version.
# Please reboot your computer, to take it effect.

slram=mtd0,0xd8400000,+0x7c00000

What you should see in dmesg is:

slram: devname = mtd0
slram: devstart = 0xd8400000
slram: devlength = +0x7c00000
slram: devname=mtd0, devstart=0xd8400000, devlength=0x7c00000
slram: Registered device mtd0 from 3543040KiB to 3670016KiB
slram: Mapped from 0xe0880000 to 0xe8480000

dev:    size   erasesize  name
mtd0: 07c00000 00004000 "mtd0"

Creating the mtd devices

After you have the support in the kernel and all the required data, you need way to access the memory. Time for creating the devices!

If you have udev just modprobe mtdchar and modprobe mtdblock and hald will create device files for you automatically.

# Create the character device. MTD will use and map this, so it's not used by
# userspace programs but is vital for MTD to function.
# If "Direct char device access to MTD devices" was compiled as module, run modprobe first: modprobe mtdchar

mknod /dev/mtd0 c 90 0


# Create the block device. This will be used. Requires the "Caching block device access to
# MTD devices" kernel feautre, if compiled as module, first run: modprobe mtdblock

mknod /dev/mtdblock0 b 31 0

Using it

Now, you can either use it as a RAM Disk, or as swap. Either way, you need to create a FS on it, I'll go with the swap way.

mkswap /dev/mtdblock0
swapon /dev/mtdblock0

This will create the swap structure in the video card's RAM, and also activate it as system swap. To specify the priority (which swap source will be used first in the swap list), use swapon's switch, -p X where X is a number between 0..32767, the higher the number, the higher the priority.

Without specifying priority, the next lowest will be used. Also, this can be added to the fstab, so you won't need to manually add this after every boot:

# <fs>                          <mountpoint>    <type>          <opts>                  <dump/pass>
/dev/mtdblock0                  none            swap            sw,pri=10

In the options field, the pri= is the same as swapon's -p switch. Please read the troubleshooting section, before closing this page.

Be warned

It's nice to have fast swap or RAM disk on your home computer but be warned, if a binary driver is loaded for X, it may freeze the whole system or create graphical glitches. Usually there is no way to tell the driver how much memory could be used, so it won't know the upper limit. However, the VESA driver can be used because it provides the possibility to set the video RAM size.

So, Direct Rendering or fast swap. Your choice.

Unlike motherboard RAM and hard drives, there aren't any known video cards that have ECC memory. This may not be a big deal for graphics rendering, but you definitely don't want to put critical data in it or use this feature on servers.

Troubleshooting

• • Q: mkswap/mkfs.whatever succeeds, but I can't swapon/mount it! Also it reads 0xff bytes! What to do now?
  • A: You have to set the BIOS setting, AGP Aperture Size to at least or equal to your video RAM. Most people say it's better to set it to the half of your system RAM/video RAM size, but it does nothing just limits the BIOS' capability to map from PCI address space.
  • A: It is also possible that you are using wrong address/offset values. Double-check that you are using the 'prefetchable' space.

• • Q: I have shut down my computer and during the next boot I see it couldn't add /dev/mtdblock0 as swap! What did go wrong?
  • A: It's trivial but at first time everyone is surprised: RAM will lose its contents if not powered. Solution: mkswap again!

• • Q: The kernel tells me: slram: ioremap failed. What is this?
  • A: You forgot to add + before the actual value in the third parameter, so it's not considered as offset code but an actual PCI address. Prefix with +, or calculate the exact ending location.

• • Q: There is no /dev/mtd0 and/or /dev/mtdblock0 device.
  • A: Read the 'mknod magic' box.

• • Q: I have enabled Highmem support in the kernel and now slram complains something about vmalloc in dmesg. I can't access the device.
  • A: Try appending vmalloc=nM to the kernel parameters, with n something like 256 or higher if needed. The default is 128

Credits goes to

This article was plagi^W inspired by the article written by Michal Schulz, available here: [1] and was summarized by me.