Dude, Where’s My 4 Gigabytes of RAM?

Jeff:

" Exactly where is that magical PAE memory supposed to appear?

It’ll only “appear” in apps that explicitly code to support the AWE (address windowing extensions) API, which makes use of the PAE extensions in the CPU. I don’t think the base OS does this, but applications might.
"

No - PAE increases the available pool of memory in total. AWE isn’t directly connected with it, though the two go together from necessity for super-huge apps on 32-bit.

a href="http://blogs.technet.com/tristank/archive/2006/05/27/430507.aspx"http://blogs.technet.com/tristank/archive/2006/05/27/430507.aspx/a

Man, are you sure 4GB will be enough at all, for Windows Vista?

“Virtual memory, in a nutshell, refers to the method of an OS presenting non-contiguous memory as if it were contiguous.”

Of course in practice, on 32bit Windows, you still have the problem of actually getting a contiguous memory block greater than ~1.5Gb.

Windows maps drivers and suchlike at the bottom of the virtual memory for each process, then at around 2Gb, you’ll find it places system dll’s! So your poor process can never actually map a contiguous block of memory 1.5gb, even though it can access 1.5Gb + memory from the upper limit (~1Gb) using the /3Gb switch.

A real pain when you’re constrained to 32Bit Windows and attempting to create memory hungry processes used memory mapped files for shared mem.

http://msdn2.microsoft.com/en-gb/library/ms810613.aspx

Jamesm:
“Of course in practice, on 32bit Windows, you still have the problem of actually getting a contiguous memory block greater than ~1.5Gb.”

I didn’t say the OS had to be any good at it…

Nice article, again. You mentioned BIOSes that move the mapped PCI memory to a very high address. Actually the Asus BIOS I have works the other way round: the RAM between 2 GB and 4 GB is remapped to 4+ GB.

I’ve recently upgraded to 4 GB because I’m dumb and curious and have too much money. Now I found that I have to enable this setting when running Memtest86 1.70 so that it can see all 4 GB. It will test them in to chunks: 116KB (or so) to 2 GB, and 4 GB to 6 GB, because that’s where the BIOS has mapped the second 2 gigs.

But for booting Windows (XP 32-bit) I have to disable memory remapping in order to get at least a measly 3 GB (the 8800’s video RAM takes up all the rest!).

AWE = EMS for the early Noughties

no… This is a real system: [snipped]

cat /proc/meminfo

[ 16 GB RAM ]

cat /proc/cpuinfo

[ 4 Opteron 2.4 GHz dual-core CPUs ]

Kris, I think how did you correlate the limitation of 32bit addressing with DOS compatibility?

So if I really want everything I get out of my memory, don’t I want 3x 1GB sticks, AND 1x 512MB stick?

then you loose the dual channel benefit.

18,446,744,073,709,551K of RAM ought to be enough for anybody.

anyway since windows has a limited life…(vista is like 5 year support)

So… if we buy x64 hardware, will a 32 bit OS still see all 4 GB, or do you have to have a 64 bit OS as well?

I won’t be going to a new OS anytime soon ( I want Vista, and don’t have the money for it yet). But my church is looking at upgrading to 4GB in the recording computer…

haha great jeff! very interesting this article!
its silly but i never thought about a limit like that in the modern computer architecture.
i just hope to live until the day that i can say: dude!! where are my 19 EB (aka Exabyte)??? hahah

ps. hey jeff i would to suggest you the topic of one of your next articles, cause i was thinking if exist a diet, or better like position to improve productivity for programmers cause i am getting pretty fat haha :D. sorry i go always off topic. Regards.

Actually, the physical processor does have, to this day, DOS compatibility in microcode and bios. (Maybe in stub form, running DOS probably won’t actually work anymore with new hardware anyway, I don’t know if anyone’s even tried it in years.) When they say x86 ISA is ‘legacy baggage’ they don’t just mean the silly register layout and instruction set.

The northbridge or HT bridge controls all access to memory, both DRAM and PCI chips (and bios and other device junk), and it’s the one that initializes PCI devices into certain slots. That’s why the cpu, and thus the OS, is incapable of seeing through it.

That, my friends, is why Apple does not offer 4GB of RAM in laptops. Which makes me wonder how all the other manufacturers explain to the rich idiots who splurged their kilobucks on their loaded Lenovos, Toshibas and HPs, why they can’t see all of their expensive RAM.

“The user-mode virtual address space in 64-bit Windows is a mere 8 terabytes… It’s the final solution, at least for the lifetime of everyone reading this blog post today.”

You think? In the early 90’s, I had a computer with 2MB of RAM. Today (17 years later), it is feasible to have 4GB. That’s 2,000 times more memory. If the same trend continues, we could have 8TB by 2024. Certainly within most of our lifetimes.

If and when Adobe’s new InDesign Suite goes to 64bit, then I will certainly buy a PC to take full advantage of those available features. Until then I will just continue to run at 2GB RAM.

Kris, you are embarrassing yourself with such comments, so the 1MB “DOS compatibility” is what makes the 4G limit ? LOL.

Ya know, I thought it was common to be using the PAE kernel these days, at least on recent NX-capable hardware.

You haven’t disabled DEP, perchance?

http://msdn2.microsoft.com/en-us/library/ms791508.aspx

I’m running X64 at home and at work, so can’t readily check
Mental note to look that up sometime.

Jeff, I think you should edit this article, because your recommendation:

“The proper solution to this whole conundrum is to use a 64-bit operating system”

is misleading, and may well cause people serious grief. There are loads of systems out there with 64-bit CPUs that only have a 32-bit physical address range. For example, the vast majority of laptops only have a 32-bit physical address range, even though many are now available with 64-bit CPUs. And plenty of desktop systems have the same issue.

People following your advice are liable to put themselves to the inconvenience of switching to a 64-bit OS only to find that it doesn’t actually solve the problem - they’re still limited to 3.5GB of RAM, but now they have the additional problems that half their devices are no longer properly supported, and their OS now has a bigger memory footprint thus reducing the amount of memory available for doing useful work… Most people will not regard this as an improvement, and they could reasonably be annoyed with your advice.

The bottom line is this: a lot of chipsets only use 32 address bits - the higher order address bits on the processor are not wired up to anything more than terminating resistors. Turning on /PAE or switching to 64-bit won’t help you if you have such a chipset because the CPU doesn’t have a way of asking for stuff above the 4GB mark. 64-bit will let you get more than 2GB of virtual address space in a process, but you’ll still be stuck with 3.5GB or so of physical memory.

So if I were you I’d qualify your recommendation before you get a horde of angry people beating a path to your door… Make it clear that you need to have a suitable chipset. Moreover, if you do have a suitable chipset you probably don’t actually need to go 64-bit. /PAE may actually be a better option in reality, given today’s less than total support for 64-bit. PAE may be a hack, but who cares? The OS deals with it; it’s no uglier than the fact that my machine has 4GB of virtual memory (my physical memory + page file is 4GB, on my 32-bit system). 64-bit isn’t about breaking through the 4GB barrier of physical addressing - it’s only really useful if you’ve hit virtual address space limits, which is really a completely different class of problem than the one you open this blog entry with.

(Although if you need more than 64GB of memory, I believe 64-bit is necessary. PAE only goes up to 64GB of physical address space.)

In short: the 3.5GB problem is usually a chipset thing, not a 32 vs 64 thing. Unless you have individual processes requiring more than 3GB of memory, /PAE is often a better solution. (With most people I know who need this much memory, it’s because they are hosting a couple of VMs, and /PAE is just fine for that.) And if /PAE doesn’t work on your system, switching to 64-bit probably won’t either.

Thanks for the tip…I am in the process of planning my new rig and was about to make that mistake. The problem is that I can’t afford to lose all of the programs that don’t run well (or at all) on the 64 bit OS. And I hear that drivers for Vista 64 are a nightmare so I guess 3 gigs of ram it is.

John
http://www.monomachines.com

I’m in the same boat as Graham, well not exactly, I can’t even find all the proper drivers for 32 bit Vista. So it’s a waiting game to first get all the necessary drivers for my hardware then for the applications I require to be compatible.

Nobody will ever need more than 4 X 10^24 exabytes of memory…

Gee, I wonder what the equivalent neural capacity of the brain is…

Oops, were comparing apples to kiwis, or should I say analog to digital.

If there is one thing that I’ve learned, if you offer more hardware capacity it will be used.
“Is your name Sahra O’Conner?”

Here’s a nickel, kid. Go buy yourself a real computer.

nitrogen:~# cat /proc/meminfo
MemTotal: 4051648 kB
MemFree: 66600 kB
Buffers: 360016 kB
Cached: 2215200 kB
SwapCached: 12 kB
Active: 818808 kB
Inactive: 2175504 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 4051648 kB
LowFree: 66600 kB
SwapTotal: 9823672 kB
SwapFree: 9823244 kB
Dirty: 272 kB
Writeback: 0 kB
Mapped: 449972 kB
Slab: 970272 kB
CommitLimit: 11849496 kB
Committed_AS: 799204 kB
PageTables: 4424 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 5024 kB
VmallocChunk: 34359733327 kB

I ran into the exact same problem on my Vista 32-bit box… the memory showed up properly in the BIOS, but not in System Information. It turns out that Microsoft recently released a KB article to describe the problem (although they didn’t go into the level of depth that you did… great job!)

http://support.microsoft.com/kb/929605/en-us

The system memory that is reported in the System Information dialog box in Windows Vista is less than you expect if 4 GB of RAM is installed

Ian, what 64 bit chipset uses 32 bit addressing ? That is by definition not a 64 bit chipset then, sounds very strange to me but it wouldn’t suprise me if you are correct that some do (probably Intel).

Go with AMD64 which was the orginal inventor of the EM64 extensions that Intel then “stole” (yes I know they can with the patent agreement from -94) and you’ll most likely be safe from such issues.

As for the prople coming here posting some printouts from various *nix systems, you really need to read the article again, this is NOT a Windows issue, my good, you are embarrasing.

I’m currently running an Intel Core Duo that is about a year old and it simply isn’t x64 capable. This is in response to the poster who “wished Microsoft had only released an x64 version of Vista”. They would have left a lot of us out in the cold.

With that said, you can see why many hardware manufacturers haven’t been too quick to come on board. x64 Win XP has only been used by a select few to date because you would have had to buy fairly high end processors to even run it. Only now are mass consumers finally being offered processors that can support the x64 OS’s (at least for Intel).

bah, Eugene, here’s a real system:

root@/proc# cat cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 6
model name : Intel Celeron (“GenuineIntel” 686-class, 128KB L2 cache)
stepping : 5
cpu MHz : 501
fdiv_bug : no
fpu : yes
fpu_exception: : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr

root@/proc# cat meminfo
total: used: free: shared: buffers: cached:
Mem: 257118208 62099456 195018752 0 0 0
Swap: 511995904 0 511995904
MemTotal: 251092 kB
MemFree: 190448 kB
MemShared: 0 kB
Buffers: 0 kB
Cached: 0 kB
SwapTotal: 499996 kB
SwapFree: 499996 kB
cat: meminfo: Invalid argument

“Could be raining!”

I just watched that movie last night, and I read this blog post directly afterwords. What are the odds?

You know what’s funny? I am running 4GB of physical RAM on 64bit hardware, but I don’t have a 64-bit version of my O/S. I really should upgrade, but since I get to see 3.4 GB of it anyway, there’s not much incentive…

Matt, I disagree that only now are consumers getting x64 CPUs. Athlon64s have been available since late 2003. I built my current PC in September 2005 and I specced it with an Athlon 64 X2 3800+ which wasn’t unusually expensive or “high-end” even back then.

Look at a major manufacturer like Dell and even some of the cheapest amd most basic systems they offer use 64-bit capable CPUs.

Okay, perhaps it is too much to expect Microsoft to exclude 32-bit completely, but why is it the default? It currently isn’t possible to buy a retail version of Vista 64. Instead you buy the 32-bit version and then follow the instructions inside to send off for the 64-bit one!

Given my experiences with XP x64, I don’t find that approach very comforting.

On the whole 64-bit / x86 that was mentioned earlier in the comments…

I avoided 64-bit carefully for a number of reasons. First, X32 is the supported standard, and I knew that Intel was coming out with multi-core processors. I figured, and I think I may have figured correctly, that people were going to be more willing to figure out multi-threading than they were going to be willing to figure out X64 architecture. I sort of saw this coming when Apple began releasing dual-processor G5s several years ago.

Much as I think the whole concept of “more processors equals better computer” argument is rather silly, it is the way the industry tends to trend. I mean, it took years for all the die-hard Apple fans to swap from Motorola to PowerPC chipsets, and they only did that because they more or less had to for speed purposes. Apple continued to support FAT applications right up the the release of OS X.

To this point, most PCs since the industry really got off the ground with the 8086 have all been based on that exact architecture. S’why we call 'em X86. Of course, I never knew if “286” was short-hand for 8286, or if it literally went 8086, 268, 386, 486, Pentium, Pentium II, Pentium III, Pentium 4, Intel Core Duo, etc. I’m sure I’ve missed one or two iterations in there.

I’m hoping they’re going to have the quad-cores available for purchase this summer when I’ll be trolling for yet another new computer. If they don’t, I’ll probably just max out a dual core system. It’s something, anyway.

@Josh,

Last I checked, the machine addresses memory in bytes, not bits…

I did have a point in there: adoption will be slow, but it WILL happen, eventually.

My curiosity is that since I never really studied 64-bit, what is its relationship to X86?

Ian, using /PAE to address memory 4 GB on 32-bit systems has its own problematic issues, according to Microsoft. There’s a performance penalty because I/O requests from 32-bit devices must be double-buffered, and some devices might not work at all, depending on the driver. I wouldn’t want to recommend THAT to a general audience, either!

"Ian, what 64 bit chipset uses 32 bit addressing ? That is by definition not a 64 bit chipset then, sounds very strange to me but it wouldn’t suprise me if you are correct that some do (probably Intel)."
By the logic of the chipset/motherboard manufacturers supporting stuff that customers are unlikely to use is pointless, and thus it’s unusual for motherboards to support the full address range of the CPUs they have. After all how many customers are going to have more than 2gb of memory?

@Brandon: I didn’t say Apple is(n’t) 64bit. I was replying to your statement saying Apple’s “behind the times technology-wise” - I even said that was what I was replying to.
If any mainstream OS is behind the times technology-wise, it’s obviously Microsoft’s.

Okay, I just checked… my XP SP2 Pro system has PAE enabled (comes automatically with DEP). So for shits and giggles I enabled memory remapping in the BIOS and rebooted Windows. I know from MemTest that 2 out of my 4 gigs will now appear in the 4-6 GB range.

Yet Windows tells me that it has only 2 GB total! Total available memory, according to Task Manager: 6 GB = 2 GB RAM + 4 GB page file.

Exactly where is that magical PAE memory supposed to appear?

Mac Pro handles up to 16GB of RAM and 3TB of disk space.

Two 3GHz 64-bit Dual-Core Intel Xeon “Woodcrest” processors.

http://www.apple.com/macpro/specs.html

And with Parallels, you can run Windows, Linux, and MacOS x at the same time.

http://developer.apple.com/documentation/Performance/Conceptual/ManagingMemory/Articles/AboutMemory.html

Charles: Mac OS has always been a little ahead of the curve in what sizes it will handle, both in RAM and Hard Drive capacity.

I seem to recall Macs handling the GB threshold more easily than did PCs.

To be perfectly clear, this isn’t a Windows problem-- it’s an x86 hardware problem

Indeed. The latest Intel Core Duo-based iMacs have the same issue. You can put 4GB in them, but they’ll never be able to use more than 3 GB.