Skip to content

DGX Spark: UMA support - #17368

Merged
ggerganov merged 6 commits into
ggml-org:masterfrom
sfudally-nvidia:dgx_spark_uma
Nov 20, 2025
Lines changed: 100 additions & 0 deletions
Merged

DGX Spark: UMA support#17368
ggerganov merged 6 commits into
ggml-org:masterfrom
sfudally-nvidia:dgx_spark_uma

Conversation

@sfudally-nvidia

@sfudally-nvidia sfudally-nvidia commented Nov 18, 2025

Copy link
Copy Markdown
Contributor

Add support for DGX Spark/UMA devices on Linux per this note on detecting available memory:
Memory reporting differences with unified memory architecture

@sfudally-nvidia
DGX Spark: UMA support

Verified

This commit was signed with the committer’s verified signature.
sfudally-nvidia Scott Fudally
GPG key ID: C40C86C0E31A914E
Verified
Learn about vigilant mode
Loading
Loading status checks…
01c791e
ggerganov
ggerganov reviewed Nov 19, 2025
View reviewed changes

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we expect cudaMemGetInfo to be fixed in the future and not need this workaround?

ggml/src/ggml-cuda/ggml-cuda.cu
ORippler
ORippler reviewed Nov 19, 2025
View reviewed changes
ggml/src/ggml-cuda/ggml-cuda.cu Outdated
slaren
slaren reviewed Nov 19, 2025
View reviewed changes
ggml/src/ggml-cuda/ggml-cuda.cu Outdated
ggml/src/ggml-cuda/ggml-cuda.cu Outdated
ggml/src/ggml-cuda/ggml-cuda.cu Outdated
@sfudally-nvidia
Updates from PR feedback

Verified

This commit was signed with the committer’s verified signature.
sfudally-nvidia Scott Fudally
GPG key ID: C40C86C0E31A914E
Verified
Learn about vigilant mode
Loading
Loading status checks…
885d972
@sfudally-nvidia
More PR feedback cleanup

Verified

This commit was signed with the committer’s verified signature.
sfudally-nvidia Scott Fudally
GPG key ID: C40C86C0E31A914E
Verified
Learn about vigilant mode
Loading
Loading status checks…
4134bc8
@github-actions github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Nov 19, 2025
@sfudally-nvidia @ggerganov
Update ggml/src/ggml-cuda/ggml-cuda.cu

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
Loading
Loading status checks…
6fcd8ec 
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@sfudally-nvidia

sfudally-nvidia commented Nov 19, 2025

Copy link
Copy Markdown
Contributor Author

Can we expect cudaMemGetInfo to be fixed in the future and not need this workaround?

Yes, have discussed with CUDA team but do not have an ETA on when that would be released

@sfudally-nvidia
Remove trailing whitespace

Verified

This commit was signed with the committer’s verified signature.
sfudally-nvidia Scott Fudally
GPG key ID: C40C86C0E31A914E
Verified
Learn about vigilant mode
Loading
Loading status checks…
580d030
ggerganov
ggerganov approved these changes Nov 20, 2025
View reviewed changes
ggml/src/ggml-cuda/ggml-cuda.cu
@ggerganov
ggerganov requested a review from slaren November 20, 2025 06:18
@ggerganov
Update ggml/src/ggml-cuda/ggml-cuda.cu

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
Loading
Loading status checks…
e9c289f
@ggerganov
ggerganov merged commit a7784a8 into ggml-org:master Nov 20, 2025
1 check passed
@matt23654 matt23654 mentioned this pull request Nov 27, 2025
Closed
@AmesianX AmesianX mentioned this pull request Dec 18, 2025
Open
Anico2 added a commit to Anico2/llama.cpp that referenced this pull request Jan 15, 2026
@Anico2 @ggerganov
DGX Spark: UMA support (ggml-org#17368)
2dc8aab 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
blime4 referenced this pull request in blime4/llama.cpp Feb 5, 2026
@sfudally-nvidia @ggerganov shaobo.xie
DGX Spark: UMA support (#17368)
fb01587 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
hogeheer499-commits added a commit to hogeheer499-commits/llama.cpp that referenced this pull request Mar 12, 2026
@hogeheer499-commits
ggml-cuda: skip UMA memory detection for HIP/ROCm builds
8674daa 
AMD APUs (e.g., Strix Halo with gfx1151) report prop.integrated=1,
which triggers the UMA memory detection path added in ggml-org#17368.
This causes the available GPU memory to be read from /proc/meminfo
(MemAvailable) instead of hipMemGetInfo().

On AMD APUs with large TTM allocations, hipMemGetInfo() correctly
reports the full GPU-accessible memory (e.g., 122880 MiB on a
128GB system), while /proc/meminfo reports significantly less
(~90 GiB), as it only reflects available system RAM.

The UMA detection path was designed for NVIDIA DGX Spark systems
and is not appropriate for AMD APUs, where hipMemGetInfo() already
returns accurate values.

Fix: guard the UMA detection block with !defined(GGML_USE_HIP).

Tested on: AMD Ryzen AI MAX+ 395, Radeon 8060S (gfx1151), 128GB
unified memory, ROCm 7.1, Ubuntu 24.04, kernel 6.18.14.

Fixes: ggml-org#18159
@hogeheer499-commits hogeheer499-commits mentioned this pull request Mar 12, 2026
Closed
hogeheer499-commits added a commit to hogeheer499-commits/llama.cpp that referenced this pull request Mar 12, 2026
@hogeheer499-commits
ggml-cuda: fix UMA memory detection for HIP/ROCm on AMD APUs
73357da 
AMD APUs report prop.integrated=1 which triggers the UMA memory
path from ggml-org#17368. This overrides hipMemGetInfo() (accurate) with
/proc/meminfo MemAvailable (too low), losing ~30 GiB on a 128GB
Strix Halo system.

For HIP builds, only enter the UMA path when GGML_CUDA_ENABLE_UNIFIED_MEMORY
is explicitly set. This preserves correct behavior for both cases:
- Default: hipMemGetInfo() reports accurate TTM-backed memory
- GGML_CUDA_ENABLE_UNIFIED_MEMORY=1: /proc/meminfo is used (system RAM mode)

Tested on AMD Ryzen AI MAX+ 395, Radeon 8060S (gfx1151), 128GB, ROCm 7.1.

Fixes: ggml-org#18159
hogeheer499-commits added a commit to hogeheer499-commits/llama.cpp that referenced this pull request Mar 19, 2026
@hogeheer499-commits hogeheer499-commits
ggml-cuda: fix UMA memory detection for HIP/ROCm on AMD APUs
a65750f 
AMD APUs report prop.integrated=1 which triggers the UMA memory
path from ggml-org#17368. This overrides hipMemGetInfo() (accurate) with
/proc/meminfo MemAvailable (too low), losing ~30 GiB on a 128GB
Strix Halo system.

For HIP builds, only enter the UMA path when GGML_CUDA_ENABLE_UNIFIED_MEMORY
is explicitly set. This preserves correct behavior for both cases:
- Default: hipMemGetInfo() reports accurate TTM-backed memory
- GGML_CUDA_ENABLE_UNIFIED_MEMORY=1: /proc/meminfo is used (system RAM mode)

Tested on AMD Ryzen AI MAX+ 395, Radeon 8060S (gfx1151), 128GB, ROCm 7.1.

Fixes: ggml-org#18159
@ehfd ehfd mentioned this pull request Mar 23, 2026
Merged
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
dda6356 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
355a08d 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
39436d1 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
ea71f95 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@ServeurpersoCom ServeurpersoCom mentioned this pull request May 23, 2026
Open
phibya pushed a commit to ziee-ai/llama.cpp that referenced this pull request May 29, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
87a1aec 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
fewtarius pushed a commit to fewtarius/CachyLLama that referenced this pull request May 30, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
112b6aa 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@KI3P KI3P mentioned this pull request Jun 19, 2026
Open
This was referenced Jun 29, 2026
Closed
Open
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (ggml-org#17368)
3111cb1 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
@123-code 123-code mentioned this pull request Jul 7, 2026
Open
MrLordCat referenced this pull request in MrLordCat/llama.cpp-with-GUI Jul 16, 2026
@sfudally-nvidia @ggerganov
DGX Spark: UMA support (#17368)
4bb28e8 
* DGX Spark: UMA support

* Updates from PR feedback

* More PR feedback cleanup

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Remove trailing whitespace

* Update ggml/src/ggml-cuda/ggml-cuda.cu

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ORippler ORippler
@ggerganov ggerganov
@slaren slaren

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning Nvidia GPU Issues specific to Nvidia GPUs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

None yet

4 participants

@sfudally-nvidia @ggerganov @slaren @ORippler