Skip to content

ROCm Segmentation fault (core dumped) with -ngl 1 #2797

New issue
New issue
Closed
Closed
ROCm Segmentation fault (core dumped) with -ngl 1#2797

Description

@grigio
grigio
opened on Aug 25, 2023

ROCm works in the docker container but if I try to offload a layer I get a segfault

rocm-smi --showmeminfo vram



======================= ROCm System Management Interface =======================
============================= Memory Usage (Bytes) =============================
GPU[0]		: VRAM Total Memory (B): 536870912
GPU[0]		: VRAM Total Used Memory (B): 25616384
================================================================================
============================= End of ROCm SMI Log ==============================


./main  -ngl 1 -m /models/llama2_7b_chat_uncensored.gguf.q2_K.bin -p "write a poem"

root@11fe149603ba:/app# ./main  -ngl 1 -m /models/llama2_7b_chat_uncensored.gguf.q2_K.bin -p "write a poem"
main: build = 0 (unknown)
main: seed  = 1692992765
ggml_init_cublas: found 1 ROCm devices:
  Device 0: AMD Radeon Graphics, compute capability 10.3
llama_model_loader: loaded meta data with 18 key-value pairs and 291 tensors from /models/llama2_7b_chat_uncensored.gguf.q2_K.bin (version GGUF V1 (latest))
llama_model_loader: - tensor    0:                token_embd.weight q2_K     [  4096, 32000,     1,     1 ]
llama_model_loader: - tensor    1:               output_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor    2:                    output.weight q6_K     [  4096, 32000,     1,     1 ]
llama_model_loader: - tensor    3:              blk.0.attn_q.weight q2_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor    4:              blk.0.attn_k.weight q2_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor    5:              blk.0.attn_v.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor    6:         blk.0.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor    7:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]
...
llm_load_tensors: ggml ctx size =    0.09 MB
llm_load_tensors: using ROCm for GPU acceleration
llm_load_tensors: mem required  = 2652.72 MB (+  256.00 MB per state)
llm_load_tensors: offloading 1 repeating layers to GPU
llm_load_tensors: offloaded 1/35 layers to GPU
llm_load_tensors: VRAM used: 81 MB
.................................................................................................
llama_new_context_with_model: kv self size  =  256.00 MB
llama_new_context_with_model: compute buffer total size =   71.91 MB
llama_new_context_with_model: VRAM scratch buffer: 70.50 MB

system_info: n_threads = 8 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 400, n_keep = 0


Segmentation fault (core dumped)

Activity

KerfuffleV2 commented on Aug 26, 2023

@KerfuffleV2
KerfuffleV2
on Aug 26, 2023
Contributor

Does setting HSA_OVERRIDE_GFX_VERSION=10.3.0 help?

env HSA_OVERRIDE_GFX_VERSION=10.3.0 ./main  -ngl 1 -m /models/llama2_7b_chat_uncensored.gguf.q2_K.bin -p "write a poem"

grigio commented on Aug 26, 2023

@grigio
grigio
on Aug 26, 2023
Author

@KerfuffleV2 No, always segfault

KerfuffleV2 commented on Aug 26, 2023

@KerfuffleV2
KerfuffleV2
on Aug 26, 2023
Contributor

Some other things to possibly try:

  1. Running with -lv (low vram option)
  2. Running with -nommq (turns off the custom matmul kernels)
  3. Try with -ngl 0 but a long prompt (say 100+ tokens). This will test whether evaluating the prompt on the GPU works.

(These things aren't necessarily going to help you personally, but testing them may help narrow down the issue.)

It's also very likely that running evaluation on APU rather than the CPU is going to be slower. So even if you do get it working, you may not benefit from using it.

jwlarocque commented on Aug 27, 2023

@jwlarocque
jwlarocque
on Aug 27, 2023

I'm running into the same issue, running in Docker rocm/dev-ubuntu-22.04:5.5-complete with an RX 6600. I've tried the first two flags with no change in behavior.

Edit: forgot to say:
Running -ngl 0 with a long prompt also segfaults (short prompts run fine on CPU).

(In my case you only see the segfault message when running from bash, if you docker run main it exits silently. Also without the GFX version override it won't build at all, so that part seems to be working.)

ghost commented on Aug 27, 2023

@ghost
ghost
on Aug 27, 2023

What devices did you pass into docker, did you pass renderDxxx in and not card0? Also do you have amdgpu-dkms installed on host?

SlyEcho commented on Aug 30, 2023

@SlyEcho
SlyEcho
on Aug 30, 2023
Contributor

VRAM Total Memory (B): 536870912

I think it's down to the small VRAM size (512MB). You can try to lower the context size and the batch size. Also using a small model like OpenLLaMA 3B.

I'm running into the same issue, running in Docker rocm/dev-ubuntu-22.04:5.5-complete with an RX 6600. I've tried the first two flags with no change in behavior.

The variable should be passed into the container with the -e switch:

docker run --rm -it \
  -e HSA_OVERRIDE_GFX_VERSION=10.3.0 -v/models/:/models --device /dev/kfd --device /dev/dri \
  llama.cpp:rocm --run \
  -m /models/open-llama-7b-v2-q4_K_M.gguf -ngl 99 -n 320 -p Hello

It's possible you need to pass in the DRI devices separately on some container runtimes: --device /dev/kfd --device /dev/dri/card1 --device /dev/dri/renderD128.

It's also possible the RX 6600 is not supported...

dataf3l commented on Oct 18, 2023

@dataf3l
dataf3l
on Oct 18, 2023

maybe try ./models instead of /models ?

KerfuffleV2 commented on Oct 18, 2023

@KerfuffleV2
KerfuffleV2
on Oct 18, 2023
Contributor

maybe try ./models instead of /models ?

That's not going to work unless the CWD is the root already, and any case it would just be the same.

They're mounting their models under /models with docker.

ekg commented on Dec 13, 2023

@ekg
ekg
on Dec 13, 2023
Contributor

I believe the problem is that the AMD APUs require a different allocator to use the common RAM pool for VRAM. See ROCm/ROCm#2014 (comment), which points at https://github.com/pomoke/torch-apu-helper.

A potential solution would be to use hipHostMalloc(&ptr,size,0) for allocation when building ROCm compatibility.

ekg commented on Dec 13, 2023

@ekg
ekg
on Dec 13, 2023
Contributor

I'll test an update the allocation patterns which exploit shared memory better. A problem is that these don't work on windows, so they'll have to be conditionally defined. https://rocmdocs.amd.com/projects/HIP/en/develop/user_guide/programming_manual.html

ekg
added a commit that references this issue on Dec 14, 2023

AMD ROCm: handle UMA memory VRAM expansions

d59c0b3
ggerganov
closed this as completedin 0f630fbon Dec 22, 2023
chboishabba
mentioned this on Mar 3, 2025
Seunghhon
added a commit that references this issue on Apr 27, 2026

cuda : ROCm AMD Unified Memory Architecture (UMA) handling (ggml-org#…

ca21628
phuongncn
added a commit that references this issue on Apr 28, 2026

cuda : ROCm AMD Unified Memory Architecture (UMA) handling (ggml-org#…

824b825
ljubomirj
added a commit that references this issue on May 7, 2026

cuda : ROCm AMD Unified Memory Architecture (UMA) handling (ggml-org#…

1c68df8
my-other-github-account
added a commit that references this issue on May 16, 2026

cuda : ROCm AMD Unified Memory Architecture (UMA) handling (ggml-org#…

3dfdc13
AlexiAlp
added 2 commits that reference this issue on Jun 2, 2026

cuda : ROCm AMD Unified Memory Architecture (UMA) handling (ggml-org#…

88f59b4

cuda : ROCm AMD Unified Memory Architecture (UMA) handling (ggml-org#…

edeca18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @grigio@ekg@SlyEcho@dataf3l@jwlarocque

        Issue actions