Something is seriously wrong with FreeBSD implementation for PowerPC: it is practically unusable as of now, freezing for more than a minute at every input. Same behavior during installation and with installed system. Pattern is consistent: you have a 1–2 seconds to type something or make a selection in a menu, then everything freezes, after half a minute fans go high speed, that condition lasts for another minute, then fans go back to normal, and system unfreezes. It is not occasional, but happens literally at every window during installation and every text command in the terminal. Installation process is rather buggy and super-fragile too: I had random quits in the middle a number or times; in the beginning when installer boots, sometimes there are numerous USB-related errors, but sometimes they are skipped. I have tried it on two PowerMacs, both reasonably fast (G5 Quad 16 GB RAM SSD, G5 2.3 DC 9 GB RAM SSD) – in MacOS I use them daily for development without any pain. I also tried using HDD instead of SDD – nothing changed. Quad has ATI GPU installed, while 2.3 has NVidia, if that matters. I wanted to have FreeBSD as a second system to work on improvement of Macports support for it (so that I can test my ports not only in MacOS but also in FreeBSD). Hope someone can help to sort this issue out. P. S. I am aware PowerPC is Tier II support, but this is a matter of system simply being usable or not at all. It obviously should not be the case that it takes several minutes to type in login and password with freezes every few letters typed.
Is a -current snapshot any better? Does 12.x work any better?
(In reply to Sergey Fedorov from comment #0) How are the HDD(s) and SSD(s) connected? > … when installer boots, sometimes there are numerous USB-related errors, > but sometimes they are skipped. … In the absence of a photograph, I might assume hardware issues.
Created attachment 242607 [details] An instance of USB errors
(In reply to Graham Perrin from comment #2) I do not think so, since it happens on two different machines which otherwise work perfectly fine. However I add a pic now of USB-related issues. Please notice those are not central to the matter though. Freezing happens regardless.
Created attachment 242608 [details] USB error upon intro screen load This happened 2–3 times out of numerous install attempts. I.e. not consistent, nevertheless did happen on both machines.
I will re-upload pics, compression killed them. Sorry.
Created attachment 242609 [details] USB errors when booting (resolution fixed)
Created attachment 242610 [details] USB error upon intro screen load
Created attachment 242611 [details] Booted system on PMG5 2.3 Installed and booted system on PowerMac G5 2.3 DC. No errors on boot, but it was badly freezing even when entering login/password and every other command.
(In reply to Graham Perrin from comment #2) > How are the HDD(s) and SSD(s) connected? Native SATA bus connection. I.e. not a PCIe card, FW drive or anything else non-default. SSD were a direct replacement for HDDs. The PowerMac where I had FreeBSD installed now has HDD in upper bay which is used for FreeBSD (I moved macOS onto SSD).
(In reply to Warner Losh from comment #1) I don’t have spare DVDs around to check that. Is there a way to make install image recognizable from a FW drive? That works fine for MacOS installation, but I could not get it work for FreeBSD. USB drive might be an option too, late PowerMacs support USB booting, unofficially.
In attempting to install FreeBSD on PPC I found that utilizing the 13.2 DVD iso with a G5 1.8 Ghz single processor and Nvidia FX5200, the install was smooth and rapid, working nicely in text mode (but no success on the graphic side). Moving forward I attempted an install of 13.2 on my favored device, a G5 1.8 Ghz dual processor with an ATI 9600XT card, but the install was painfully slow... marginal graphic support (card is "supported"). So not to be thwarted, I pulled out my last remaining G5, an identical G5 dual 1.8 with the 9600 card and attempted the same install... once again, the process was incredibly slow with the same outcome observed as with the first dual. Just to confirm my suspicion, I wiped the drive on the original G5 single processor and began an install. Once again, with the single, the process was quick and smooth. So basically I found 13.2 very workable on a G5 single. However, it has been my experience and that of others that install on a multiprocessor PPC is exceedingly challenging and as noted by opening poster renders 13.2 essentially unusable on these devices. Good news: Swapping in a 9600Xt card into the G5 single has yielded fully functional FreeBSD on PPC. Bad news: 13.2 would currently seem to be a no go on the multiprocessor devices. Hopeful that this may addressed and remedied to some extent. P.S. all drives standard HDD.
Does 13.1 work on multiprocessor/multicore G5 machines? Unfortunately my FreeBSD G5 died 5 years ago, so I haven't been able to maintain the G5 at all, I've been using only the Power9. I recommend testing 12.x (latest), 13.1, and 13.0 (if 13.1 is slow). That could help narrow down when the slowdown occurred.
(In reply to Justin Hibbits from comment #13) To answer your question I've tried several release on my PowerMac G5 Quad (PowerMac 11,2 with pcie bus) using the DVD drive. Here are the results 12.4 Boots and installs just fine. 13.0 Kernel loads but painfully slow, very similar behaviour to Sergey's issue. Loads normally until "Root mount waiting for: usbus2" then fans are roaring and takes forever to get to the Install menu. 13.1 Pretty much the same as 13.0, except even worse as it simply reboots on its own from time to time. 13.2 Doesn't load at all, Black screen with fans roaring immediately. 14.0 Latest Snapshot, Same observation as 13.2. According to what I've seen, the Toolchain/ABI starting with release 13 has departed from GCC4 which makes it even more red herring. It would be really awesome to get a PPC970/MP functional 64bit release with modern toolchain ! I'd like to help testing/troubleshooting as I fear this is the latest opportunity we have until this platform fall into oblivion. Cheers, Michael
(In reply to Mitch from comment #14) Thank you very much for testing! P. S. It would be great if someone could address this and fix in the current master of FreeBSD. There are many people using PowerPCs, and it remains the only affordable/accessible option for the most end-users as well as developers. Given that other realizations of BSD apparently are fine, it is certainly not something unfixable.
Any update on the matter from PowerPC port maintainers? We have FreeBSD working on G4 but broken on G5, this is very upsetting.
FreeBSD 14 (ppc 32-bit) does not boot on G5 at all. Tries to load in OF, then quits. (I have probably not tried 13.x ppc on G5, not sure if that one works. Initial report was about ppc64 version.)
I can confirm this behaviour. I had a fine running FreeBSD 12 test disk which I nulled to try the latest stuff (14). But the 14 installer (RELEASE and STABLE from 18th of April) is so slow that it is impossible to use.
can also confirm now that 12.4 is installing and running fine on a dual g5 powermac.
(In reply to Sergey Fedorov from comment #15) Unfortunately I think the number of committers with PowerMac G5 hardware supporting FreeBSD is dwindling as the machines age. My only G5 that ran FreeBSD died back in 2018, so I'm unable to do anything significant for it. That said, I'll support as best as I can. Regarding the "USB error upon intro screen load" that's often solved by unplugging/plugging the device (happens all the time with one of my FTDI debuggers), not a problem with FreeBSD itself. The "Root mount waiting for usbus2", unless you have root on USB you can try setting the hw.usb.no_boot_wait tunable; it looks like usbus2 is having enumeration problems (just a guess). For 13.x performance, you can try testing a commit before, and after, c583b02587 (PowerMac timebase sync for G4), which should be a nop for G5, but might not be. Other possibilities are pmap related (superpage support was added in the 13 timeframe, I believe).
If anyone can build a kernel without c583b02587 to test I can try that. Since 12.4 has not even working packages anymore I can't do anything.
(In reply to Denis Ahrens from comment #21) I can try to cross compile a 14 PPC64 Kernel without c583b02587 on my x86_64 server if I can find instructions on how to achieve this. I also have a functional PowerMac 11,2 Quad to test.
(In reply to Mitch from comment #22) You can do something like: git checkout main git checkout -b tracking-branch git revert $HASH make kernel-toolchain TARGET_ARCH=powerpc64 make buildkernel KERNCONF=GENERIC64 TARGET_ARCH=powerpc64 will build it. you can install it to the destdir fo your choice: mkdir $HOME/powerpc-test-kernel make installkernel KERNCONF=GENERIC64 TARGET_ARCH=powerpc64 NO_ROOT=t
(In reply to Justin Hibbits from comment #20) If someone could build the image to try (or if installing from a FireWire drive works, then I can clone DVD image there and just replace the kernel, but I still need the new kernel), I will definitely test it.
Hello everyone, In my attempt to build a Kernel without c583b02587, I was initially confronted to this conflict below mitch@bsd:~/freebsd-src $ git status On branch tracking-branch You are currently reverting commit c583b0258728. (fix conflicts and run "git revert --continue") (use "git revert --skip" to skip this patch) (use "git revert --abort" to cancel the revert operation) Changes to be committed: (use "git restore --staged <file>..." to unstage) modified: sys/conf/files.powerpc modified: sys/powerpc/powermac/macio.c modified: sys/powerpc/powermac/platform_powermac.c Unmerged paths: (use "git restore --staged <file>..." to unstage) (use "git add/rm <file>..." as appropriate to mark resolution) deleted by them: sys/powerpc/powermac/platform_powermac.h deleted by them: sys/powerpc/powermac/tbgpio.c I've simply removed both files mentioned above in the unmerged section and I was able to build a Kernel which I assume was the right thing to do, if not please let me know. Thanks to Warner's instructions I then cross compiled a test kernel against the main branch from the freebsd source tree which appears to be the current 15 revision. You can find it here : https://box.ochem.net/cloud/index.php/s/Q6aCBiFdWrk69rb I haven't test it myself but I assume you could simply rename it to kernel.txz and do an in place replacement for testing purposes. If needed I could also build a Stable/14 version of it. Cheers, Mitch
I tested the kernel from Mitch and sad news is that the problem is still there.
(In reply to Denis Ahrens from comment #26) Other big changes that are PPC970 relevant are in the pmap logic. If you take a look at the git log of sys/powerpc/aim/mmu_oea64.c and sys/powerpc/aim/moea64_native.c, you can try to revert (git revert, or a spiritual revert if there are conflicts) some of them and see what impacts. Some options: e2d6c417 (superpages) a79540111 (micro-optimizations of PVO-PTE logic) bc94b7009 (re-merge ISA3 HPT with moea64 native HPT) -- this one *will* have conflicts. Another shot in the dark is e44ed9d34 (atomics) This is all just a guess based on trimming through `git diff origin/releng/12.4..origin/releng/13.1 sys/powerpc`. There may be other issues, but these are the first I see as possibilities.
Created attachment 251098 [details] timebase sync hack I'm not sure my symptoms exactly match what's described here, but it seems like timebase sync may be broken on G5. ofwcons freezes for about a minute every so often, the fans go full blast at the same time because powermac_thermal is hung. The system is usually (always?) responsive over ssh while this is happening. Anyway, the attached hack seems to help in my case, as least. Only tested on the later PCIe dual and quad G5s. I'm not sure timebase sync is even supported on PCI-X powermacs (definitely not on my Xserve G5 with weird sync via I2C).
(In reply to ben from comment #28) I think your patch is in the right direction, but I think the explicit memory barriers are overkill in part. We likely need to use acq/rel semantics with the atomics, and always do a load_acq() when checking cpu_done. Something like using: atomic_add_rel_int(&cpu_done, 1); (in both cases) while (atomic_load_acq_int(&cpu_done) < mp_ncpus) ; And maybe do the same for tb_ready.
Created attachment 251202 [details] tbsync patch 1 (In reply to Justin Hibbits from comment #29) Okay. Thanks for the pointers. My machine seems happy with the attached patch. Hopefully I understood enough of atomic(9). As you suspected, it didn't work without also changing tb_ready.
Comment on attachment 251202 [details] tbsync patch 1 All the atomic_set_*()s should really be atomic_store_*(), `atomic_set` is a bit-set (OR operation). It's incorrect in the existing code, too. It really doesn't make a significant difference, since the end result is the same (cpu_done should be 0 to start with, so setting nothing doesn't change that, tb_ready should be 0 to start with, so setting 1 bit does the same thing), except in the last instance to try to clear tb_ready. In the tight loop checking if cpu_done is high enough, you don't need the acq barrier in there, because the thread fence occupying the loop does the same thing, and acq places the barrier after the operation, so in the same place anyway. Glad to see this patch overall works for you, though! I'm really surprised those barriers are needed, since they're not needed in the mpc85xx case, which this is derived from.
Created attachment 251226 [details] Kernel Trap
Comment on attachment 251226 [details] Kernel Trap That's an illegal instruction trap, which is odd. You'd need to 'x/i 0xc00000000a1a9c28' to see what the instruction is.
Comment on attachment 251226 [details] Kernel Trap Sorry, I tried to add a comment with this screenshot but it didn't quite work as expected :) So first of all reverting c583b02587 did nothing at all as I'm observing the same symptoms as Ben. I've managed to painfully install the lastest snapshot with a machine continually freezing (running okay for 15 secs then fans running full blast during which the computer is unresponsive for a minute of so). Once the install completed, I did a reboot and I was welcomed by a Fatal Kernel Trap like the one I've posted, sometimes I get to see it and others the machine reboots immediately. I then tried to take an image from the drive with my FreeBSD PC but I can't see any partitions on it (I don't remember if the install wizard defaults to UFS or ZFS). Please let me know if that would be of any use for troubleshooting. I haven't yet tried Ben's patches but if it works for him there is a chance it would work for me as we're using the same HW, I'm super happy that we're making some progress ! Also there is something rather strange I've noticed with recent Images, If I boot them while holding C (cdrom) all I get is a black screen with fans roaring immediately but if I use the option key instead and wait for the watch cursor to turn back to a regular mouse cursor I'm able to boot just fine, I'm guessing that might be caused by a bunch of OF variables. Thanks everyone for your efforts so far !
(In reply to Justin Hibbits from comment #33) Yesterday I've tried Ben's patch and my machine was way happier, unfortunately the same problem still occurs, just less often. I was able to install 14.1 and use it for a few minutes before the machine crashed while idling. Just trying to understand why we'd want to use memory barriers with the powermac_smp_timebase_sync function. Is it because PPC is using a weak memory model which would be problematic in an SMP context like explained below ? https://preshing.com/20120710/memory-barriers-are-like-source-control-operations/ https://preshing.com/20120930/weak-vs-strong-memory-models/ You mentioned that the code is derived from mpc85xx, do we have such a machine in the field which is running a modern release properly ? Also how was FreeBSD handling timebase sync before the introduction of plateform_powermac ? I'll try your suggestions and test it again.
(In reply to Mitch from comment #35) mpc85xx platforms I've personally tested with FreeBSD are the AmigaOne X5000 and AmigaOne A1222. I wrote the mpc85xx timebase sync specifically for those targets, after seeing strange hanging behavior. As for why we want to use memory barriers, we really need just execution barriers (isync), I think, because we need to make sure the timebase is correct before it's unlocked by the BSP. This can only be ensured by using a barrier between setting the timebase and declaring done (atomic_add_int() of cpu_done). But, yes, a weak memory model does mean we need more explicit syncs where on strong models they would be implicit (but pay the penalty on all accesses). We really shouldn't need any syncs for tb_ready, because it can be done lazily. The only sync we should really need is the cpu_done. Before this timebase sync we used a synchronization mechanism at AP launch time, where we simply "hoped" that they were close enough to all get the same timebase. This is obviously problematic. It was changed during the 12-CURRENT time frame. The powermac sync change was done in 2021, though.
(In reply to Justin Hibbits from comment #36) Thanks for your detailed reply, yesterday I've installed git and tried to clone the FreeBSD source tree during which the machine crashed. I've archived /var/crash here, The initial crash was self induced to make sure dumping into /var/crash would work and Vmcore.1 is the Kernel crash dump which was created after my Git clone attempt. https://box.ochem.net/cloud/index.php/s/BEcGjstARB8bz9J
(In reply to Mitch from comment #37) To see what's causing the program interrupt you'd need to examine the kernel at crash time. Though, it's possible there's a compile issue that's generating code for newer powerpc64. Just check objdump on your kernel, for 0xa5081a8 address, and see what's at that address.
I got a Power Mac G5 Model A1047 and tried the patches in attached to this bug and nothing changes, it still unstable. As information, "git checkout 6c346639ba986a285e456099bc2faf1fe17954df" seems run OK on my G5. Next commits are unstable but I don't see any reason by looking at the code changes.
(In reply to Alfredo Dal'Ava Junior from comment #39) If anyone could make an image from the said commit, I will try it on G5.
(In reply to Alfredo Dal'Ava Junior from comment #39) On which branch are you working ? I wasn't able to find commit 6c346639ba986a285e456099bc2faf1fe17954df nor what it does.
(In reply to Mitch from comment #41) This commit is from stable/13 branch, but I think it's only coincidence, if I build the next commit it freezes. Other commits like 25f4ddf, fb0d2551 and 303b7702 are working on this G5 as well. One interesting thing is it works after after a cold boot, but they freeze after a soft reboot. Anyway, here is the the ISO from 6c34663 https://1drv.ms/f/s!ArFAjrJ44JgggQrEVu5Msz8VUagv?e=AupOie