'vte_terminal_select_all' doesnt select "all"

marked gnome-terminal#7858 (closed) as a duplicate of this issue

marked this issue as related to gnome-terminal#7858 (closed)

Would it make more sense to have a "copy all" feature instead of a "select all" feature?

I can't imagine any scenario where a "select all" is not followed by a copy operation.

Anyway, a big +1 for getting this feature restored.

locked this issue

mentioned in issue #2520 (closed)

unlocked this issue

marked #2520 (closed) as a duplicate of this issue

marked this issue as related to #2520 (closed)

Fixing this depends on glib!2189 (merged).

I see that this was merged. Does that mean that the patch on #2520 (closed) is not needed? I'd like to know whether I should keep working on it or leave it (if you haven't had time to look into it yet, excuse me).

Thank you for your time.

changed milestone to %vte 0.68

mentioned in commit ab85f122

I got here from googling about this extremely weird regression in such quite basic behavior... Seeing the current state of affairs regarding this, I cannot do much beyond acknowledging that it is a well known issue that has been poorly handled. So as to attempt being constructive here, I'd like to share why I think that.

Quoting from gnome-terminal#288 (comment 1019219):

the commit is intended to fix the crash only, and will be reverted when the underlying issue is fixed

Not disagreeing as a quick fix for a crash! I myself am a developer, and would do the same... as long as the expected life for this kludge was a couple weeks, maybe a month, tops!!

Until then, it will not be reverted, nor will I introduce a pref for this.

This is where the decision got sideways (always IMHO, anyway). Breaking the complete feature for more than a year, made it from a very understandable decision, into a very bad one, to be honest. Publishing an official release, at all, with this change, was an outright mistake.

Now this breakage got into the release of Ubuntu 22.04, a Long Term Support version, so we'll have a nice, broken build of Gnome Terminal for the years to come. I guess other distros will have included this too in their release cycle. I sincerely believe it is a pity that this was allowed to happen.

So if there is anything constructive I can add, is to encourage reflecting about the process of decision making by which kludges like this can end up being included in official releases, which then spread across all of its users. I understand that shipping a version that could crash with too many lines in the buffer is also not ideal, but almost anything would have been better than just restricting the feature to select the current screen. For example, from the tip of my mind: using the scrollback limit as the select-all limit, would have been a much better compromise. Another alternative: rethink the stance on "nor will I introduce a pref for this". Anything but the extreme solution that was deemed as best choice, of shipping a totally broken feature. I guess, paraphrasing Linus, to aim for a bit of "don't break UX" kind of thinking.

changed milestone to %vte 0.72

locked this issue

changed milestone to %vte 0.74

unlocked this issue

marked #2619 (closed) as a duplicate of this issue

marked this issue as related to #2619 (closed)

mentioned in issue gnome-builder#2070 (closed)

marked gnome-builder#2070 (closed) as a duplicate of this issue

marked this issue as related to gnome-builder#2070 (closed)

A friend was asking me about gnome-terminal and I was curious to see if there had been more feedback here. Seeing that not, I'd only wish to encourage resolution of this issue, especially in the name of users who not necessarily voice their opinion in bug trackers, but just arrive to gnome-terminal and assume that the current state of things is the normal behavior one should expect from a terminal application.

Hopefully Ubuntu 24.04 LTS doesn't come around and still bundles this as part of their frozen feature set, all the way until 2029. It would be such a bummer if it did.

If someone is interested...

I fixed the issue in bte https://github.com/cafe-desktop/bte (reverting the culprit commit)

bte is a vte fork which use ctk https://github.com/cafe-desktop/ctk

Ctk is a gtk3 fork (can be installed without conflicts)

And both (bte+ctk) are working inside cafe-terminal https://github.com/cafe-desktop/cafe-terminal

cafe-terminal is a mate-terminal fork (it requires ctk theme installed to improve the look)

mentioned in issue gnome-builder#2095 (closed)

mentioned in issue gnome-terminal#8031

mentioned in issue #2495

changed milestone to %vte 0.76

Could we please get it reverted for 0.76? People are complaining here and elsewhere. @chpe you having tagged the issue with that milestone gives me hope.

With the underlying GLib issue being fixed, can we be certain that we won't crash under reasonable circumstances? I mean, with infinite scrollback and driving the terminal to its fullest it's just a matter of minutes to produce scrollback data that doesn't fix in RAM and then the kernel starts OOMing or whatnot. But how fair is it to say that it's extremely unlikely for this to happen in real life, and it's a user error if the user starts to deliberately push the limits to the extreme? IMO fair, non-crashing, convenient use for the vast majority of users is more important.

I've bumped the GLib requirement. Not sure what to do with "max_allowed_version".

I have two casts to vte::grid::row_t, most of the similar code elsewhere casts to long instead. They're the same, so it's cosmetical only.

0001-Revert-widget-Limit-select-all-to-the-writable-regio.patch

A big select-all will also crash (or take very looong) while trying to transfer the text to X11. So IMHO a limit is still required, even if it's a large one.

Would dumping everything straight to a file (without going through the process of selecting and transferring to X11) be more feasible?

We have a "Save Contents" menu entry in unstable gnome-terminal versions only.

I guess we could enable that in stable versions as well if it writes the file without building it all up in memory first; even if it means gnome-terminal blocks for the duration of this operation. @chpe What do you think?

That being said, I'd really like to find a better solution for 0.76.

There's an easy workaround for the currently missing "really Select All" functionality.

Scroll to the top with Shift+Home. Select a bit of text. Scroll back to the bottom with Shift+End. Extend the selection using Shift+click.

With a mere 10 M line scrollback at 80 columns, filled with ls -lR output, a total of 500 MB of data (UTF-8, no text attributes) (whereas I have 8 GB of RAM) twice caused a complete lockdown of the X11 session for 4-5 minutes until gnome-terminal-server got killed, and once I couldn't recover from the situation even after 10 minutes, I had to force a reboot.

So the problem is serious, and disabling the "Select All" menu entry does not fully solve it.

There are other problems too with giant scrollbacks.

The scrollback is stored on disk. On some systems this might be a more scarce resource than RAM. Filling up /tmp might lead to other applications functioning incorrectly. It also probably causes VTE to misbehave. Ideally it should be tolerant against a corrupted scrollback buffer, but in practice, this has been untested through quite a few changes, so more likely than not it actually cannot handle this situation flawlessly. Also let's note that rewrapping on resize needs a bit of additional scrollback storage on disk for its duration.

Rewrapping on resize becomes slower. On my computer, pre-#2657, it's roughly 0.2 seconds for 1 M lines (becomes noticeable), i.e. around 2 seconds for 10 M lines (annoying), or 20 seconds for 100 M lines (unbearable).

Searching in the scrollback buffer, jumping to the previous prompt etc. all become proportionally slower.

(But these are at least good in the sense that the cost (time) is proportional, unlike with the OOM-ing situation of copying to X's Selection buffer or clipboard.)

What could be a solution?

Limiting the "Select All" functionality is not good. It's annoying for many-many users who would use it sensibly. It's not a full protection either, there's an easy workaround that's still easily subject to OOMing.

Limit the amount of data that we copy to an arbitrary amount, let's say 100 MB? If the data is larger then the copy operation would silently fail?

Add a warning to gnome-terminal-preferences? If the value is above a certain threshold, let's say 1 M lines, then show a warning to expect degraded performance and the exhaustion of system resources, we don't recommend it, do it at your own risk?

Limit scrollback size? Forget unlimited, cap at 1 M lines? In g-t, or in VTE (i.e. for all other frontends too)?

Or limit the size to 1 M in g-t-p, but allow the user to increase it in dconf? Could be an easy and good enough way of communicating that you should only do that if you really know what you're doing.

While at it, enable "Save Contents"? That 500 MB of data I was talking about was saved so instantaneously that I first though the feature was broken :) It works great.

Add a warning to gnome-terminal-preferences? If the value is above a certain threshold, let's say 1 M lines, then show a warning to expect degraded performance and the exhaustion of system resources, we don't recommend it, do it at your own risk?

The default scrollback limit in GNOME Terminal and Console is 10 K lines so allowing 1 M lines without warning - 100 times as much - would be a usable upper limit I think.

The 10 K limit can often be too short when I'm troubleshooting something and want to save what I've already tried and diagnosed.

mentioned in issue gnome-terminal#8057

Something like this for gnome-terminal's gtk3 branch? The profile editor window shows a warning on scrollback sizes >= 1M lines. (The warning text wraps if a translation is longer.)

~~0001-profile-editor-Warn-about-large-scrollback-buffer-si.patch~~

I'd be fine with reverting the VTE limitation along with this g-t change. Yes it could still crash (as can it now, if you extend the selection), but the user is warned.

Needs English proofreading (e.g. should it be "the exhaustion"? or find a totally different wording?).

Needs a string change announcement, or approval if committed after Saturday.

gtk4 port yet to be written.

WDYT?

I would prefer to have a limit for the sake of reliability and performance, but maybe too many people would be upset if it was (re-)introduced. You said the scrollback is stored on disk. What if you limited how much can be kept in RAM and displayed, but provide means for g-t etc to provide a menu command to show the backing file in the filer and/or flag it to be retained after the tab is closed?

The warning is fine without "the" before exhaustion, but it would be better if it began with "A large scrollback buffer," or "Large scrollback buffers." And it might be better to use "may" instead of "can". As a native English speaker that's more by instinct than formal learning, but this might help.

I would prefer to have a limit for the sake of reliability and performance

As the author of a VTE-based terminal, you can cap the max lines you offer. Or do you mean to still offer arbitrarily large scrollbacks, but limit "copy" and similar operations to let's say 1M lines?

As far as I understand, chpe is reluctant to introduce arbitrary limits. They are, well, arbitrary, we can't know if they're way too small or way too large depending on the user's system and other circumstances. We probably can't query the system resources and load information in a cross-platform way, and even if we could, we wouldn't know the formula to compute the safety limit. E.g. why limit at 1M lines if the user's system can safely handle 2M?

I was shocked to see that X11 can't handle 0.5 GB of selection data on a computer with 8 GB of RAM. I don't know how well it scales. I don't know whether Wayland has the same problem. Etc. It's unreasonable to study their source code or drive experiments to figure out the formula.

I'm hoping that maybe he'll be fine with my proposed solution whereas only a UI warning has an arbitrary (and hopefully safely low) threshold but the actual behavior doesn't. Probably we should also amend VTE's API documentation to suggest that frontend terminal apps show a similar warning. But I don't know yet what he thinks of this overall direction.

You said the scrollback is stored on disk. What if you limited how much can be kept in RAM and displayed

It's very limited how much is kept in RAM. The problem is not here.

but provide means for g-t etc to provide a menu command to show the backing file in the filer and/or flag it to be retained after the tab is closed?

I'm not sure I understand you... what's the problem you try to address here?

The scrollback is currently stored in 3 compressed and encrypted files. The files are unlinked right after creation, so you can't even find them in a file manager (except if the OS provides a special interface, such as /proc/<pid>/fd on Linux). This way if the terminal crashes then the operating system automatically frees up the data. Similarly, if the OS crashes then the file system will reclaim that area at the next boot.

The encryption key is destroyed as soon as the given terminal (VTE) is closed, therefore the real contents of the files aren't recoverable. Even if they were (i.e. even if we stripped off the compressing and encrypting layers), the file format is internal to VTE and is subject to change at any time without notice, just like any internal data structure in VTE's memory. You'd need to study the file format in VTE's source code to understand what's in those files. And you'd still need some variables from VTE's memory that aren't written to these files.

What would be the use case for anyone ever wishing to access these files? And how is it relevant to the bug we're discussing? I don't get it.

I've marginally mentioned the "Save Contents" menu entry which I'd love to enable even in stable versions. Maybe accompanied with a "Save Contents as HTML", too. They'd be a safer (OOM-wise) replacement for copy-pasting. They'd include all the data, not just the scrollback but even the onscreen, mutable parts. They'd create a new file, as they should; rather than giving access to the scrollback's backing files which are in a vastly different format, don't include the mutable part (and a bit more than that), and keep changing after the requested "Save Contents" operation.

Large scrollback buffers

I wrote this initially, but then removed that "s". Should have gone with my initial instinct. Thanks for the linguistic comments!

0001-profile-editor-Warn-about-large-scrollback-buffer-si.patch

Linguistic changes only.

As the author of a VTE-based terminal, you can cap the max lines you offer. Or do you mean to still offer arbitrarily large scrollbacks, but limit "copy" and similar operations to let's say 1M lines?

That's a good point that I can choose my own limit. I had only recently learnt that vte saves the buffer to disk and hadn't thought about how that works, so I thought I'd prefer a RAM only approach. However, now I see that's probably not practical, because assuming a user has an adequate amount of RAM and/or swap is a worse assumption than disk capacity being larger than RAM.

but provide means for g-t etc to provide a menu command to show the backing file in the filer and/or flag it to be retained after the tab is closed?

I'm not sure I understand you... what's the problem you try to address here?

I was thinking about the problem similar to the one Jake Dane described. What if you want access to the start of a very long history, but don't realise until you have already exceeded the scrollback limit? Again, I didn't understand that the scrollback on disk is protected by encryption and unlinking, which you've just explained.

The scrollback is currently stored in 3 compressed and encrypted files. The files are unlinked right after creation, so you can't even find them in a file manager (except if the OS provides a special interface, such as /proc/<pid>/fd on Linux). This way if the terminal crashes then the operating system automatically frees up the data. Similarly, if the OS crashes then the file system will reclaim that area at the next boot.

Ah, interesting. Is the use of 3 files, instead of just one, something to do with solving the problem of how to truncate the history from the start efficiently?

I've marginally mentioned the "Save Contents" menu entry which I'd love to enable even in stable versions. Maybe accompanied with a "Save Contents as HTML", too. They'd be a safer (OOM-wise) replacement for copy-pasting. They'd include all the data, not just the scrollback but even the onscreen, mutable parts. They'd create a new file, as they should; rather than giving access to the scrollback's backing files which are in a vastly different format, don't include the mutable part (and a bit more than that), and keep changing after the requested "Save Contents" operation.

I implemented "Save Contents" in ROXTerm as a workaround for this issue with "Select All", and it was well-received (#250). You imply it shouldn't be enabled in stable versions; why is that? The only potential problem I'm aware of is that it can block the UI thread. Perhaps we should open a new issue to make an asynchronous version of vte_terminal_write_contents_sync?

Ah, interesting. Is the use of 3 files, instead of just one, something to do with solving the problem of how to truncate the history from the start efficiently?

No. Truncating is done efficiently within one file, using a special method reusing the beginning of the file. Search for "snake" in VTE's source if you're interested in the layout.

The three files are: text_stream for the raw text in UTF-8, attr_stream encodes the attributes and OSC 8 hyperlinks (along with the start/end positions of their runs, pointing to text_stream), and row_stream is a mapping from row numbers to the corresponding offsets within the previous two files.

You imply it shouldn't be enabled in stable versions; why is that?

It's not enabled in stable gnome-terminal versions. Probably because it's blocking the entire app. But I don't see that as a real life problem. I think we should enable it. You can also enable it in your app (I think the VTE feature is there unconditionally).

row_stream is a mapping from row numbers to the corresponding offsets within the previous two files.

I should think it's difficult to deal with soft wrapping of long lines, making sure the Gtk.Adjustment properties remain consistent over the entire buffer.

No. Truncating is done efficiently within one file, using a special method reusing the beginning of the file. Search for "snake" in VTE's source if you're interested in the layout.

Understood. The snake's head chases its tail.

[Save Content is] not enabled in stable gnome-terminal versions. Probably because it's blocking the entire app. But I don't see that as a real life problem. I think we should enable it. You can also enable it in your app (I think the VTE feature is there unconditionally).

I already did. A non-blocking version would be nice, but I expect you've got more important things to deal with. I'd better stop taking up your time in case you're trying to beat the freeze for Ubuntu 24.04.