One thing to keep in mind when looking at GNU programs is that they're often intentionally written in an odd style to remove all questions of Unix copyright infringement at the time that they were written.
The long-standing advice when writing GNU utilities used to be that if the program you were replacing was optimized for minimizing CPU use, write yours to minimize memory use, or vice-versa. Or in this case, if the program was optimized for simplicity, optimize for throughput.
It would have been very easy for the nascent GNU project to unintentionally produce a line-by-line equivalent of BSD yes.c, which would have potentially landed them in the 80/90s equivalent of the Google v.s. Oracle case.
Remember that "yes" is the bottom end of complexity here, not the top. As the utility grows larger that stops looking like "asking for trouble" and starts looking like "often the only sensible solution". Expecting people to extend programs in C is often "asking for trouble", after all!
One point in favor of this simple version is that it's immediately obvious that it doesn't do the same thing as the OpenBSD version. In OpenBSD `yes a b c` will only print "a" while in GNU it prints "a b c". I did not catch that when I was reading the more complicated modern version.
At the risk of sounding like a copyright newbie, shouldn't that be covered by just doing a 'clean room' implementation? As long as you can verifiably prove that you didn't copy the source, it should fall under general use (as there's really only one way of doing such a thing), right? Much like Apple can't patent rectangles, although they tried.
Normally I think readability is more important than speed. But in this particular case, I think GNU is doing the right thing optimizing the code to the limit.
This is the beautiful part of Unix: small tools that do only one thing well. Programs following this philosophy are very good abstractions. They do one very well defined thing so you can use them without having to understand how they work. I have used Unix for years and I've never felt the need to read the source code for `yes`. And because they do a very small thing, even if you need to read them, the overhead of optimization is not that much, for example, the optimized GNU yes is just under 100 LOC if you remove comments and help boilerplate. Yes, it's longer than the BSD version, but it's just a matter of minutes to understands what it does.
I totally disagree. Nobody will ever want to use `yes` at 10 GB/s. They will want it to be reliable, and this sort of over-optimisation increases the risk of bugs.
I've used 'yes' many times to generate huge amounts of data quickly. Back then, it never had the small string optimisation, but you could always run 'yes InsertReallyLongStringHere' to spew out data much faster than /dev/urandom or even /dev/zero
I'm glad it runs fast, and I hope that all OS utilities are optimised (and tested, of course!) instead of making their source code pretty. The fact is, most people want to use programs, not read them.
I agree, all I remember is that when I tried it, /dev/zero sometimes sucked performance-wise. I can't recall the exact circumstances as it was some time ago, and could have been on any of Linux/FreeBSD/SunOS/HP-UX/IRIX - perhaps it was the fastest common way at the time?
On a recent x64 Linux, /dev/zero seems plenty fast enough now:
$ dd bs=8k count=819200 if=/dev/zero of=/dev/null
819200+0 records in
819200+0 records out
6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.331137 s, 20.3 GB/s
$ yes | dd bs=8k count=819200 of=/dev/null
819200+0 records in
819200+0 records out
6710886400 bytes (6.7 GB, 6.2 GiB) copied, 0.959551 s, 7.0 GB/s
So how did that redirect even work; should we be doing a `mknod` to make a "yes" device to make the comparison work (can we, does it help other than in my naive imagination).
I know this works, but how come we can see the output of pv when it is redirected to /dev/null? Maybe I just don't understand how pipes and redirection works since I rarely use Linux :(
> I know this works, but how come we can see the output of pv when it is redirected to /dev/null?
From pv's man page:
> Its standard input will be passed through to its standard output and progress will be shown on standard error.
> Maybe I just don't understand how pipes and redirection works since I rarely use Linux :(
The Windows/DOS command line has the same concepts[0], though it's probably used less often: by default a process has 3 FDs for STDIN (0), STDOUT (1) and STDERR (2).
At a shell
* you can feed a file to STDIN via <$FILE (so `</dev/zero pv` will feed the data of /dev/zero to pv), the 0 is optional
* you can pipe an output to a command (other | command) or the output of a command to an other (command | other)
* you can redirect the STDOUT or STDERR to files via 1> and 2> (the "1" is optional so 1> and > are the same thing) (you can redirect both independently)
* you can "merge" either output stream to the other by redirecting them to &FD so 1>&2 will redirect STDOUT (1) to STDERR (2) and 2>&1 will redirect STDERR (2) to STDOUT (1), you can combine that with a regular redirection so e.g. `command >foo 2>&1 ` or with a pipe (`command 2>&1 | other`)
And you can actually create more FDs to use in your pipelines[1] though I don't remember ever seeing
You're redirecting stdout to /dev/null and pv is writing to stderr. If you use &> instead, stderr and stdout will both be redirected to /dev/null and you will see no output at all.
While 'strace yes > test2' is just a constant stream of write() calls.
The difference matters if you're benchmarking e.g. some new SSD compared to a tmpfs on a machine with 100+ GB of RAM. It's always better if the tools have less overhead, because the comparison is more meaningful.
Also consider that it can be faster to write to a local network than to disk. I've never done it, but I imagine that the kernel's not going to want to deal with your /dev/zero calls if it's spending all of its time writing to a 10GB switch. I can imagine some very specialized storage servers that could spend most of their time writing from memory buffers to a network switch, or if you're troubleshooting a slowdown in the networking itself.
When I started this comment, I didn't think you were measuring what you thought you were measuring with those straces. 'strace yes > test2' only watches 'yes', not '> test2' (which is handled by the shell). Here's what your command outputs:
How does this possibly work? File descriptor 1 is supposed to be the terminal, not a file! Of course, the magic of file redirection:
open("/tmp/test2", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 # Open the file, get FD 3
fcntl(1, F_DUPFD, 10) = 10 # Copy FD 1 (the terminal, STDOUT) to FD 10 temporarily
close(1) = 0 # Close original FD 1
fcntl(10, F_SETFD, FD_CLOEXEC) = 0 # Close FD 10 (the terminal, copy of STDOUT) when exec() is called
dup2(3, 1) = 1 # Copy FD 3 (the file) to FD 1 (STDOUT)
close(3) = 0 # Close FD 3 (the file's original descrptor)
I had the impression that you thought "yes" pipes the output to shell, and then the shell writes to disk. That's incorrect. " > file" means redirecting to a file, and therefore all write system calls actually write to disk.
That's exactly what I thought. Now, had you asked me how redirection worked I would have said "the redirection operators cause the shell to attach a file to the descriptor," but I'd never actually thought through the implications of that in terms of what syscalls get made, and the output of strace presented a rather visceral demonstration of the implications of this clever bit of design.
The ability to get 10 GB/s dummy date into a pipe from the command line could come in handy at some point, for stress testing or something. I am not sure if it is over-optimized. (And even with the optimizations the risk of bugs should be very small.)
which is a common way of doing it (for any command with any inputs and outputs, not just the above ones), i.e.:
command < input_source > output_dest
All three pv command invocation variants, the one here and the two above, work. And it becomes more clear why they are the same, when you know that the redirections are done by the shell + kernel, and the command (such as pv) does not even know about it. So in all three cases, it does not see the redirections, because they are already done by the time the command starts running (with its stdin and stdout redirected to / from those respective sources). And that is precisely why the command works the same whether it is reading from the keyboard or a file or pipe, and whether it is writing to the screen or a file or pipe.
/dev/zero doesn't "write" anything in the sense that yes writes, since it's a character device and not a program. The Linux kernel's implementation of /dev/zero does not write one byte at a time.
You're right of course; and actually I believe the kernel will simply provide as many bytes as the read() requested; so the speed should mostly depend on how you access /dev/zero. IE, the user above was using cat and I think with dd and a proper block size it'd be much faster.
I was under the impression that cat automatically used a sane size for reading. Now that I think of it I cannot think of a source, other that to point out my own anecdotal experience.
When I was writing raspbian images to SD cards to use on a raspabery Cat and DD took within a few seconds of each other on an operation longer then a minute. Since then I have been using cat where I could, but I didn't think to right down the numbers though.
Note that cat+gnu awk was faster than just gnu awk - but mawk was faster still (reading a not entirely small file).
And in a similar vein of gp comparing Gnu and Openbsd, note that openbsd cat is a little more convoluted than the simplest possible implementation (at least to my eyes):
yes will give repeated data though, not just zeros, seems like it's more useful here - as others have pointed out, also uses less syscalls than /dev/zero.
If you want to maximize readability and simplicity then writing `yes` in C is a bad choice. It is much easier, cleaner and shorter to write it in python or just use the shell which would normally be using the `yes` in the first place. Since `yes` is used in shells and builtins can be considered very reliable, here is a implementation as a shell function:
yes(){ while :; do echo "${1:-y}"; done; }
Python:
import sys
while True:
if len(sys.argv) > 1:
print(sys.argv[1])
else:
print("y")
And if you don't need the features of `yes` and only need a loop that prints 'y', then there really is hard to beat the simplicity of:
The C example has three includes, two conditionals, two loops, and one function definition. The python example has a single include, conditional, and loop.
For readability purpose it is easier to go through each lines of the python program than the OpenBSD C code. Its not massively different, but its distinguishable enough that I would choose the python version if I wanted to maximize readability, minimize syntax requirement and did not want to use shell script.
The Shell function is in my view the superior choice if the audience is a programmer than know the shell script syntax. It is just a single loop and is written in the environment that the program is intended to be used in. The only drawback there is the speed.
Most of what makes the C program bigger comes from the fact that the C program does more. Your python example doesn't call pledge(). Remove that from the C program and it drops to one include, one conditional, and two loops. Further, counting the two loops against C doesn't make any sense: it's entirely up to the programmer whether to have a conditional containing two loops, or a loop containing a conditional. Both languages could naturally do it either way.
Exactly. That's the reason I write stuff in C instead of my favorite interpreted language, Ruby. When you write something in C, that's it. No large interpreter plus runtime needed.
You've been on a tear of uncivil and unsubstantive commenting, and it has to stop. Often a good strategy for this is to slow down. High-quality reflective posts are what we're after instead of dismissive, snarky reflexive ones, and the former come more slowly.
If the speed of yes is bounded by memory speed, doing anything useful will almost certainly consume that data at a far lower rate. Putting it on a disk, pushing it over a network, etc. will almost always be slower than yes is able to generate data.
The typical use case is piping it into another running program. Maybe someone wants to do that really quickly rather than putting it on disk or pushing it over a network.
It's not about running 'yes' at 10GB/s, it's about less overhead to do a simple job. If this version of yes is 100x faster, that implies it using 1% of a cpu to do the same work that would otherwise occupy 100% of a cpu. This leaves more of the machine to do what is likely to be the intended task.
Unix userland tools have evolved over decades to be as efficient as possible because they have historically underpinned everything the operating system does. The faster each tool works, the faster the other tools that depend on them work. If increased efficiency results in a bug, that bug can then be fixed, making it a net gain for system stability.
Easy! Write a program that does a brute force check of Goldbach's conjecture on all integers. For each integer that passes the check, print a line. If you can prove this correct (or incorrect) you'll probably win a Fields medal.
I agree with the general point that you could prove such a simple program correct relatively easily but that does still have a cost, which is always a concern in an open-source project. You still need someone to step up and do that work and continue to verify it periodically in the future – if that code is doing complicated things with buffering, that opens up possible odd bugs due to stdlib, gcc, maybe even kernel behaviour changes which might not affect simpler programs.
Not a huge bit of work to be sure but for a non-commercial project you might have trouble finding a volunteer who cares about that tedium.
Absolutely, I once trivially proved a 1000,000 line implementation of return 0 to be correct. I don't know why all comments are bothered by how much overkill this yes implementation is. Maybe they don't hold 100,000 PhDs like we do, am I right?
It's not really optimized to the limit – or perhaps it is, but then the limit is fairly easy to reach.
When I saw this item here I reached for the terminal and wrote a simple (and non-compliant) alternative that simply initialises 8150 bytes of "y\n" and then loops a write forever. I understand that it is not a fully standard-compliant yes, and that maybe GNU yes is indeed fast, but that awfully simple program that takes all of 10 lines (everything included) and took me all of a minute to write performs just as well as far as pv is concerned.
(I eventually completed a feature complete yes but I still think that simply not using `puts` is hardly optimising to the limit.)
A nitpick, but I notice that the BSD implementation do not catch errors when printing. In theory it could get EINTR and only write partial amounts of argv[1], especially if the argument string is really long and the program runs for a extended amount of time. The GNU version do catch EINTR.
Naturally it could be that the C function puts used by most people in OpenBSD is implemented with built in EINTR catching loop, or that OpenBSD do not interrupts writes.
You'd have to check the various POSIX standards to be sure - and have to then verify that the OS/libc actually follows them, but you can pretty much rely on every libc's I/O wrapper functions to handle interrupted system calls or incomplete writes. I've never seen any code check the return value of a printf() to verify that all the characters were printed.
As you say, the GNU versions definitely handle EINTR - the linux man page for puts() just says it returns a non-negative value on success, it's not even specified whether it returns the number of bytes written or not.
`puts` is an "stdio" function, not a system call. It won't EINTR, it correctly resumes if the underlying write() EINTRs. If it does get an error, it will return EOF, then you'll have to call `ferror()` to find out which error.
But the point stands; there's no error handling in the OpenBSD version. But that could be considered a design decision; the OpenBSD version never gives up on writing data until you kill it; the GNU version bails at the first error.
I think the GNU version is surprisingly readable for what it does. I have seen code that is a lot less readable than this and doesn't have the performance benefits described in the article.
I generally look at some BSD sources when I want to know how some unix tools are working, they are always much much more readable than the GNU equivalent.
It's to make sure the buffer is flushed. Streams are closed, but not explicitly flushed at exit. Stdout only because it's the only filehandle with a buffer in yes.
In normal use, GNU `yes` does unbuffered IO on stdout. However, it does use buffered IO for --help and --version messages; it sets atexit(close_stdout) to cover both of those cases at once, rather than handling them both separately.
According to the article/experiment you don't have to bork up the code much, just copy your stuff into a big buffer before you start printing it (and print using write(2) instead of puts)
This works easily for the default case, which prints "y\n" (two bytes), which is likely to divide BUFSIZ. To handle the general case with the same efficiency, you have to have a buffer size that is a multiple of both BUFSIZ and the length of the string to be printed. It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been).
> It appears that GNU yes will not do that and simply does unaligned writes in this case, which is likely to be considerably slower (possibly slower than write would have been).
Why would it be slower to do a single, say, 8190 bytes write instead of 2730 3-byte writes?
And it's also a source of many bugs (in the past and likely in the future as well). In most use cases today I'd rather have a slightly slower userland which is easily read (and audited) then one which compromises quality for speed in edge cases.
This work certainly has value but it is frustrating to keep up with everything being changed. And if changes affect me do the fashions and attitudes of those making the changes have synchronicity with the way that I use computers?
In the old days I think we would have left yes in c because the compiler will build it on any platform.
My first experience with Linux is porting land.c to one of the commercial Unix. Now I first met TCP/IP on a 3B2 and later met systems sold by SMI and DEC and SCO and we all mostly constructed packets the same way and a guy called Stevens had written some nice books about this that everyone had. I think I recall the commercial and free BSD also did things the usual way. But whoever figured out this interesting phenomenon land.c demonstrated happened to be a Linux user and this platform had some different ideas about it. I saw rewriting the relevant parts as an annoying, menial task.
The /r/programming discussion of this is interesting [1].
Someone does a Go version and gets the same speed as GNU yes. Someone else tries several languages. This person got the same speed in luajit, and faster in m4 and php. Ruby and perl about 10% slower, python2 about 10% slower still, and python3 about half that. The code is given for all of these, and subsequent comments improved python3 about 50% from his results, but still not up to python2.
I had to smile, as this thread is a microcosm of programming language stereotypes: with the python programmers tweaking code to get that extra 10% (nobody mentioned pypy..), someone trying to get javascript to work in a long running program, of course a rust implementation that isn't working well just yet (but we're all rooting for it.. set that compiler flag and rerun).. One comment about perl (turns into a one liner..). Nobody bothering to redo the Lua, Ruby and PHP. And for fun somebody throws down some fortran.
Cow means "clone on write". It's a generic wrapper for types that have both a borrowed version and an owned version. For example, `Cow<'a, str>` can hold either an `&'a str` (a borrowed string) or a `String` (an owned string), and a `Cow<'a, [u8]>` can hold a `&'a [u8]` (a slice of bytes) or a `Vec<u8>` (an owned vector of bytes).
In that Rust program, Cow is being used here so if the user provides an argument it ends up with an owned vector that contains that argument plus a newline, otherwise it ends up with a borrowed byte slice that contains "y\n". That said, there's not really a whole lot of point to using Cow here since allocation is a one-time cost and nobody cares if yes starts up a few microseconds slower, but the usage of Cow is limited to just a couple of lines so it's not really complicating anything (the actual write() call just ends up taking a &[u8] byte slice, so the usage of Cow is strictly limited to main()).
Oops, typo. I've updated my comment accordingly, thanks.
Also, I didn't realize it was Clone-on-write. Interesting, and the documentation does confirm this. I say "interesting" because the actual operation involved is `to_owned()`, not `clone()`, seeing as how `clone()` on a `&str` just gives you the same `&str` back.
One night for fun, I wrote a fizzbuzz (without actual printing) in almost all of the languages mentionned here (except fortran), and benchmarked it with 10000 executions each.
10k was just sufficient to get actual differences. Of course I went crazy and did something like 10M, but waiting 10 minutes to get the same relative differences didn't bring anything.
Remember this is without any printing: C is fast, if I remember correctly it was something like 0.015 or 0.03s while at the other end of the spectrum I had to wait 0.1s for JavaScript. C results varied from simple to double because of the OS I guess because it was so quick.
It isn't a really significant benchmark though, for any language. It was one of those nights. Ah, and I remember a large difference between go run and go install.
Actually the second python version is about on par, however it's running on a slower system. He got 6.78GiB/s from GNU yes and 6.76GiB/s with python3. Some one later compares it with python2 which in fact is slower with this codebase.
m4 is orders of magnitude slower (notice the different unit).
If you write the Python script using bytes (not Unicode), then Python 3 is faster than Python 2, at least for me. Python 3 is 20 % slower than GNU yes and Python 2 35 % slower than GNU yes.
e: I think in your last sentence you are comparing results from different computers.
I was under the impression Python was magnitudes slower than compiled native languages, even when you aren't abusing Pythonic features that are computationally expensive like list comprehensions. And this isn't even in Pypy where this loop would be JIT'd and then run natively anyway, unless cpython does some kind of JIT? Is is just that the buffer is so large that the kernel handler is taking so much of a % of the runtime that the inefficiencies of the language stop mattering?
If anyone, like me, is wondering what "yes" is used for. You can use to pipe "y" to commands that require interactivity, so if you just want to say "y" to all the inputs, you can use "yes" to do this:
Yes, but that's the only error it suppresses, and that's because "not existing" isn't really an error for a tool that's trying to delete something. In my example foo/bar existed but I didn't have permissions to delete it, and as you'll see it printed out that error.
Back when I worked at the Genius Bar at Apple Stores I saw a customer come in and talk to a 'Genius' about their MacBook being "slow". After a quick bit of troubleshooting, he just opened up 4 terminal windows an ran yes in all of them, and did some hand wavy explanation about diagnostics.
> In 2006, the yes command received publicity for being a means to test whether or not a user's MacBook is affected by the Intermittent Shutdown Syndrome. By running the yes command twice via Terminal under Mac OS X, users were able to max out their computer's CPU, and thus see if the failure was heat related.
For reference, I had that exact same issue on my MBA - went on for years. Many times while watching youtube or doing something Garageband related.
At the time I had 10.11 on it, but that OS had itself been upgraded from 10.10.
When 10.12 came out, I decided to install fresh - so I backed everything up and installed onto a new SSD drive. The issue has now gone away completely....
Hm, my old Dellbuntu laptop does the same. It's been dropped a few times in its 10 years of service, and now sometimes will shut down if I tap it too brusquely, regardless of the temperature. The BIOS will report it overheated, and i don't know what's going on.
the limit isn't the processor, it's how fast memory is. With DDR3-1600, it should be 11.97 GiB/s (12.8 GB/s)
I don't understand this reasoning. Why is it being limited to main memory speed? Surely the yes program, the fragments of the OS being used, and the program reading the data, all fit within the L2 cache?
It is NOT limited to the external RAM speed and the best proof is that it actually uses over 40GB/s of memory bandwidth.
For each byte of data passing through pv:
1. the byte is read from yes memory to CPU registers by the write syscall handler in the kernel
2. the byte is written to kernel's internal buffer associated with the pipe
3. the byte is read back in the read syscall called by pv
4. the byte is written to a buffer in pv memory
5. and thas's the end because write syscall executed by pv on /dev/null very likely doesn't actually bother reading the submitted buffer at all
edit: Actually it might only be 20GB/s because on Linux pv seems to use the splice syscall to transfer data from stdin to stdout without copying to userspace.
This is also the reason why further "optimization" of this program in assembly was a fool's errand: the bulk of CPU load is in the kernel.
All good valid points, but I'm still a bit surprised that the limit is not higher, I thought that the L2 cache was over an order of magnitude faster than main memory (plus, as someone pointed out in the reddit thread, the peak memory performance should really be double the quoted 12GB/s due to dual channel memory).
The actual throughput then, once you include OS copying, is either 2 or 4 times the quoted speed (depending on splice usage), so we're either at main memory theoretical speeds, or double main memory speeds. Intuitively, I'd still have expected that it should be a larger multiple.
(A quick search can't find me any reliable Intel L1/L2 cache speeds/multiples to quote, so I admit this comment is more speculation than it should be!)
L2 cache and copying "y" bytes have very little to do with this; I suspect if you could produce high-granularity timings it would almost all be in the syscall overhead.
A read() syscall takes longer than a getpid() syscall because read() has more work to do, it actually does a data copy of len bytes, which takes some time (and will be faster/slower if data is cache hot)
What we call the "syscall overhead" is what happens before and after the actual data copy, switching between user and kernel mode.
You make that overhead negligible by calling read() with a large size.
(Many, many years ago I was working on the Zeus web server, and we went to surprising lengths to avoid syscalls for performance.)
Snap!
IIRC, ZWS used shared memory to mirror the results of the time() syscall across processes, to save a few nanoseconds on some operating systems :) That was before Linux and other OSs used techniques like the vsyscall/VDSO mentioned in the stackoverflow discussion...
Very rough numbers - the 9.13/9.33 difference flip-flopped when I ran the commands again. Binding both processes to the same core is definitely a performance hit though. There might be some gain through a shared cache, but it's lost more through lack of parallelism.
I tried 2/4 as not sure how 'real' cores vs 'hyperthread' cores are numbered. These numbers are from a i7-7700k.
Assuming pv uses splice(), there is one only copy in the workload: copy_from_user() from fixed source buffer to some kernel allocated page, then those pages are spliced to /dev/null.
If the pages are not "recycled" (through LRU scheme for allocation), the destination changes every time and the L2 cache is constantly trashed.
I only learned of pv from this article so I can't speak much about its buffering. I would guess that the kernel would try to re-use recent freed pages to minimise cache thrashing. But anyway, on the 'yes' side, the program isn't re-allocating its 8kb buffer after every write(), so there's a lot of data being re-read from the same memory location.
Sure, but this is a pipe between two tiny processes, hopefully with very little else being run on the computer at the time (otherwise all bets are off for any benchmarking). There's no kind of 'real' I/O going on (in terms of stuff like screen, disk, NIC, and so on)
There's no reason that the L2 cache needed to be flushed at any point - the caches are all dealing with physical memory, rather than virtualised address space, so the fact that there are two processes here shouldn't stop the caching from working.
I am probably way behind the current state of the CPU judging by the downvotes I got so if you are saying there is no reason and the data can be written into a device without leaving the CPU I will just concede my ignorance.
Don't fret about the downvotes, these magic internet points aren't redeemable anywhere :)
It's completely possible for the data to not (all) leave the CPU. If the caches are large enough, then the pages full of "y\n" will be still resident in the cache when the next iteration of the program overwrites the same pages again. Then the CPU has no need to send the original page out to main memory.
If you did for(;;) buffer[(i++)%size] = 'y'; then you'd be correct. However, you do i/o. And the 'y's appear at the i/o driver and have to be made visible for the device it's driving, which can be anything, including a process on another CPU core in another socket. If they remained in the issuing CPU cache, I fail to see how the destination device could possibly see them. There are some devices which can snoop the cache (like SoC GPUs and other CPU sockets on some archs) but the snooping is much slower than memory bus. Writing to memory is the only way which a) guarantees the data is available elsewhere and b) is the fastest.
You could make "yes" faster with the tee() syscall. Keep duplicating data from the same fdin (doesn't actually copy) and it becomes entirely zero-copy.
next step is playing tricks with memory mapping so that the buffer is large in virtual memory but only takes a single page in physical memory, fitting in L1 (at least on the read side).
I've realised straight tee() is actually wrong - it works fine for piping to something but "zcyes > /dev/null" won't work, it needs vmsplice() like mrb said.
I was not going to post this because hacker news has this ethic (?) of down voting anything that seen as not positive. Perhaps we should have discussion about that, I'm not sure that's a good thing but I'm not in charge here.
The top comment is:
"It's a shame they didn't finish their kernel, but at least they got yes working at 10GiB/s."
which as an OS guy, someone who has been working on Unix for 30+ years, as a guy who was friends with one the QNX kernel guys (they had perhaps the only widely used microkernel that actually delivered), that's hugely amusing and spot on. The GNU guys never really stepped up to being kernel people. Bitch at me all you want, they didn't get there. It's a funny comment, especially coming from reddit.
> hacker news has this ethic (?) of down voting anything that seen as not positive
We must not be reading the same Hacker News...
Anyway, the comment you're quoting is just a shallow jab that belittles the GNU developers' work without contributing anything new or meaningful. It's telling that you had to spend two paragraphs to justify cross-posting it here.
You say that and then immediately become an example of what hes talking about. This "shallow jab that contributes nothing new or meaningful" is, in some circles, known as a "joke." I'm continually frustrated by people who think that misinterpreting comments as harmful is a useful activity.
Not really -- I didn't downvote that comment. And I still dispute the notion that negativity is rare or always shunned on this board: to the contrary, it's so commonplace that an actual rule [0] had to be added to try to sway things in the other direction.
Jokes have their place, but bringing up the failure of Hurd in every GNU-related post is banal. And saying they "never really stepped up" to your level as a mighty kernel developer, as if the people who brought us glibc and coreutils lack an understanding of OS internals, just seems rude and curiously out of touch.
So negativity here is not rare, when people don't like something they are fast to jump on it.
What I was trying to get at is this, if you care about your upvotes, hacker news promotes a sort of hive mind. Which is somewhat like "say only nice things unless you are clearly swatting down something that is obviously wrong".
Which is mostly fine, fantastic in fact. I'm fine with it, hacker news is really pleasant because of the (more or less ) quality of the posts and especially the quality of the comments. I'd much rather have it be this way than a free for all, those go bad pretty fast.
So I'm for the hive mind, I was just pointing out that you can't make jokes and be upvoted. The joke wasn't banal at all IMO. GNU has done a lot of good, I've been there since the beginning and paid attention along the way. They have also been pretty self serving with their choices, every project has to sign away their rights and then GNU takes full credit for the project even though they had nothing to do with it other than it being GNU troff for example. Given their tendency to take credit for stuff that they didn't do, and their claim that they can do an OS but clearly can't, that joke is funny as heck. If you don't get that, sorry, you haven't been paying attention.
() The quality of the posts in the area of programming, especially systems programming, is spotty. Some stuff is great, stuff I didn't know (there is a lot of that here and I'm very grateful for it, it's why I stick around), some stuff is meh, and then there is stuff like "wow, look at $OBVIOUS, isn't that cool?" that gets upvoted. That last one I just don't get, but whatever, the good stuff is good. The signal/noise ratio here is better than any other programmer oriented site I've found.
Years ago I read a similar experiment about max. CPU data flow. Guy was testing how much data can his CPU pass in a second. He was writing it in C, using some Linux optimization, optimizing code for CPU caches, using some magical C vectors that are optimized for such purpose. He got some help from someone working at Google. I tried to find that post but never succeeded. Does anyone here know it?
This will attempt to open a child shell process with whatever output of "yes" is, interpreted as shell script.
But before that, parent shell has to buffer until EOF. With "yes" output being unbound, this means unbound buffering / memory growth. That is, until OOM killer take notice and shut it down.
Whole thing will probably take few seconds to a minute (depending on how much free RAM there is vs. how fast it is), peg a single CPU core in the process, and will recover cleanly (except for shell in question being terminated).
...unless system in question has single CPU and large+slow swap. Then yes, bring-it-to-its-knees.
Single CPU is fine nowadays. It will be slowish if you ran it with no nice or priority.
RAM can be tweaked by preventing Linux heuristic overcommit via sysctl vm.overcommit_memory=2. (0 is only recommended if you do not run broken applications. It so happens that many JS and Java VMs are broken on memory pressure.)
I'm guessing it tries to read the output and return it. Since yes doesn't actually terminate it's going to generate a massive temporary variable to store it's continuous output.
It's come to my attention that lower numbers are not better here. I have filed a bug, we'll get to the bottom of this shortly. I want to apologize to all our users, this issue does not reflect the values and principals we at Pure JavaScript `yes` hold dear https://github.com/Sequoia/yes/issues/3
That kind of question doesn't make sense for open source code.
Somebody wanted to optimize 'yes', so they did. There doesn't need to be a good reason, just like there doesn't need to be a good reason for a person to read a certain book or watch a certain movie, other than they want to do it.
The question was "do we need it? (for a specific use case)" rather than "should we do it?".
And it does make sense for Open Source code because the resources are limited hence other features and/or bug fixes might be more important than pushing data at full speed
> And it does make sense for Open Source code because the resources are limited hence other features and/or bug fixes might be more important than pushing data at full speed
That's not how open source works, though. There's not a group of people obligated to work on the GNU utilities. There's not a central project manager ordering people around telling them what changes to make.
Declaring that 'yes' is generally fast enough already doesn't imply that it was fast enough for the person who spent their time optimizing it. Somebody needed (or just wanted) 'yes' to be really fast, so they did the work and submitted the changes back.
> If one just makes it faster without major benefits that patch is likely to be rejected
IF the project were overwhelmed with incoming patches, then I would agree that there's probably higher priority changes than optimizing 'yes'.
But I'm almost certain the GNU project is not being overwhelmed with changes to the core utilities. Everything else being equal (code quality, readability, test coverage, etc.) there's really no reason to reject a patch that demonstrably improves the project, even in some silly way like making 'yes' really fast.
This is software engineering. Thinking about the tradeoffs and whether they are appropriate or warranted is bread and butter. It very much makes sense to ask whether all of this is merely optimizing for a benchmark at the expense of other factors that might turn out to be more important, such as long term maintainability, portability, behaviour in some common cases which are significantly unlike the benchmark, and so forth.
Several people have made some of these points here and in other discussions; and not only do they make sense, they are an important part of the discipline.
Is the optimized implementation documented well enough that other people coming to it anew can understand it in another 10 years' time? How many of the magic constants have to be tweaked as the operating system evolves? Is it clear to maintenance programmers how they need to be tweaked? Does optimizing for the benchmark skew the implementation too far away from one common case where one is only wanting a few "y"s (because, say, the program being told "y" only asks 17 questions) resulting in every invocation of the yes program rapidly generating KiBs or even MiBs of output, that lives in pipe buffers in kernel space unused and only to be thrown away? Does it make more sense to put buffer size optimizations in the C library where all can benefit? What is the benefit of optimizing for Linux-only system calls on GNU Hurd? Will we optimize yes for GNU Hurd, too, in the same codebase? How much conditional compilation will we end up with? If the C library is improved to do better in the future, is it a problem that the optimized for benchmark program no longer automatically gains from it? How much of a problem is it that a GNU program is now not portable to systems other than Linux, and is locked into one operating system?
And what about other benchmarks? Many have noted that this benchmark pretty much relies on the fact that the output of the yes program is simply thrown away, and no real work is being done by anything else on the system. What about a benchmark where it is? What about a benchmark where it is important that yes issues a system call per line, yielding the CPU to other processes so that they can process that line before yes bothers to print the next? What about a benchmark that measures kernel memory usage and treats lower as better? What about benchmarks that measure how small the yes program is; not just in terms of code but in terms of memory, I/O, and CPU usages; because the memory, I/O, and CPU should actually be given to more important programs than yes on the system that are actually the system's main function? What about low impact?
Yes, it's fun to focus narrowly and optimize yes so that it generates as much output as it can for one specific toy usage. But in engineering one has to ask and think about so much more.
A detail you seem to be missing is that you are not limited to a shell script. The shell sets up the pipeline, but the members of the pipeline can be written in arbitrary languages and just have each stdout linked to the next processes stdin.
As a result you can process very large volumes of data and consume (not waste, consume) significant system resources to perform your processing.
I would just pre-allocate a static array of "y\n" of size BUFSIZ, write it out in a loop, and call it for the day, skipping the whole malloc and filling loop business.
Make the static array BUFSIZ * 1024 to trim the syscalls by a factor of 1000.
Have a few pre-written arrays for the common cases: "y", "Y", "n", "N", etc. Those are the fast cases (or the benchmark optimized cases, like, what Volkswagen did). Have another pre-allocated static array to fill in with other input.
Let me check that I have correctly understood the scenario you're anticipating.
1. Someone uses "volkswagen" as a verb meaning "cheat in benchmarks".
2. Volkswagen takes them to court, on the basis that it is slanderous or libellous to associate Volkswagen with cheating in benchmarks.
3. Counsel for the defence reminds the court that Volkswagen were in the news for a protracted period for a benchmark-rigging that saw them hit with a multi-billion-dollar fine, slashed a third off their stock price, saw their CEO resign, led to there being a "Volkswagen emissions scandal" page on Wikipedia, etc., etc., etc.
4. [THIS IS THE BIT I'D LIKE YOU TO FILL IN FOR ME.]
By volkswagen arguing that this is industry standard behaviour, and the entire media campaign is already libel and slander.
And they’d have a pretty good case with that, considering the original report all this media campaign was based on called out 6 carmakers, but the media (and the poster) only call out Volkswagen.
No, but you don’t wanna get sued for libel and slander. If VW can realistically show that it is industry-standard behaviour (which is pretty obvious), then they might even have a chance to win.
The VM subsystem is much slower. Also possibly a not-up-to-date GNU grep, on my (ancient) MBP I get 26MB "native", 780MB GNU and 2.8GB for TFA's C program (2.9 if I replace the malloc by a stack-allocated array, weirdly enough)
With that malloc overhead, I expect GNU yes to be slower when only a few bytes are read from it.
So, what's the distribution of #bytes read for runs of 'yes'? If we know that, is GNU 'yes' really faster than the simpler BSD versions?
Also, assuming this exercise still is somewhat worhtwhile, could startup time be decreased by creating a static buffer with a few thousand copies of "y\n"? What effect does that have on the size of the binary? I suspect it wouldn't get up much given that you can lose dynamic linking information (that may mean having to make a direct syscall, too).
Unless you are statically linking, one malloc doesn't significantly affect your startup time. When only a few y's are read, time is going to be dominated by ld.so, by a large margin.
Measurements are really noisy, but I seem to get significantly better numbers than that when I use fsplice() on a pre-generated few pages of file data instead.
I thought this was a fascinating read but it left a serious question lingering in my mind, which is a little out-of-scope for the article, but I hope someone here can address.
Why did the GNU developers go to such lengths to optimize the yes program? It's a tiny, simple shell utility that is mostly used for allowing developers to lazily "y" there way through confirm prompts thrown out by other shell scripts.
is this a case of optimization "horniness" (for lack of a better word) taken to its most absurd extreme, or is there some use case where making the yes program very fast is actually important?
The stated use case for the perf improvement was "yes(1) may be used to generate repeating patterns of text for test inputs etc., so adjust to be more efficient."
I've personally used it for generating repeating text and filling disks in system testing, so I appreciate it being faster at those tasks. I also sometimes use it as a signal generator for a hacky load generator, like so:
But doesn't this make the typical use case (just a few "yes"s needed) slower, since first it has to fill a buffer?
I would write() the buffer each time it gets enlarged, in order to improve startup speed.
Also: The reddit program has a bug if the size of the buffer is not a multiple of the input text size.
And it's increasing the buffer by incrementing one at a time, instead of copying the buffer to itself, reducing the number of loops needed (at cost of slightly more complicated math).
If your only overhead is filling an 8K buffer, I don't think your user is going to care. Taking one microsecond instead of one nanosecond doesn't matter all that much when you're going to lose way more than that in pipes, the kernel, the program you're piping it to, etc.
Maybe filling a disk or flooding a network connection as in
yes | ssh server "cat > /dev/null"
But yes take arguments so there might be more use cases:
$ man yes
NAME
yes - output a string repeatedly until killed
SYNOPSIS
yes [STRING]...
yes OPTION
DESCRIPTION
Repeatedly output a line with all specified STRING(s), or 'y'.
> But doesn't this make the typical use case (just a few "yes"s needed) slower, since first it has to fill a buffer?
One could put a write in the memcpy loop, so that first it writes one copy of the string, then two, then 4, 8 (etc.), meaning time to first byte is short, but it quickly gets up to the eventual asymptotic speed with a full buffer.
https://github.com/coreutils/coreutils/blob/master/src/yes.c
https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c