Demystifying the Go Scheduler

Demystifying the Go Scheduler

1. 1. DEMYSTIFYING THE GO SCHEDULER MATTHEW DALE
2. 2. Go Concurrency Crash Course The atomic unit of concurrent work in Go is the goroutine Started by invoking a function with the go keyword ! Invoked function runs concurrently with the current thread of execution Goroutine synchronization is accomplished with channels Message passing construct Buffered or unbuffered
3. 3. The Go Scheduler Attempts to efficiently schedule goroutines on available resources The scheduler will pick up a new goroutine when the current goroutine… 1. Finishes 2. Makes a blocking system call E.g. reading a file 3. Makes a blocking Go runtime call E.g. reading from a channel 4. Invokes another function (only happens sometimes)* Taken from http://morsmachine.dk/go-scheduler M - Machine (OS thread) P - Context (Go scheduler) *As of Go 1.2 ( https://golang.org/doc/go1.2#preemption ) G - Goroutine
4. 4. If the current goroutine is blocked on a system call… The OS thread processing the goroutine idles waiting for the system call to return The context that was scheduling gorotuines on the blocked thread is moved to a new thread (or a new one is created if none are available) Taken from http://morsmachine.dk/go-scheduler
5. 5. HOW DOES GO ASK FOR RESOURCES ABSTRACTED BY THE OPERATING SYSTEM?
6. 6. The syscall Package All calls that can block an OS thread go through the syscall package (except cgo calls) Responsible for informing the Go runtime that a potentially blocking system call is about to happen
7. 7. src/pkg/os/file.go, line 91 src/pkg/os/file_unix.go, line 186 Let’s follow a call to file.Read…
8. 8. src/package/zsyscall_linux_amd64.go, line 831 (generated) src/pkg/syscall/asm_linux_amd64.s, line 44
9. 9. src/pkg/runtime/proc.c, line 1502
10. 10. Notes on Host Setup Modify /etc/security/limits.conf Go will quickly use up the default allotment of file descriptors on most Linux distros The default Go HTTP server will panic if listener.Accept() ever returns an error, killing your service Example configuration ec2-user soft nofile 100000 ec2-user hard nofile 100000 root soft nofile 100000 root hard nofile 100000
11. 11. FILE I/O DEMO!
12. 12. BUT HOW DOES THAT WORK WITH NETWORK I/O? ! WOULDN’T THAT JUST CREATE A BILLION THREADS?
13. 13. The netpoller Runs in its own thread and polls the OS’s asynchronous I/O network interface for data Goroutines asking for network I/O block waiting for the netpoller to give them data, not for a system event Go treats Unix sockets and network connection file descriptors the same, so Unix socket I/O will not block OS threads See http://morsmachine.dk/netpoller for a great, concise description of the netpoller
14. 14. RESOURCE STARVATION OR WHY DID MY GO SERVICE STOP RESPONDING?
15. 15. WEB SERVICE PERFORMANCE DEMO!
16. 16. NOT SO GOOD…
17. 17. Which Problems Are Essential vs Accidental*? Essential A computer can do a finite amount of work per unit time If there is more work than resources, then work must be delegated, queued or dropped Accidental Sometimes you and the Go scheduler disagree about what is the most important work to do *Essential vs accidental complexity is a problem decomposition strategy proposed in Frederick Brooks Jr.’s book The Mythical Man-Month
18. 18. Simplify vs Complexify Simplify - synchronous instead of concurrent Still requires determining how to do CPU load balancing Pushes performance problems to the front, letting the caller tune on their side Complexify - refactor into micro services and add queues Lots of additional code just to solve a “simple” problem
19. 19. Think bite-sized Refactor long-running, non-blocking workloads to add more function calls The Go scheduler will sometimes preempt the running goroutine when it makes a non-inlineable function call Explicitly call runtime.Gosched Causes the current running goroutine to yield to the scheduler
20. 20. Think micro services Keep heterogeneous workloads in different Go processes One service handles quick or I/O heavy tasks One service handles long-running OS handles resource scheduling
21. 21. runtime.LockOSThread Locks the current goroutine to the current thread and doesn’t allow any other goroutines to run on that thread Call runtime.UnlockOSThread to unlock the thread.
22. 22. Extra Go Scheduler Topics There are a lot of topics relevant to the Go scheduler but beyond the scope of this presentation, including… Tailoring GOMAXPROCS for your workload How cgo calls interact with the scheduler
23. 23. Sources and Suggested Reading http://morsmachine.dk/go-scheduler http://morsmachine.dk/netpoller http://dave.cheney.net/2014/06/07/five-things-that-make- go-fast http://golang.org/pkg/runtime/ Code examples used in this presentation can be found at https://github.com/matthewdale/ GoSchedulerDemo
24. 24. QUESTIONS?