Example: https://gitlab.com/internet-cleanup-foundation/web-security-...

Running the 'make' command in this repo will setup all requirements and then run code checks and tests. Running it again will skip the setup part unless one of the dependency files has changed (or the setup env is removed).

The big downside of Make, alas, is Windows compatibility.

GNU Make works fine on Windows. The sources come with a vcproj to build it natively, or you get it from ezwinports. At my dayjob, we have a pretty complicated build with GNU Make for cross-compiling our application to Arm and PowerPC, and it works on Windows, even with special Guile scripts to reduce the number of shell calls which are extremely slow on Windows.

VBScript works better on Windows, IMO. Also works out of the box on all Windows versions since at least 2000 (on Win9x it was shipped with IE).

Not really... not since XP, anyway. Unless you have a space in your username (which is a terrible idea for many other reasons), your "Documents" path is C:\Users\JohnSmith\Documents. "Program Files" is pretty much the only important path which is likely to have spaces, and your makefiles (hopefully!) don't need to touch that.

For a software to work fine on Windows, it must support spaces in files and paths. Also Unicode in files and paths.

VBScript does, GNU Make doesn't.

> which is a terrible idea for many other reasons

If you use make to setup stuff, it's very possible you'll need to access "c:\Users\All Users" which does contain space in username. Also "c:\Program Files (x86)\Common Files" which contain more than one.

DOCUME~1 Documents

or

<SYMLINKD> ALLUSE~1 All Users [C:\ProgramData]

That is entirely correct and really the most glaring downside of Make. In my opinion: If you have spaces in your dependency names, just stay away from Make as far as possible.

I’ve just installed Visual C++ 6.0 Professional on a WinXP VmWare machine. Took less than a minute, BTW — modern SSDs are awesome. The default installation path under program files also contain spaces, it’s "C:\Program Files\Microsoft Visual Studio\VC98"

BTW, they have a bug even on the very first welcome screen: http://const.me/tmp/vc6.png

>ezwinports

Interesting, hadn't run into this before. What's the advantage over MSYS2?

Can you break down your problem into a bunch of rules of the form “To produce this file, I need to run these shell commands, which read those other files over there as input”? If so, Make can take care of figuring out which steps actually need to be run.

You'd have to give me a _very_ compelling reason to support developers who use Windows, when Windows lacks these essential tools. Besides, don't people who develop on Windows live in WSL?

(I also write code to run on Linux, but still prefer Gradle.)

I’m just getting into Kotlin and gradle isn’t something I’ve used before since I’m mostly web, .net til now.

I can barely understand why you'd want to develop on Windows (ok, for non-Windows-only products) with it, but without it...

I've been actively using WSL for over a year along with Docker and set up the Docker CLI in WSL to talk to the Docker for Windows daemon.

Performance in that scenario is no different than running the Docker CLI in PowerShell, or do you just mean I/O performance in general in WSL? In which case once you turn off Windows defender it's very usable. WSL v2 will also apparently make I/O performance 2-20x faster depending on what you're doing.

WSL adds a lot if you're using Docker IMO. Suddenly if you want, you can run tmux and terminal Vim along with ranger while your apps run nice and efficiently in Docker. Before you know it, you're spending almost all of your time on the command line but can still reach into the Windows cookie jar for gaming and other GUI apps that aren't available on Linux and can't be run in a Windows VM.

It's acceptable for relatively infrequent file access, but will eat you alive if you're doing anything that involves lots of random file access, or batch processing of large sets of small files, or stuff like that.

That's dealing with 10k+ line projects spread across dozens of files quite often, and even transforming ~100 small JS / SCSS files through Webpack. It's all really fast even on 5 year old hardware (my source code isn't even on an SSD either).

Fast as in, Webpack CSS recompiles often take 250ms to about 1.5 second depending on how big the project is and all of the web framework code is close to instant to reload on change. Hundreds of Phoenix controller tests run in 3 seconds, etc..

https://insights.stackoverflow.com/survey/2019#technology-_-...

Windows 47.5%

MacOS 26.8%

Linux-based 25.6%

BSD 0.1%

87,851 responses

(The Stack Overflow survey is a poor representation of the entire development community, but it's worth something, maybe the best we have.)

VS2019 supports Clang and cmake now.

As opposed to what exactly? Any other alternative, e.g. separate shell scripts, "npm run" scripts in package.json, running a Docker image, hell even cmake or other make-like tools - does stuff you don't know about without reading the files either.

I tend to write my Makefiles to create as much of a local dev environment as possible for every project. Using Python virtualenv/Pipenv/Poetry, Ruby vendored dirs, custom Gopath per project (using direnv), etc. Most tools support some sort of isolation/localisation, but it's often just not on by default.

I have seriously considered hiring someone to audit and prune all the random little libraries and tools I've installed over the years for that one-off time I had to process a weird file format or wanted to try something from HN.

https://godarch.com/

Every boot is a fresh install. Any one-off only becomes persisted unless I add it to my recipes.

https://github.com/pauldotknopf/darch-recipes

It does sound like the right idea. This is hypocritical as someone who doesn't use it but I hope more people use it.

At that point, I use a scripting -not shell- language (which is not as implicit)

I don't program in c anymore, so my major workflow is all in one language; I write my server in the same language that do s the compilation, which is in the same language that does utilities like creating network tunnels to my lab nodes

That said, I wholeheartedly agree with the comment.

It's sad that something so central to a project and so useful and important to so many people seems like it hasn't advanced ... ever.

Developers generally do the minimum with Makefiles and get out. They are similar to 1040 forms in popularity.

I've always had a dream of a redesigned "make" system... with import statements, object oriented rules, clear targets, rules and files clearly seperated, structure and organization... sigh.

As I'm not the most intelligent developer, I'm sure there's a better, more sophisticated way to do this but I got really frustrated and gave up.

That said, mileage may vary. It was originally built by Google so it has quirks. I find it is best suited for projects with compiled dependencies and large repos.

Otherwise, I was going to add that Gradle as a build system is very advanced and improves upon make in many ways.

It is easy to add command line arguments to the tasks, configure them using files (json, yaml), environment variables, to split the task definitions into several modules/namespaces.

There's always some level of bootstrapping a project (installing packages/software, compiling libraries and dependencies) where it's easier to just to write a shell script than to program python to do. E.g. How do you get fabric installed on a system?

There's also been this longevity of sorts that Make seems to have gotten right. People just keep going back to it because it's simple.

I used bash scripts for years, but for a lot of reasons made the switch:

- It was always painful to create small libraries of functions used across multiple scripts in a project

- It's difficult to consistently configure them with per-user settings. I've written bash implementations of settings management, Invoke handles this for me.

- I'd still have to reach for Python whenever I needed to do anything with arrays or dicts, etc.

- Getting error handling correct can be a chore

Invoke has a lot of nice to haves to:

- Autogenerated help output

- Autocomplete out of the box

- Very easy to add tasks, just a Python function

- Easy to run shell code when needed

- Very powerful config management when needed

- Supports namespacing, task dependencies, running multiple tasks at once and deduplicating them

It's not perfect, but it's a lot better than my hand rolled scripts were.

GNU Make even comes with a vcproj file for building a native binary with Visual Studio. Worked fine for me. Building it with Guile support though is difficult, but fortunately Eli Zaretskii provides native binaries through his ezwinports, and they worked pretty much flawlessly for me. Of course you will usually need a shell to execute recipes, but Make itself runs natively. For more information, see README.W32 in the sources.

If you just want a Make, there is [2] which can be installed separately and is part of the GNUWin32 collection.

[1] https://www.msys2.org/ [2] http://gnuwin32.sourceforge.net/packages/make.htm

I had the opposite problem of wanting to develop some stuff for Windows from a Linux environment, and I settled on running a linux VM and copying binaries over by scping to WSL, which works reasonably well.

It is also a nice document of what you can build and how.

I also like it because Netlify supports it, so you can get it to run make to deploy your site when you push a commit, giving you a lot of control about your CI, while keeping it simple.

But the Make manual is pretty comprehensive as a guide and reference: https://www.gnu.org/software/make/manual/make.html

Also (as with most things) knowing what name some concept has makes it easy to search for references. For example the terminology of rules (target, prerequisite, recipe): https://www.gnu.org/software/make/manual/make.html#Rule-Synt...

Things I tend to google often because I forget and some are used more often than others are: automatic variables, implicit rules, conditionals and functions.

One trick that really helps making Make complete is making your own pseudo state files and understanding the dependency system. One of the best features of Make is its dependency resolving. You generally write rules because you want a target (a file or directory) to be created, based on prerequisites (dependencies) according to a recipe (shell scripts). Make figures out that if the prerequisites didn't change, it doesn't need to run the recipe again and it will reuse the target. Greatly saving on build time.

Because Make relies on file timestamps to do its dependency resolving magic if you don't have a file there is not much Make can do. So what you can do instead is create a pseudo target output yourself. For example: https://github.com/aequitas/macos-menubar-wireguard/blob/mas... Here a linter check is run which creates no output. So instead a hidden file .check is created as target. Whenever the sources change the target is invalidated and Make will run this recipe again updating the timestamp of .check. Also note the prerequisites behind the pipe (order-only prerequisites). These don't count toward the timestamp checking, but only need to be there. Ideal for environment dependencies, like in this case the presence of the swiftlint executable.

http://matt.might.net/articles/intro-to-make/

For learning advanced techniques: "The GNU Make Book" by John Graham-Cumming.

https://www.youtube.com/watch?v=fkEz_oVh0B4

https://nostarch.com/gnumake

All in all tho, it's a fantastic place to write down long CLI commands (ex: launching a dev docker container with the right networking and volume configurations) that you use a lot when working on the project.

Our Jenkins pipeline also relies on the Makefiles, literally just invoking `make release`, which is also pretty awesome.

Make does support includes (https://www.gnu.org/software/make/manual/html_node/Include.h...) which allow for some form of code reuse across projects. But then you encounter the balance between DRY and clarity. There are always exceptions, so you try to make stuff to universal, but then its hard to grok the code. And I feel that if I start to use functions I'm using Make wrong and that kind of logic better fits in shell scripts called from the Makefile. Makefiles (the way I use them at least) should be simple to read and explain themselves. But it's often hard to balance this with the features Make provides, like implicit rules and automatic variables. And if I ever turn to generating Makefiles (other than for C projects where it kind of expected) I will probably retire.

Oh absolutely. It's fantastic for that. Our build pipeline actually relies on that; every project has a "release" target that is basically for the CI to use.

> Make includes

Yeah, I looked into that, and I think I had the same conclusion.

> scripts called from the Makefile

That's what I'm thinking is the way to level up this kind of system. Although then, why have `make init` instead of just `./bin/init` ?

In the `make init` example. It doesn't matter how many intermediate steps are involved `init` is the end-state I want to achieve. So in most of my Makefiles the `init` target will fan-out into requirements as wide and deep as it needs, including running apt to install missing system dependencies. But then the good part. If a dependency is already fulfilled Make won't have to run it again. Although sometimes its hard or clunky to convert some dependencies into 'files' so Make can do its dependency resolving work properly.

https://github.com/ruby/rake

If it sounds interesting, check out https://www.npmjs.com/package/shelljs

PS: I do my primary development on windows, but my production environment is ubuntu. node apps "just work" on both environments. truely cross platform.

But of course I'm immediately skeptical of this idea a la https://xkcd.com/927/ (Standards). For instance, maybe this is what npm and all the rest thought they were doing. Certainly Rake in the ruby world tried to do this, and I never really liked it, so clearly they missed the mark somehow, at least for me. But then when I feel discouraged about the ability to improve on things, I think about how I felt this way when I first heard about Git. Why would you implement a new source control system when we already have subversion? Sure, svn has its frustrations and warts, but this new thing is just gonna have its own frustrations and warts and now we'll just have another frustrating warty thing and we haven't really gained anything. And this is totally true! Git is super frustrating and warty. Except that it's also way better than subversion, much faster and far more flexible. It was a revelation when I started using it. So I think back to Linus when he was thinking about creating git and think that he probably didn't have this discouraged uncertainty about improving things; he just had ideas for a better way and he went out and did it. (And yes, I know it was influenced by bitkeeper and other DVCs exist, so it's not like he invented the concept, but my point stands.)

So maybe someone could make a better Make?

E.g. these four pages are the university of Cambridge masters in computer science:

https://www.cl.cam.ac.uk/teaching/1819/part1a-75.html

https://www.cl.cam.ac.uk/teaching/1819/part1b-75.html

https://www.cl.cam.ac.uk/teaching/1819/part2-75.html

https://www.cl.cam.ac.uk/teaching/1819/part3.html

(Or a MOOC, but the links above are easy to browse text, syllabuses and lecture notes, not a load of videos.)

Numerous times I'd be sitting in a class and we'd go over a solution to some theoretical problem, and I'd realize that this solved a problem that had taken me days to discover on my own (and this solution was usually better than what I'd come up with).

If you are the kind of person who can work through everything on your own (including what may seem like the the boring parts), I highly recommend doing so.

I've thought about going back for my CS degree a lot but can't really justify the cost and time investment vs self teaching. But it's something that's always been in the back of my mind.

(Sadly though this exposure was not in the context of a theoretical course on data structures, but rather in the context of reading the docs for HashMap as my university dropped older courses and languages to jump on the bandwagon of becoming a "Java school".)

Another example: the automata class I took went over pushdown automata, and I immediately saw that it would solve the issues I'd been having with a finite state machine I was using to handle input for a game.

Oh and recently I needed to put different sections of a screen on different layers so that no 2 adjacent sections were on the same layer. I realized that this was basically just graph coloring, so I was able to find a solution in minutes instead of hours.

I'm sure their are people who can get through most of a CS curriculum on their own, but I'm not that disciplined. I've also never met anyone who was. It has been immensely helpful.

The Cambridge course isn't perfect, but they do a very good job of making as much teaching material as possible publicly available.

https://www.cl.cam.ac.uk/teaching/material.html

Anyway, either of them will let you mirror a website so you never have to worry about it going down.

2. SQL / relational database schemas. Persistence opens up a lot of capabilities. And databases themselves are very well-optimized; if you do any nontrivial data manipulation it's likely that whatever the query planner comes up with will be faster than your first idea of how to do it by hand.

3. Graph searches. An awful lot of problems can be solved by knowing how to turn problem into a graph search. Make sure not to fall into the trap of thinking a graph search is limited to paths through space - you can solve problems like "get through this dungeon with keys and doors" by adding duplicate nodes for the different states.

4. Sequential Bayesian Filters. Are almost as useful as graphs, but aren't in a standard CS curriculum so you'll look like a wizard. These solve the problem of "I want to know a thing and I know how it changes over time, but I only can get rough estimates for its current state." Kalman Filters are simple and give great results when applicable. Particle Filters have lower quality but are applicable to more problems and dirt simple to code.

I loved this book: https://users.aalto.fi/~ssarkka/pub/cup_book_online_20131111...

and also Thomas Schoen group does great work on Sequential Monte Carlo (SMC), MCMC for sequential data :) http://user.it.uu.se/~thosc112/index.html

They are also building a probabilistic programming language for sequential data! https://github.com/lawmurray/Birch

I don't like the EKF much and prefer the UKF. The core filtering code is a little more complex but they're much easier to actually work with; you can give them arbitrary functions like a particle filter.

Particle filters have the advantage of being able to handle arbitrarily wacky distributions. But they are random and do some wacky things in edge cases. They'll behave much more poorly in low-evidence situations than other filters will. And they fall over spectacularly if you switch from low-evidence to high-evidence (there's a workaround for this but it's still counterintuitive). Finally they're just more computationally expensive than the others.

Birch sounds interesting, I'll take a look.

Horror story re SQL: in my SaaS I skipped sql and went with cloud datastore (NoSql) and regret it. basically (to simplify) you can't query your data without doing a full table scan (IE Slow).

Why? Any project of sufficient complexity is very hard to test. If all you're doing is code -> build -> run to debug your code, you can very easily break something that's not in your immediate attention.

The problem is that good unit testing is hard, and time consuming. It can be so time consuming, that unless you can really plan in advance how you test, you could spend more time writing test code than real application code. (This is what happens when writing professional, industrial-strength code.)

So, when a hobby project becomes sufficiently interesting enough; such that the code will be so complicated that your code -> build -> run loop won't hit most of your code, you should think about how to have automated tests. They don't have to be "pure, by the book" unit tests, but they should be an approach that can hit most of your program in an automated manner.

You don't need to do "pure" mocking either. If you're writing something that calls a webservice, you could write a mock webserver and redirect your program to it. If you're writing something that works with pipes, you could have a set of known files with known results, and always compare them.

The goal is that you should cover most of your program with code -> build -> tests; and only do code -> build -> run for experimentation.

There's a qualitative difference between working in a well-tested code base that's very hard to describe convincingly. A lot of my early development experience was in code bases that had little or no testing. Experiencing a well-tested code base totally changed things for me. Instead of work being a death-of-a-thousand-cuts experience, it became pleasant, steady progress.

I had the luxury of taking a well known data process and rewriting it with integration tests (input in, matching output with a golden file). It changed my professional life. Whereas before our deployment process included a 3 day wait and manual data checking on stage, after I was able to do deploys multiple times a day with confidence.

Made a believer out of me.

And TDD seems to create so many tests that you get huge false positive rates. I recently jumped on a project and I made a couple of fairly small code changes (a couple of hours) which caused 100 tests to fail. I then spent the next two days going through and correcting all 100 tests none of which found an issue in my code.

I have definitely dealt with code bases like that, and that sucks. But I have also dealt with code bases where the tests were great, and that's an amazing experience.

To do TDD well, I think it's important to release early and often and to reflect on one's experience (e.g., with weekly team retrospectives). That way if people are doing something unhelpful, like writing very duplicative tests, pretty soon they'll become an impediment to progress. The team will learn to write the useful tests, while skipping the ones that might fit some hypothetical pattern. It also helps people learn to design for testability; often, painful tests are a sign of bad design of the production code.

(Ruby, Rails)

They all seem a little fanatical in their pro unit test talk and don't discuss the downsides.

tl;dr:

- Focus on "automated testing", don't get obsessed with philosophising about "the true nature of a 'unit'", or other such dogma.

- Be empirical: base your rules on what works; don't base your work on "the rules".

- The goal of testing is to expose problems in our program: "test failure" is a success, because we've found a problem (even if that problem is with the test!). Anything else is secondary (e.g. isolating the location of failures, documenting our API, etc.). Avoiding this goal defeats the point (e.g. choosing to ignore edge cases).

- Focus on functionality rather than implementation details, e.g. 'changing a user's email address' rather than 'the setEmail method of the User class'. This improves reliability and makes failures more useful/meaningful (i.e. "this feature broke" vs "this calling convention has changed").

- Mocking is a crutch: it works-around problems that can usually be avoided entirely during design; it can still be very useful when a design can't be changed (e.g. adding tests to a legacy system).

- Testing a real thing is objectively better than testing a fake thing; we should only mock if testing the real thing is unacceptable.

- If two components always exist together, pretending that they're independent is a waste of time and complexity.

- Having some poor tests is better than having no tests. Tests can be added, removed and improved over time, just like anything else.

- "Property checking" is a quick way to find edge-cases and scenarios we wouldn't have thought of.

- Fast feedback loops are important. Reducing I/O and favouring pure calculation usually speeds up testing more than reducing the number or size of tests (e.g. "unit" vs "end-to-end"). Incidentially, this is also how we avoid having to mock.

To me, the most important chapters are

- https://www.obeythetestinggoat.com/book/chapter_mocking.html - https://www.obeythetestinggoat.com/book/chapter_purist_unit_...

Having said that, the concepts are universal.

[1] https://pragprog.com/book/bopytest/python-testing-with-pytes...

Mocking is a huge design smell. The more mocks or integration tests your projects requires to get full coverage the less modular your program is. A program that uses many mocks is a sign of very very poor design. You will find the code more complex to reason about and much harder to reuse code without necessitating a lot of glue code to make things work together. Without proper knowledge you won't even know the program is poorly designed.

I will grant you that 90% of programmers out there don't know how to design programs in a truly modular way, so most engineering projects will require extensive mocking. In fact most engineers can go through their entire career without knowing that they are making their programs more complex and less modular then it needs to be. Following certain design principles I have seen incredibly complex projects require nearly zero mocking (very very rare though).

Mocking indicates a module is dependent on something. Dependency is different from composition.

     Dependencies                                Composition


           C                                        C
 +---------------------+
 |                     |       +----------------+       +-----------------+
 |     A               |       |                |       |                 |
 |                     |       |                |       |                 |
 |        +----------+ |       |                |       |                 |
 |        |          | |    in |                |       |                 |  out
 |        |          | |    -->+       A        +------>+         B       +-->
 |        |    B     | |       |                |       |                 |
 |        |          | |       |                |       |                 |
 |        |          | |       |                |       |                 |
 |        |          | |       |                |       |                 |
 |        +----------+ |       |                |       |                 |
 +---------------------+       +----------------+       +-----------------+

left: 'A' exists as wrapper code around B and is useless on its own. To unit test A you must mock B.

right: every module is reuseable on its own. Nothing needs to be mocked during unit testing. No dependencies.

The only exception to the right example where you MUST mock is a function that does IO. IO functions cannot be unit tested period, they can only be tested with integration tests.

There's a name for the left approach. It's called Object oriented programming using inheritance or composition(the oop version of composition; not functional composition) as a design pattern. (both are bad)

There's also a name for the right approach. It's called functional programming using function composition.

I don't advocate that you strictly follow either style. Just know that when you go left you lose modularity and when you go right you gain it. All functional programming does is force your entire program to be modular down to the smallest primitive unit. Extensive mocking in your program means you went too far to the left.

tangent: Another irony around this world is that a lot of functional programmers (javascript and react developers especially) don't even know about the primary benefit of functional programming. They harp about things like "immutability" or how its more convenient to write a map reduce rather than a for loop without truly ever knowing the real benefits of the style. They're just following the latest buzzword.

In your functional example A is an input to B (or vice versa?), how do you propose testing one of the modules without first instantiating the other one?

  #oop

 engine = Engine(10)
 car = Car(engine)
 car.drive() #result 8

  class Car:
    def __init__(self, engine):
     self.engine = engine

    def ignite(self):
     self.engine.energy =- 1

    def run(self):
     self.engine.energy =- 1

    def drive(self):
     self.ignite()
     self.run()
     return self.engine.energy

 class Engine:
  def __init__(self, energy):
   self.energy = energy


 # ignite not testable without engine
 # run not testable without engine
 # drive not testable without engine and a car
 # ignite, run, and drive are not modular cannot be used without engine. 
 # engine testable with any integer. 
 # Car useless without engine
 # engine useless without car




 #functional \
 def composeAnyFunctions(a,b):# returns function C from A and B. See illustration above. 
  return lambda x: a(b(x)) 


 def ignite(total_energy):
  return total_energy - 1

 def run(total_energy):
  return total_energy -1 

 drive = composeAnyFunctions(run, ignite)
 drive(10) #result 8

 # compose testable with any pair of functions
 # run testable with any integer
 # ignite testable with any integer
 # drive testable with any integer
 # all functions importable and reuseable with zero dependencies. 
 # input_energy -> ignite -> run -> output_energy

you don't necessarily need the car or engine to simulate the energy output of driving.

Static functions avoid state. You put them in an object in java because java has to have everything in an object. In any other language these would just be top level functions namespaced into a package or something. You are basically using java in a more functional way. Which is fine.

Smalltalk is possibly the only OOP language that lets objects be compose-able and modular. Check out Pharo if you're interested. If you learn smalltalk well enough, you could apply its principles to traditional OOP languages and gain the modularity benefits.

I've seen Java 8 functional stuff get unreadable.

But other than that, is there any other concrete downside?

For using Pure Functions, I don't see any downside to this. Aside from it being impossible to use for outside the program side-effects like IO to device.

Isn't that a little extreme?

Anyway, this is what I am saying:

If you use functional programming your code will be more modular and reusable because the paradigm forces you to be that way.

If you use Object Oriented Programming your program will automatically be less reusable and less modular but more object oriented.

This is all I am saying. Your mistaken statement that I am promoting one style over the other is based off of this assumption: Modular programs are better than less modular programs. This is not True.

Something like a physics engine is a better fit for OOP then it is for functional. Although your program will be less modular as a result, OOP is still a better fit because physical objects are easily modelled with OOP objects.

Trees, graphs and algorithms involving things of that nature are a better fit for objected oriented programming then functional because many of these algorithms involve mutating nodes. Again, if you follow this style your program will become less modular overall as a result.

The ideal program is one that spans the spectrum of both OOP and functional. When it calls for it use functional or OOP depending on context. Overall for complex web applications that most startups make, in my opinion, the program should be more functional then it is OOP. A web request is basically a function that takes in a request as an input and outputs a response. The form factor of a function better fit for this, and you get high modularity as a side benefit. There is no point in simulating the request/response paradigm in a stateful Object while losing modularity in the process.

For a game. OOP is better in my opinion. Gaming entities involve constant mutation of things with state so OOP is a better fit. UI is a better fit for OOP as widgets are better represented by objects (FRP aka react&redux, imo works well but is an awkward abstraction)

There is one exception to this rule. In general Objects in object oriented programming are not compose-able. However, Smalltalk is an object oriented language where objects ARE compose-able. Smalltalk is the language that coined the term "object oriented" and although it is no longer popular as it was before it is still a very robust language and learning from it has huge benefits.

- https://dataintensive.net - Really deep dives into different types of data storage solutions, their history, and how they actually work.

- http://www.cattell.net/datastores/Datastores.pdf - Good paper that helps differentiate similar but different datastores. Really helpful when you're trying to pick a modern data solution.

It goes pedagogically through the way things are typically done in a relational database in such a clear way that word-for-word it's one of the best tutorials I've seen... but it also weaves a narrative of "how can this be done better/more scalably/more reliably/more flexibly-to-business-needs" in pointing to a streaming/event-sourcing architecture. You may or not need the latter right away, but it's a fantastic tool to have in your toolbox to be able to say "ah, this new requirement feels like it would benefit hugely from this architecture."

Especially for OP who's starting to think about the "why" of messaging queues, this could be a fantastically valuable first step.

[0] https://www.youtube.com/watch?v=fU9hR3kiOK0

[1] https://www.confluent.io/blog/turning-the-database-inside-ou...

There are some really cool data structures out there you might not know about. One of my favorite basic ones that I get a lot of use out of is the trie [1] (a.k.a. prefix tree). Very useful for IP calculations.

Also look into probabilistic data structures [2], very amazing things can be done with them.

[0] https://en.wikipedia.org/wiki/DTrace

[1] https://en.wikipedia.org/wiki/Trie

[2] https://en.wikipedia.org/wiki/Category:Probabilistic_data_st...

I keep hoping that someone will build a dtrace(1) CLI that transpiles to bpftrace.

[0] https://techcommunity.microsoft.com/t5/Windows-Kernel-Intern...

Also on a semi-related note, I think as a self taught programmer, it's easy to get stuck on things that seem cool but are just procrastination enablers (I know, I've been guilty of it for 20 years). Like, if you're about to start a new project and you want to flesh out what it's about, you really don't need to spend 5 hours researching which mind map tool to use. Just open a text document and start writing, or get a piece of paper and a pen. It won't even take that long.

I spent about 1.5 hours the other day planning a substantially sized web app. All I did was open a text file and type what came into my head. For fun I decided to record the whole process too[0]. I wish more people recorded their process for things like that because I find the journey more interesting than the destination most of the time. Like your journey of eventually finding message queues must have been quite fun and you probably learned a ton (after all, it lead you to message queues, so it was certainly time well spent).

[0]: https://nickjanetakis.com/blog/live-demo-of-planning-a-real-...

If I want to search a text file for a specific string, why wouldn't I just use `grep "hello" myfile.text`, or if I wanted to do it on a directory of text it's a minor change of `grep -R "hello" .`.

Why would I go through the trouble of opening a Python interpreter, or writing out a Python script to do the equivalent in Python?

Or if I wanted to grab the third column of a CSV, I would for sure just use the `cut` command or maybe `awk` (depending on what I'm doing).

For more complex parsing you can often pipe together a few commands and maybe convert it into a 5 line Bash script to make it a little easier to create variables, etc.. It becomes something you can whip up in 1 minute.

Then there's also more involved text parsing that doesn't require piping a bunch of commands together or shell script glue, in which case it comes back to using grep with its various flags and potentially a regexp. It's a natural fit for the problem and you can iterate on it so quickly.

Inevitably someone will read these arguments and think “those are just bad programmers”, but your point was that you “don't see the value of learning shell scripting”. The value is in not spewing absurd code like that. Shell commands are fast and efficient. There isn’t an emphasis on libraries because instead you use tools. Is `grep` not enough? Try `the silver searcher`[1] or `ripgrep`[2].

Are shell scripts the best instrument for every job? No, but no tool is.

[1]: https://github.com/ggreer/the_silver_searcher

[2]: https://github.com/BurntSushi/ripgrep

My point is that the cost for learning and using shell scripts is just too high compared to just using a modern language that's just as terse and a lot more powerful and flexible. Context switching from one language to another isn't free either.

imo the only time shell scripting was practical was when the only major programming languages were C, C++, and Java. imo even Perl5 is more practical than shell scripting.

Also I doubt that the python program you mentioned was that much bigger than a shell script

A good series of piped commands with tools available basically everywhere can solve problems you had no idea could be so simple to solve.

  man bash  
  man awk  
  man sed  
  man grep

sed + awk - https://www.amazon.com/sed-awk-Dale-Dougherty/dp/1565922255

awk - https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

general *nix text processing - https://www.tldp.org/LDP/abs/html/textproc.html

awk: - GNU Awk User Guide - https://www.gnu.org/software/gawk/manual/gawk.html#Getting-S...

- Grymoire guide - http://www.grymoire.com/Unix/Awk.html

sed: - Grymoire guide - https://www.grymoire.com/Unix/Sed.html - Official docs - http://sed.sourceforge.net/#docs

https://www.ibm.com/developerworks/library/l-sed1/index.html https://developer.ibm.com/tutorials/l-awk1/

awk is pure magic.

big timesaver: control-r

basic but useful all over:

    for i in *.c; do cp "$i" "$i.bak"; done

Quoting and expansion issues are a pain in shell languages...

You should almost always quote your Bash variables too. Here's why: https://nickjanetakis.com/blog/here-is-why-you-should-quote-...

- Shell scripting for running commands and managing files

- Unix utilities for processing text

These happen to complement each other nicely. It's not that bash is better at manipulating text than Python, it's that Python makes it painful to invoke commands (like those Unix utilities) and pipe data between them (e.g. https://news.ycombinator.com/item?id=17733865 ).

I can't recommend this book enough. I have a CS background, and still had quite a few "I can't believe this thing has been hiding in plain sight!" moments while reading it.

Are there good books for data intensive desktop apps? Like games or CAD design tools?

Property based testing aren't quite formal methods, but I think they are a good stepping stone. And they also somewhat force your code into an algebraic/functional style which also make it amenable to refactoring, better testing and is easier to understand.

Design tools like Swagger can help one think through services w/o diving into code. Code itself is a liability and should be thought of as "spending" not creating. Code is a debt.

Refactoring and code understanding tools, if you use PyCharm (you should, it is free in all senses), learn how to navigate into your libraries. Read you libraries.

I also recommend understanding message routing systems used in file sharing, like CHORD.

If you don't have a strong background in the math behind theoretical computer science, you might benefit a lot from an understanding of the formal rules around boolean logic, symbolic logic, & state machines -- especially, rules about when certain kinds of things are equivalent (since stuff like demorgan's law are used for simplifying and optimizing expressions a lot, and rules for state machines are used to prove upper limits on resource usage).

If you don't already, learn to use awk. It's a much more powerful language than it seems, and fits extremely well into the gap between command-line prototyping in shell one-liners & porting a prototyped tool to python or perl, and so it's a huge time saver: it is faster to write many kinds of tools in a mix of shell and awk and then rewrite them in python than it is to write them in python in the first place.

Totally agree on awk. I use it almost every day for quick little one-liners. Big time saver.

Also agree on state machines, because from there it is a short hop to understanding formal grammars and the foundation of compilers and languages, which has been immensely useful in my career.

Understanding how different languages, or inputs more generally, can be transformed into meaningful outputs is pretty satisfying. It's a topic that almost seems to transcend the realm of computer science.

It took me nearly a decade of working in distributed systems to be introduced to TLA+ and other tools in this space. Until then my knowledge had been built from textbooks describing the fundamental data structures, algorithms, and protocols... but those texts take an informal approach with the mathematics involved. And since I was self-taught I was reading those texts with an eye for practical applications than for theoretical understanding. I had no idea that a tool existed that would let me specify a design or find potential flaws in systems and protocols, especially concurrent or parallel systems, with such ease.

I think type theory and category theory have also been great tools to have... but I think mathematics in general is probably the more useful tool. Being able to think abstractly about systems in a rigorous way has been the single-biggest booster for me as a practitioner.

Can you instantiate this claim with an example? I'm somewhat knowledgable in both math and computer science theory but have yet to feel as though my math background has helped me in practical CS.

We tracked it down to a race between two services in the data plane. It turns out the VM controller wouldn't wait for the network controller to unplug a virtual interface before requesting the interface to be plugged back in. There was a lack of co-ordination happening. However it only happened when the network component was under heavy enough load that it would take too long to respond before the VM service finished rebooting the VM -- usually it was fast enough and the error wouldn't appear.

I managed to model this interaction at a high level in TLA+. From there I had suspected that the error was in the mutex locking code in the async library this system depended on so we refined the model pretty close to that implementation. As I recall we found that the mutex code wasn't the culprit -- a fine result. We ended up implementing some light-weight co-ordination mechanism to ensure that the VM service waits to acknowledge the progress of the network service.

Since then I've continued to use TLA+. I find that programming languages are insufficiently expressive enough to describe high-level interactions between other processes, network events, and humans.

So to me it seems that you could have gone straight to the lightweight coordination mechanism without the TLA+ model. And anyway, if there was a problem with the mutex, you could test that theory by doing additional logging or an experiment around the mutex functionality.

Start with the example you gave of messaging middleware. There are many BS CS curricula that do not address this at all. Also he mentioned that he had already learned about named pipes on his own. For many applications, named pipes could be a perfectly valid alternative to some external message queue system.

Looking at the items submitted, the vast majority are core skills that would necessarily be picked up by people who need to work with them. The idea that someone would not know about Makefiles or debugging or profiling or SQL just because they were a hobbyist or self-taught is ludicrous. If you are serious about C programming, whether it's a job or your hobby, you are going to learn about Makefiles. Likewise anyone seriously working on a data-centric application is likely to become well-versed in some database technology, up until a few years ago that would have automatically been relational.

And one other thing. Some of the most important skills in programming are in the domain of software engineering. Software engineering is very poorly addressed by many BS CS programs. So again, whether they have good SE skills is often not going to be determined by whether they have a BS degree or not. It's not even necessarily determined by whether they are working in a professional environment. It's mainly going to be a factor of their motivation to self learn and above all practical experience.

That's not to say it's required; some of the best professionals I know have non-CS degrees (one in fine art -- painting) or no degree. But if you're still young, I submit that a CS degree is totally worth your time.

Notice how I did not say that "the developers that certain degree holders find to be most 'polished' and easiest to work with will on average not have a degree".

Notice how I also did not say that a CS degree was not worth people's time.

We all, generally, specialize. I have never worked with 'big data', implemented any part of a commercial web site, or worked on HPC(just to give a few examples). What may be useful to one may be useless to another.

Definitely think about Software Engineering. Maintainability, debuggability, extensibility. Don't get lost in stupid details. Don't nitpick the coding style or try to optimize code that isn't currently meeting the requirements(if there are requirements, and if there aren't you should try to define them).

Vim or Emacs for powerful text editing.

A low level language. Sometimes Python doesn't cut it, or it is pretty suboptimal. If you're writing a trading bot, is speed of execution not important?

Operating System knowledge can be helpful at times. I bought one of the No Starch books "How Linux Works" and it is very helpful.

The command line and by that I guess you should know the common Linux commands (cat, grep, sort, uniq, head, tail, ls, top) if you use Linux and how to chain them together via pipes. To give some context, I can write one command which would require 8 lines of Python (saves you valuable time). If you use Windows, learn enough Powershell to be comfortable with it. On occasion I'll use Powershell over Python even though it is dirt slow for reading files.

https://www.w3schools.com/sql/

It has a database you can query and add to and what not.

I got my start in product support where a lot of our problems could be solved in SQL. We were encouraged to learn it, but most people were happy with the most basic syntax for DML. I worked through the entirety of Dr. Widom's course and it gave me the fundamental data understanding to be a real value to the teams I've worked for, and I used that opportunity to transition into being a full time developer.

Imperative shell, functional core.

The external shell of code in a project is responsible for network connections, console IO, etc. But the internal guts of a program should be largely functional, that is, instead of mutating (changing) values, consider returning different forms of the same value.

Decisions (branches, logic) are made at one level, data dependencies at another.

The talk Boundaries by Gary Bernhardt describes this model in detail: https://www.destroyallsoftware.com/talks/boundaries

They're a really nice UI for bootstraping projects, and even some small databases. A current project of mine for a client uses sheets as a backing store for an email collection list. Since there's only a few hundred rows, it makes sense at this scale and works really well for non-technical users.

VSCode or PyCharm (assuming you are still a Python developer) could be a good place to start. I'm always surprised when I see professional developers coding in Sublime Text and debugging with print statements (or their equivalent). Usually you have better options than that, especially for statically-typed languages - but even for JS and Python.

I've learned more messing up when using fancy git rebase stuff than any tutorial.

https://www.oreilly.com/library/view/git-pocket-guide/978144...

I also use magit (https://magit.vc/) - inside spacemacs (http://spacemacs.org/)

I found this zine (not free) to be helpful:

http://ohshitgit.com/

    git checkout -b
    git log
    git rebase -i

Using rebase results in a cleaner history and simplified workflow in many cases. However it also means that when you have a disaster, it can be truly unrecoverable. I hope you have an old backup because you told your source control system to scramble its history, and you don't have any good way to back it out later.

For those who don't know what I mean, the funny commit ids that git produces are a hash signifying the current state of the repository AND the complete history of how you got there. Every time you rebase you take the other repository, and its history, and then replay your new commits as happening now, one after the other. Now suppose that you rebased off of a repository. Then the repository is rebased by someone else. Now there is no way to merge your code back except to --force it. And that means that if your codebase is messed up, you're now screwed up with history screwed up and no good way to sort it out.

That result is impossible if you're using a merge based result. The cost is, though, that the history is accurately complicated. And the existence of a complex history is a huge problem for useful tools like git bisect.

I do get what you mean though. You effectively create new commits with an alternate view of history. I don't get quite why/how that causes a situation in which the code can't be merged? I don't rebase much, I prefer merging. Is there any resource that can explain why rebasing might be dangerous like that?

In general if branches diverge too far then you have difficulties merging no matter which strategy you use and sometimes if it diverges too far it just becomes hopeless. Mostly though if you are working in a team, commit daily and merging/rebasing frequently it should present fairly few problems.

I find I never run the actual command git bisect. I just do `git log --decorate --oneline --graph` and eyeball a good commit to start from and then basically do it by hand using commit messages to aid in making reasonable guesses as to where to try but following the basic binary search philosophy. Works well enough even with a complex history.

You rebase your private branch off of a shared master and pull in other people's commits. Someone else pushes out their rebased version using force. More commits are made on top of the other people's commits, including reversing some bad commits. You try to rebase off of the shared master.

In your last rebase, you are trying to replay all of the commits in your history that are not in the remote history. However git does not understand which local commits are from you and not pulled in on the previous rebase. It therefore tries to play them on top of the remote master if it can make sense of them. Which means that you bring back the reversed commits. You might find conflicts in code that you have not touched. You resolve them as best you may. And now you've got the definitive version of what happens, and no way with the screwed up history to figure out why it is going to go wrong. Then you force commit because that is how a rebase flow works..and everyone is screwed.

I agree on branches diverging too far. Merge early, merge often.

If you never run the command to git bisect, you should try it. What it's for is finding the random commit that recently broke a piece of functionality that nobody realized would break. Because nobody realized it, the log messages will say nothing useful. And you don't need to figure out where the change is - just write a test program for the breakage, run git bisect, and look at the offending commit.

Absolutely not. Force pushing a shared master is probably the worst sin one can commit with git. I guess you already have come upon the 'why' of it.

A "Rebase workflow" works so that devs use rebase to 'move' their work on an updated master after a pull/remote update, resolve potential conflicts locally, and do a fast-forward push to origin/master. This also works on copying work between different feature branches just as well.

SmartGit does a splendid job with this. In the Log window where you see all your commits and how they are related to each other, there is a Recyclable Commits checkbox. Turn that on and everything in the reflog shows up in the log tree, just as if it were any ordinary commit. You can right click one of these commits and add a branch there, or do any of the other operations you can do in the commit log window.

Same thing for stashes. Did you know they are just commits too? I didn't, until I clicked one of the stash checkboxes in SmartGit. Then the stash showed up in the tree just like any other commit.

I don't understand why so many developers are resistant to the idea of using a powerful Git GUI like SmartGit. For me it is like having a superpower compared to the meager options the Git command line gives you.

Even if you like the command line, it's not like you have to choose one or the other. You can use SmartGit and the Git command line, whichever makes any task easier for you.

https://www.mail-archive.com/dri-devel@lists.sourceforge.net...

I agree that rebasing should not happen in the main branch(es) of a repo.

A useful one that I'll add is what I call the sword command: `git log -S<word>` This one allows one to list commits that contain a particular change. This has been useful in tracking down old changes

    git pull
    git clone
    git status
    git commit -a
    git push
    git diff

(I actually prefer Magit, to the point where I sometimes run Emacs just for Magit when I'm using a different IDE.)

The comment I replied to says:

> Like they know enough to commit and push but that's about it.

In that vein, I was suggesting what I consider to be the most basic git commands outside of the "clone, commit, push" workflow.

[0]: https://news.ycombinator.com/newsguidelines.html

Git reset —soft and git reset —hard

Git cherry-pick

Git rebase

Git reflog

Git stash

Understanding this makes you able to manipulate branches and commits like a wizard. Once I learnt those, I can get myself out of the hairiest git problems.

[0] https://www.syntevo.com/deepgit/

For Google, check out the study guides for their certifications, specifically Cloud Architect (basic overview), Cloud Developer, and Data Engineer.

https://cloud.google.com/certification/

Be sure to follow along with the recommended Coursera and Qwiklabs tutorials and do the exercises. You'll learn about all kinds of neat stuff, like scalable application design, container technology, monitoring+metrics, various types of database technologies, data pipelines (including Pub Sub messaging), SRE best practices, networking+security, and machine learning.

I currently work on AWS, and don't find it a good starting point for diving in to these things quickly, but most companies use it so it wouldn't hurt to learn I guess. I still recommend GCP over AWS to start with, as their technology is far more interesting and focused, and quicker/easier to work with.

https://www.amazon.com/Pragmatic-Programmer-journey-mastery-...

Very frustrating, as I considered the first edition to be essential and upon reading your comment, instantly went to purchase the 2nd edition.

Edited to add: Found a date, Amazon is listing it as October 21, 2019.

It looks like you can get a DRM-free beta version of the ebook on their website, with free upgrades to published version once it's finalized: https://pragprog.com/book/tpp20/the-pragmatic-programmer-20t...

Org-mode in Emacs. My poor man's project management for a number of projects consists of a todo-<project>.org file listing all the planned features, the pending TODO items, the DONE items for the current working release, and release notes describing features and changes for each of the released version. In one place, I have the future features, immediate todo items, current completed item, and the history of all past releases in an org file, making things simple to access and manage.

For theoretical stuff, learn about transaction.