Recent searches
Search options
Administered by:
Doing a (brief) presentation Monday on long term software development (>10 years). Things you regret in year 8. I'd love to hear things you found out and wished you'd done differently. Or conversely, what worked out really well. Will turn this into a public presentation & blog post. To kick it off: PLEASE KEEP YOUR CODE AS SIMPLE AS POSSIBLE.
@bert_hubert spend the extra hour/day/month/year to get that API right. Not having to document and explain over and over while an API is awkward is going to suck the life out of you.
We took two+ years to nail down the @couchdb API and not a day goes by where I don’t appreciate that we did.
@bert_hubert The main benefits:
- you don’t have to change the API in future and your API clients do not have to upgrade, *ever.*.
- you’ll end up with an API that multiple generations of developers will appreciate learning, citing as a good example to follow and generally be appreciative of your work.
@janl @bert_hubert@fosstodon.org nice! Also saves a lot of cognitive load. Will add this!
@bert_hubert Having worked on one project for 17 years, the best decision I've made was to minimize dependencies and choosing tech that seemed like it'd stick around for a while.
I've had much more recent projects where I already regretted using a library that broke within a year or two and isn't being maintained anymore.
① Are other people using it? (easy to check via distros like Alpine or Gentoo)
② Could I end up maintaining it? (either due to upstream passing me maintenance, or needing to fork/patch it)
Question ① isn't very important but makes sense to check, could otherwise end up missing a better dependency without doing so, and sometimes turns out the dependency is used by no one because it's unusable.
Question ② is probably the most important, could skip it for throwaway scripts but well… temporary sometimes spans for a very long time.
Note: I tend to consider not just libraries but also utilities, buildsystems, language implementations as part of the dependencies. Could even consider deployment like packaging into distros in there as well but can't really call that a dependency of the software.
@lanodan @bert_hubert @ayo In some cases it makes sense to code your own packages instead of relying on external parties. Minimising the number of dependencies stands out to be the most effective way to make your system future proof and easy to maintain.
Operating systems, file formats, protocols, … do change, and dependencies can help on this part, so you'd have to be careful to not minimise too much.
And picking a dependency should mean keeping an eye on it, hence the question of possibly ending up maintaining said dependency.
And future proof… I think the best way to approach the future proof ideal is to allow independent implementations to be made.
You can have code which should work well in the future but realistically there's no telling what will happen in roughly 10 years. Which to me is a rather short timespan for software, but enough for big changes to happen.
@lanodan @bert_hubert @ayo It is always a fine balance between what is good for the short term and what is wise for the long term. Especially in web development there are new frameworks and related components popping up left, right and center. One should always think twice before jumping on the next hot thing that comes along. Keep it simple and have future maintenance in mind. That might mean that you have to do a little bit more work now in order to save time and money in the long run.
Like web browsers have pretty good API compatibility between each others and APIs very rarely break, same thing for NodeJS itself.
One thing that's lacking is the JS standard library, so if you want to target both browsers, NodeJS, … you're definitely going to need wrappers and maybe few libraries. Which okay makes sense, you're targetting different platforms.
But then you have build tools and frameworks that are so API-unstable that keeping up is a challenge.
Which makes no sense and just means that using them undermines what otherwise would be very stable platforms.
@lanodan @bert_hubert @ayo I never managed to get into the whole server side JS thing. My background is PHP and now with version 8.4 coming up thing are starting to look very bright on the LAMP stack.
The problem with most JS client side frameworks and tools is that they 'try' to turn JS is some sort of OOP thing which it just isn't. This results in some half-hearted construction with namespaces and classes that don't seem to make things better.
@bert_hubert The mistake is always in the piece of code you dismissed at the start of debugging because "it can't be there".
@bert_hubert
The point I would make most loudly is “avoid implicit magic”, but “keep your code simple” covers that quite well.
A few more:
· Migrating from one dependency to another is always expensive, but it’s even more expensive to pause the migration halfway through.
· Document document document. The product owner will move on, your lead dev will move on, other devs/QA/etc will join and leave…
· No heroics. That works for startups (sometimes), but you can’t pull it off long term.
@dcoderlt @bert_hubert Agreed, so called 'smart code' is just stupid. Create simple code that anyone is able to maintain and refrain from creating 'complicated' structures. Use names that make sense and document! Draw diagrams and write down what the reason is why you chose a certain solution over others. This will not only help others working on that code in the future, it will also help you when you have to work on your own code in something like four months from now.
@alterelefant @bert_hubert
Yep. Just keep in mind that sometimes, complexity is unavoidable and it has to live *somewhere*. A project that is actively developed for 8+ years probably contains at least one example of that.
@dcoderlt @bert_hubert There is indeed always a bit of complicated business logic somewhere. The trick is to document its inner workings very extensively in writing and using diagrams. Break things down into smaller portions where possible.
@bert_hubert one problem I’ve seen: having foundational code only one person knows about and the rest of the team is discouraged from touching. bottlenecks while they’re there and major trouble when they leave.
@bert_hubert anything over a decade should probably consider a state machine ...
Two things that worked well from maintaining https://jpylyzer.openpreservation.org/ (a relatively small project) for 13+ years:
1. At a really early stage, taking up the offer from a colleague to re-factor the core module from my "GW BASIC-style" (their words!) Python to proper object-oriented code, which turned out to be MUCH easier to maintain long-term.
Also more in general, recognizing that others may be better at certain things, so resist the urge to do everything yourself.
2. Always updating the documentation *immediately* after any code changes. I've seen this go wrong in various similar projects, with docs lagging badly behind code to the point where it's nearly impossible to bring it up to date without a major investment of time/resources.
Do not rely on consultants to write your code. They cost a fortune, rarely have the appropriate skills, are not the least bit invested in you, and, after the fifth Anderson kid touches your code it will be such a rat's nest of cruft, it'll be far easier to burn it all and start over than to fix even what appears to be the simplest bug.
Instead, hire, retain, and reward good team members for as many of those ten years as possible.
@bert_hubert @djswagerman The lie of self-documenting code becomes more apparent as time passes. Unless something is _crystal clear_ (and even if it is) take time to document it while it is fresh in your mind!
@onorio @bert_hubert @djswagerman That 'self-documenting code' still needs external documents that explain the concept on a higher / more abstract level. Always include diagrams and drawings in order to visualise relations between components and things like standard workflows. More documentation is better.
@bert_hubert having worked on the same code base for 30 years: Don’t rely on bugtracker references, explain in the commit msg or comment. (Bug trackers come and go). Any undocumented feature (or bug!) will be used by someone who depends on it with their life.
People never ever test release candidates, just release. Any extremely clear log message that says exactly what happened is technobabble to endusers.
Anyone offering you under $100 for help will be demanding $100k support services. Killing of things takes years even if RFCs help you with deprecation statements.
People use 15 year old Linux kernels and distros in production.
Bad hacks are really hard to get rid of. You will forget why you did something weird and rediscover after painful hours why it was smart - document it properly !
@bert_hubert as a former infrastructure IT advocate. Plan statistics and measurements into code so you can track volume and choke points. Please please build security, backup, and redundancy into your code from the beginning. And involve the #infrastructure team. Pretty please.
@bert_hubert 10 Jaar gewerkt aan een goed geïntegreerd workflow systeem voor de Nederlandse gemeenten…
… en dan kom je tot de ontdekking dat de aanbesteding voor vijf jaar met een verlenging van nog een keer vijf jaar de deadline is. In die zin dat ze alles weg-flikkeren en helemaal overnieuw beginnen. Net wanneer zo ongeveer alles perfect gestroomlijnd loopt.
Wat had ik anders gedaan als ik me dat voor het achtste jaar had gerealiseerd? 
@bert_hubert having worked on a by now 25-ish year old code base for the past 7 years: accept that idioms change, and that different parts of the code base will feel different. Do not use a different idiom for a small change; should you wish to change, do it in full. Try to keep people with the code for its entire life. Should that exceed a career, get people up to speed every half decade or so to stick with it for the next few.
@bert_hubert Clean up deprecated parts, lest they come to burden your maintenance. Keep up to date with external dependencies; the work required becomes bigger if you don't regularly update. Failing that, adopt maintenance of those dependencies (possible, if they're open source and e.g. otherwise abandoned), or write them out. Capture your reasoning and requirements in writing, as you will forget important details.
@bert_hubert Use boring/mainstream technology; the new hotness may in a few years turn out to become an otherwise abandoned sidetrack. Stick to standards, and even then don't use much beyond the well trodden happy paths, implementations of the standards will have bugs there.
@bert_hubert define simplicity. Simplicity in lines of code, or conceptually. Those can be opposed. You might be able to write „3 simple lines of code“ using sophisticated language features, but you do something conceptually difficult. Or you can do sth conceptually simpler but with more complicated lines of code.
Honest question: what do you look for in these cases?
@bert_hubert Don't quickly hack tests together to get coverage, making the test code maintainable is as important as the core code, if not more so.
@jelte that’s meta! My standards may be too low. I’m already impressed if a project has some decent tests at all!
So far, it looks like the top three tips are:
Simplicity
- minimize dependencies
- appropriate frameworks
for scale, etc. (?)
- ?
Continuity
- keep the team together
- train new members before
oldsters retire
Transparency
- Document, document, document
- instrumentation?
Building "cloud native" is like signing an open ended apartment lease that allows your landlord to tear down the building with you still inside, while simultaneously doubling your rent twice a year.
Maybe I'm far too cynical, but, based on the last 10 years, I feel fairly confident in my assessment.
Problem is, how do you *not* do cloud native if the entire talent pool only knows that?
@bert_hubert @lolcat @arclight “Cloud native” is een generieke term voor het toepassen van open source technologieën als Kubernetes. Het impliceert niet per se het gebruik van hyper scalers (maar leidt er vaak wel toe)
@bert_hubert @lolcat @arclight
Is there a somewhat official definition of "Cloud Native"?
I see it most commonly used for "build yourself on or glued together third party solutions you have no control over".
Because I don't like that angle I think of it more as building solutions with "cloud native methodologies": build adaptable and resilient software.
These are things that you can do in any system but "the cloud" makes them more of a requirement than other platforms.
I'm very interested in this because I entered the software industry in 2013 and only ever worked with companies hosting in the cloud.
I call myself a "cloud baby" because of that 
Looking forward to @bert_hubert post!
@lolcat @bert_hubert @arclight
we did self-hosting at the last job but with DC prices likely to jump with all the crypto/AI silliness, that's going to be a bit of a gamble too if you're not signing long term leases.
@paul_ipv6 @lolcat @arclight but do you think consuming services will shield you from that risk?
@bert_hubert @lolcat @arclight
it will buffer the cost bloat. crypto/AI have a whole slew of other risks beyond just make DC owners stupidly greedy.
@lolcat @bert_hubert @arclight many originally “cloud native” concepts like object storage, containerization, message queues etc are a complete lock at this point, the standard operating procedure going forward. You should use them regardless of whether you use public megaclouds. And you can still own 100% of your stack like before.
@lolcat @bert_hubert @arclight
To me its more like building a camping van. If you do not like your landlord, you can move elsewhere. If you get too attached to the local infrastructure, it becomes an poorly built house with a bad lease.
@lolcat @bert_hubert @arclight But "cloud native" only means that your dependency is k8s/docker instead of Linux/libc, which is a reasonable bet at this point?
You can self-host k8s right now and it will likely get easier over the next decade.
Nonstandard "serverless" by one of the Big Three OTOH is a whole different beast -- that can not be expected to be a commodity in a decade.
@Sweetshark @lolcat @arclight this is not how it is being interpreted by the people I talk to. They consume databases and all kinds of higher level services. They mostly aren't targeting kubernetes or linux.
@bert_hubert @Sweetshark @arclight
[random thoughts removed]
On second thought, I'm going to sit this out for a bit to listen..
@bert_hubert @lolcat @arclight It is certainly convenient to use some SQL instance on one of the three big cloud providers these days if you run your workloads there and that locks you to that provider to a certain extent.
But: this lock-in is neither more nor less than any other hosted SQL solution you would buy. So this is not cloud-specific.
@lolcat @bert_hubert I am so absolutely thankful that the codes I work with have zero networking component. Put them in a container to control (almost) all the system dependencies, run them in isolation until they're done, harvest the results for post-processing. They run in an HPC cloud but from the application's perspective, everything they need is local.
Does your code execute across nodes? Does it access shared storage?
If so, there's an important networking component - you've just chosen to ignore it. Maybe that's OK in your case. Maybe it's not - you'd have to understand what it's doing on the network to have any idea.
Not dinging you - just pointing out that containers aren't magic.
Tooting to apologize for the tone of my earlier response. It's worded all wrong, and comes across way to agro. I was trying to make a rather banal point about popular attitudes toward containerization. It probably didn't need to be made at all, and certainly not in such a clumsy way.
Thanks - I shouldn't have expressed my thoughts in a response to Arclight's toot - sounded too much like a personal attack. I can be too reactive to statements like "just use containers," which is NOT what Arclight wrote, but an internal dialog I let bubble over into the outside world.
@lolcat @bert_hubert No offense taken :)
Clearly there's some networking component in our architecture but most of the qualified engineering codes in containers run in isolation. We do have some MPI codes which coordinate a number of nodes as a group. Most codes just talk to the filesystem; the networking happens around them at the system level, not the application level.
Containers are used mainly for configuration control - you define your application and dependencies, you qualify the code in the container and on the controlled hardware, and the container doesn't change until the next software release which is generally once every year or so. It takes at least a month to write up, review, and approve the QA docs so web speed code changes aren't possible or really even desirable. These codes are intended as tools not services, change is infrequent because the underlying science & engineering don't change quickly.
We're also dealing with QA requirements that we keep our safety related codes in a runnable state indefinitely ("for the life of the facility" which in some places is expected to be 80 years.) It's a very different universe than managing web apps. :)
@lolcat @bert_hubert Nuclear reactor vendor. "Move fast and break things" is not a societally-acceptable operating principle. :)