Don't forget changing the day 1/3 of the time. But more importantly, if I'm scrambling to solve an incident, the last thing I want to do is deal with time conversions in my head.
You just do it once and think in UTC going forward. But as I said to the other person, if you're only dealing with one timezone it's fine, just when you have multiple it's a lot easier to just deal with one time and let everyone convert.
I'd expect everyone who works in computer related jobs to know their UTC offset.
I'm able to subtract 8, but if I'm scanning logs it's one more thing to process.
If it's local time I know instantly when something happened, without having to do mental math.
Is there anything wrong with ISO8601 (including timezone offset) for storing times? They're in my local time, but they can be accurately converted to any other timezone.
No problem as long as it's all local, but it's a big annoyance to the other teams if I'm trying to coordinate with the West US team who're on PT, the East coast on ET, Europe on CET, India on IST, Australia on BBQ...
It's just easier for everyone if we agree on UTC and everyone knows their own offset.
Yeah. I log in json + unix timestamp nanoseconds, and just convert it to something human-readable on display (github.com/jrockway/json-logs). I am not sure why logs need to be "pretty" at time of logging when they can be made pretty at time of display. Doing it at time of display means you can just use local time, and then nobody is confused.
I find all the holier-than-thou UTC worship especially ironic given that this post is about recurring scheduled tasks, for which naïve UTC almost never provides the expected results
UTC is the right choice for representing specific moments in time (so yes, log timestamps) but there are so many pitfalls outside of that narrow use case
Nobody hurt them. Please don't do this kind of childish rebuttals. It looks really silly on HN. If you don't want to use UTC, that's one thing. You do you. But there are reasons for sane professionals (who weren't hurt by anybody) to expect their coworkers to use UTC when possible. Because using local timezone leads to some problems including the one in TFA.
Well me, I have been burned by fixing this UTC bug at like 8 companies and overall more than four year of my life (between other things, each project took at least six months of people poking at things) because at an eventual scaling point it starts biting your ass everywhere in every service.
Just because it worked once doesn't mean it's good.
I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST, and is only one hour off of California (where we all were) for five months, and the same as here the rest of the year.
I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.
Whenever you are unsure whether to use a clever solution or follow the globally accepted standard in your work as a DevOps or Software engineer, always choose the standard.
Among other things, in an incident, people’s brains aren’t working - the more they have to remember about your particular system, the more likely they are to forget something.
it was already a 40 year old standard at the time you're talking about.
awareness of UTC being the correct choice has definitely increased over time, but UTC being the correct choice has not changed.
you say reddit servers use UTC now, which implies there was a cutover at some point. were you still at reddit when that happened? were you still hands-on with server maintenance? any anecdotes or war stories from that switchover you want to share?
because I can easily imagine parts of the system taking a subtle dependency on Arizona being Reddit Standard Time, and the transition to UTC causing headaches when that assumption was broken. your memory of this "clever" trick might be different if you had to clean up the eventual mess as well.
Hold on, I'm not a sysadmin guy. Are you folks saying the server should not know what part of the world its in, that basically it should think it's in Greenwitch?
I would have thought you configure the server to know where it is have it clock set correctly for the local time zone, and the software running on the server should operate on UTC.
From a logging perspective, there is a time when an event happens. The timestamp for that should be absolute. Then there's the interaction with the viewer of the event, the person looking at the log, and where he is. If the timestamp is absolute, the event can be translated to the viewer at his local time. If the event happens in a a different TZ, for example a sysadmin sitting in PST looking at a box at EST, it's easier to translate the sysadmin TZ env, and any other sysadmin's TZ anywhere in the world, than to fiddle with the timestamp of the original event. It's a minor irritation if you run your server in UTC, and you had to add or subtract the offset, eg. if you want your cron to run at 6PM EDT, you have to write the cron for 0 22 * * *. You also had to do this mental arithmetic when you look at your local system logs, activities at 22:00:00 seem suspicious, but are they really? Avoid the headaches and set all your systems to UTC, and throw the logs into a tool that does the time translation for you.
The server does not "know" anything about the time, that is, it's really about the sysadmin knowing what happened and when.
you've got it backwards - the server clock should be in UTC, and if an individual piece of software needs to know the location, that should be provided to it separately.
for example, I've got a server in my garage that runs Home Assistant. the overall server timezone is set to UTC, but I've configured Home Assistant with my "real" timezone so that I can define automation rules based on my local time.
Home Assistant also knows my GPS coordinates so that it can fetch weather, fire automation rules based on sunrise/sunset, etc. that wouldn't be possible with only the timezone.
Windows assumes computer clocks are local time. It can be configured to assume UTC. Other operating systems assume computer clocks are UTC. Many log tools are not time zone aware.
that's the difference between "aware" and "naive" timestamps. Python has a section explaining it in their docs (though the concept applies to any language):
1) Most software gets its timestamps from the system clock
2) If you have a mismatch between the system time and the application time, then you just have log timestamps that don't match up; it's a nightmare - even more so around DST/ST transitions
Would he by any chance refer to it as Zulu or Zebra time? The Z-suffix shorthand for UTC/GMT standardisation has nautical roots IIRC and the nomenclature was adopted in civil aviation also. I sometimes say Zulu time and my own dad, whose naval aspirations were crushed by poor eyesight, is amongst the few that don’t double-take.
While I agree on this particular instance, there are two types of things future engineers have to clean up after: Their predecessors thinking too small (and picking the easy route) or too big (and adding needless complexity).
One is not necessarily and in all instances less of a mess to clean up behind than the other.
What if it's your personal machine? I'm thinking about jobs I've set up... thing is, I actually do want those to align to DST in most cases. For example, ZFS scrub should start after I leave for work so that it has the greatest chance of being done by the time I get home. (It's too loud to run overnight.)
I can’t quantify how much time my team wasted in diagnosing production glitches on checking the wrong time offsets but it was substantial. One of our systems wasn’t using UTC, and given enough time the fact that Slack wasn’t using it either does become an issue. When an outage transitions to All Hands on Deck everyone needs to get caught up to what’s going on preferably under their own power so you don’t suffer the Adding Resources to a Late Project problem.
So that first alert that came in ?? minutes ago you need to align with the telemetry and logs in order to see what the servers were doing right before everything went to shit.
This shouldn’t be hard to deal with if the timestamp is always serialized with the offset: I’m much more picky about always persisting the offset than about always persisting UTC
That's still a bit risky as Arizona might just change its time zone definition on a whim. I'm an engineer on one of the big calendaring applications, and it's mind-boggling how often stuff like this happens world-wide, sometimes short-notice (a few weeks in advance). It regularly gives us headaches.
Agreed, watching the rate of changes to timezone databases will rapidly disabuse you of the notion of any constant. It's rare that a day goes by without an update to some definition somewhere, which is astounding.
Closely related pet peeve: Log display web UIs that allow selecting display timezones including UTC absolutely insist on deriving my preferred time format (12/24 hours) from my browser language preference.
If nothing else, selecting UTC should be a very strong hint to any UI that I am capable of parsing YYYY-MM-DD and 24 hour times.
If you use <input type=time>, the browser uses locale or user regional preferences… even if the app is in an application domain (e.g. medicine, military, science) where 24h is preferred even in 12h-predominant locales. There is no way for the app to say to the browser “this time field should be 24h in all locales”, the only option is to build a custom time field
Some browser APIs respect `LC_TIME`. Others say "fuck standards, I'm using `LC_MESSAGES`".
This means that if those locales differ about say, y/m/d ordering, it is quite possible for the way the browser formats a date to be different than the way it parses the date.
Operating systems generally allow user override of locale settings, and browsers generally respect that; I use a locale which officially has 12 hour time as its standard (Australian English), and I always override it to 24 hour time in user preferences (although Australia is rather inconsistent, e.g. in Sydney, train services use 24 hour time; in Melbourne, the metro trains use 12 hour time but the regional services use 24 hour time)
Interesting, so you're saying that that OS-level time preference is available via JavaScript? I wasn't able to figure out how to query that in a little bit of trying, so I assumed there was no API for it.
If there actually is, I'm now even more upset at that log web UI.
For me, the first returns 24 hour time and the second returns 12 hour time. Because 12 hour time is default for my primary locale (en-AU), but I override it to 24 hour time in my macOS settings.
I know the same works on Windows, and I’m sure it works on Linux too, just the way you configure it is different for each OS.
I would especially like to call out the Scandic Hotels chain for this behaviour as well. Booking a hotel room should not involve me wondering if I booked it for the wrong day when I'm not in an European time zone.
That's interesting because I strongly prefer that timestamps are stored as UTC and converted to my timezone on the fly for easier debugging. I dunno about using the browser language choice (do sites really do this)? Usually a simple conversion with javascript is fine (javascript knows the local timezone).
That is the correct and only sane way to determine date format, timezone is not the same as formatting preference. But yeah it sucks. Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale.
In firefox you can try it out in about:preferences by setting en-CA as the topmost option in "Choose your preferred language for displaying pages"
And I just checked, firefox is capable of understanding system-wide split locale settings, if you only want LC_TIME
> That is the correct and only sane way to determine date format, timezone is not the same as formatting preference.
That's unfortunate, but then wouldn't the sane way for localization-aware software be to ask the user and not make any such assumptions?
> Assuming you're using the en-US locale, have you tried using the en-CA locale? It has ISO8601 formatted dates, and is otherwise pretty similar to the en-US locale.
Thanks, I'll give that a try tomorrow! If my log exploring UI of, well, not quite choice actually respects that, I'll be very grateful :)
In a way it is asking the user. Date formatting rules are traditionally tied to locale, so the user picks their locale and their expected date formatting rules are derived from that.
On Linux you can mix and match via LC_*= environmental variables, but that appears to be complexity the browser vendors didn't expose in their UI.
Java allows setting the default timezone at jvm level and everyone in our org were setting their favourite TZ which was mostly PST somewhere in the code.
We had application logs and system level logs with different timestamp and someone decided a certain db column has to be a string of the format 'yyyy-mm-dd hh:mm:ss". You can imagine the confusion and the "fun" time we had while debugging logs from multiple systems at specific times.
Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.
> Finally, one lead put their feet down and mandated that setting timezone to PST should be the first thing all applications do and db columns should be considered PST unless otherwise required.
That seems to me like a really bad mandate. If you are going to mandate something, mandate UTC. If someone forced me to change a system from UTC to something else, I’d not be very happy. It’s the kind of decision which makes you seriously question if you want to work there any more.
Does that mean you ran PST during daylight savings time and half the year your logs were “off” by an hour? Or did you actually run Pacific Time and deal with the clock changes twice a year?
> I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
Also important here is that the date stays the same! Even though I've gotten used to instinctively decoding UTC, that part is still frustrating sometimes.
There are many solutions, but when you're actually running a site with millions of daily visitors and one person focused on ops, you do the easy thing, not the right thing, unless those happen to coincide.
If this is a complexity battle, I'd rather that sysadmins have to use a tool to read logs vs. figuring out why the server is melting down because all the jobs are running 2x or answering questions about why jobs didn't run at all.
In my old gig, jobs ran for many many HOURS and consumed most of the IO on the server (processing reams of data), which meant that when you got them running 2x because of overlap, it was a trainwreck.
In my experience, it's really not pleasant having to work with logs that require a dedicated viewer compared to regular old text files I can use Unix commands available everywhere (`tail`, `head`, `wc` etc), with, i.e. not just on the server producing the logs, but also the device I'm viewing them on.
That said, I absolutely prefer UTC times in logfiles. I'd rather do some math every time than make wrong assumptions about the local timezone of the file even once. (Even correctly indicated local times are more annoying, since I have the math for the offset of my timezone to UTC memorized, but not that of every possible server time to mine.)
Yup. There's always some kind of tool anyway so why not. I mean, even if you read the logs as they come out of a screaming line printer, the tool is the thing that takes log messages and spits them out the printer port.
How hard can it be to write a log cat utility in awk?
Nooo... we have a bunch of metrics and logs reporting systems at my company. Some of them are in UTC, some of them display my local time, some of them display China Time, and I'm trying to collaborate with colleagues in London and Australia who get data displayed in their local times as well. When I'm working to address an incident and combing through multiple systems to try to correlate events, it's a pain in the ass having to constantly double-check which time zone this data is in.
I'd rather the logs be consistent, and not rely on every single person who ever looks at the system logs make sure to use the correct tool the correct way.
In my experience, the hard part is getting everybody else to do that. And then also getting them to actually include the timezone in their communication with you.
The problem is a lot of times when you are looking at logs you are already very far off the happy path of things you look at often and want to invest resources into displaying well.
This is not smart, this is just surprising to the next person who is going to maintain your ”smart” tricks. Thank god they switched to utc, that is what everyone expect.
Actually, back then, 18 years ago, most people expected your servers to be in Pacific or Eastern time, depends on where your company was headquartered, because none of us really had global technical workforces back then. We all pretty much worked in one office and used the local time zone, because often our servers were in the building with us or in a datacenter nearby.
Case in point, before reddit I was at eBay, and we kept all those servers in Pacific time, since the entire technical workforce was in Pacific time, as well as all of the servers.
Making blanket statements like that without considering the context of the time is usually not a good idea. ;)
> most people expected your servers to be in Pacific or Eastern time
I was there back then, working for shops people have heard of, and I honestly don't know where you're getting this idea from. Some places did things wild and wacky when they were wee small, but most of us quickly learned that such shenanigans (like fun server naming conventions) start to fall apart and maybe we should do things differently.
Ah you think UTC is your ally? You merely adopted UTC. I was born in it, molded by it. I didn't see DST until I was already a man, by then it was nothing to me but blinding!
In the 1980s, PT and ET were common. I was working at Bell Labs then, and one of my jobs was to change the time zone (back then it was two words) on the testing machines, as needed. This is stuck in my memory since to change the timezone, you needed to edit the Unix kernel source code and recompile it!
2000 for me. That was the first time I had users from outside my own time zone, so I figured it was better to just use UTC for everything and just convert depending on the user's settings. I think I just applied the thinking to the whole server.
It was probably the smartest option at the time, given the context. As long as it was documented properly, no one on such a small team would have been surprised. Spend more than the 30 minutes you've spent here on HN so far, and you may learn a thing or two about what is "smart", and how that is inextricably linked to the given situation.
If you're like 5 people, each with a list of TODOs that doesn't fit on one screen, it's pretty smart to just do something quick and good enough, then move on, revisit it in the future.
To define AGI, we'd first have to define GI. Humans are very different. As park rangers like to say, there is an overlap between the smartest bears and the dumbest humans, which is why sometimes people can't open bear-proof trash cans.
It's a similar debate with self driving cars. They already drive better than most people in most situations (some humans crash and can't drive in the snow either for example).
Ultimately, defining AGI seems like a fools errand. At some point the AI will be good enough to do the tasks that some humans do (it already is!). That's all that really matters here.
They may require training but that training is going to look vastly different. We can start chatting about AGI when AI can be trained with as few examples and as little information as humans are, when they can replace human workers 1:1 (in everything we do) and when they can self-improve over time just like humans can.
Nah, it's more than that. My very boomer parents just sit on the couch doom scrolling all day, and they were late getting smartphones.
Although they also got us Ataris in the early 80s and internet access in the late 80s, so they were technology forward through their whole lives. So maybe they lived more like late Gen X...
I think you are correct. Tech was binning people and older Gen X slid forward identifying with late Gen X much more than those on the other side of the divide.
"The new legal drama TV series “Cupertino” was just announced as part of CBS’s 2026-2027 season, as reported by entertainment trade publications Wednesday morning. The show ... is set to star Mike Colter (“Luke Cage,” “Evil”) as a lawyer whose tech startup employer cheats him out of his stock options. The show’s logline frames this as a David vs. Goliath fight, in perhaps the first time in history that a lawyer for a tech company has been framed as an underdog."
You tell the inspector. Point out that the hallway is 25cm short of code. I'm assuming since they are using metric they aren't in the US, but at least here the contractor can't say no the inspector without losing their license.
I knew this title looked familiar! It was required reading when I took Searle's course. I always thought it funny that CogSci majors (basically the AI major at Berkeley in the 90s) were required to take a course from a guy who strongly believed that computers can't think.
It would be like making every STEM major take a religion course.
Not sure that equivalence works, cognitive science doesn't require that people believe that computers can think; and STEM doesn't require that people think of the world in a purely mechanistic way - e.g. historically, many scientists were looking for the rules of a lawgiver.
Not a bad idea, actually. Religion is a big deal and it can only help to know the basics of how it works. Some of the fanboi behavior common in tech is at least religion adjacent.
Way back when I worked at eBay, we once had a major outage and needed datacenter access. The datacenter process normally took about 5 minutes per person to verify identity and employment, and then scan past the biometric scanners.
On that day, the VP showed up and told the security staff, "just open all the doors!". So they did. If you knew where the datacenter was, you could just walk-in in mess with eBay servers. But since we were still a small ops team, we pretty much knew everyone who was supposed to be there. So security was basically "does someone else recognize you?".
Well, you put a lot of trust in the individuals in this case.
A disgruntled employee can just let the bad guys in on purpose, saying "Yes they belong here".
That works until they run into a second person. In a big corp where people don't recognize each other you can also let the bad guys in, and once they're in nobody thinks twice about it.
way back when DC's were secure but not _that secure_ i social engineered my way close enough to our rack without ID to hit a reset button before getting thrown out.
late reply but, no, i really needed to hit the button but didn't have valid ID at the time. My driver's license was expired and i couldn't get it renewed because of a outstanding tickets iirc. I was able to talk my way in and had been there many times before so knew my way around and what words to say. I was able to do what i needed before another admin came up and told me that without valid ID they have no choice but to ask me to leave (probably like an insurance thing). I was being a bit dramatic when i said "getting thrown out" the datacenter guys were very nice and almost apologetic about asking me to leave.
There's some computer lore out there about someone tripping a fire alarm by accident or some other event that triggered a gas system used to put out fires without water but isn't exactly compatible with life. The story goes some poor sys admin had to stand there with their finger on like a pause button until the fire department showed up to disarm the system. If they released the button the gas would flood the whole DC.
My point is that while the failure rate may be low the failure method is dude burns to death in a locked server room. Even classified room protocols place safety of personnel over safety of data in an emergency.
It wasn't Equinix, but I think the vendor was acquired by them. I don't actually blame them, I appreciated their security procedures. The five minutes usually didn't matter.
If my whole team is based in Pacific Time, I'm gonna use something close to PT so I don't have to do hard math when reading raw logs.
If we're all US based, I still want something in the USA. The math is easier.
reply