Panic Inc.

Panic Blog

May 17th, 2017

Last week, for about three days, the macOS video transcoding app HandBrake was compromised. One of the two download servers for HandBrake was serving up a special malware-infested version of the app, that, when launched, would essentially give hackers remote control of your computer.

In a case of extraordinarily bad luck, even for a guy that has a lot of bad computer luck, I happened to download HandBrake in that three day window, and my work Mac got pwned.

Long story short, somebody, somewhere, now has quite a bit of source code to several of our apps.

Before I continue, three important points:

  • There’s no indication any customer information was obtained by the attacker.
  • Furthermore, there’s no indication Panic Sync data was accessed.
  • Finally, our web server was not compromised.

(As a reminder, we never store credit card numbers since we process them with Stripe, and all Panic Sync data is encrypted in such a way that even we can’t see it. Read more.)

The other important fact is that I feel like a monumental idiot for having fallen for this.

How did this happen?

Story

HandBrake had been nagging me for some time to install an update. I finally decided, for whatever reason, to do the update. There was a note in HandBrake’s update dialog that the incremental update was not available, and that I’d have to download an entirely fresh copy from their server. I didn’t think too much of this, as we’ve been in a similar situation with a broken Sparkle update channel once before (the worst).

So, I managed to download within the three day window during which the infection was unknown, managed to hit the one download mirror that was compromised, managed to run it and breeze right through an in-retrospect-sketchy authentication dialog, without stopping to wonder why HandBrake would need admin privileges, or why it would suddenly need them when it hadn’t before. I also likely bypassed the Gatekeeper warning without even thinking about it, because I run a handful of apps that are still not signed by their developers. And that was that, my Mac was completely, entirely compromised in 3 seconds or less.

By the time news broke of the HandBrake infection, git credentials had already been stolen from my Mac and used to clone several of our source code repositories, according to our logs.

As soon as I discovered the infection on my Mac, I disabled it, took the Mac out of commission, and we began the incredibly lengthy process of changing all of my passwords, rotating the relevant secret keys throughout our infrastructure, and so on, to re-lock our doors and hopefully prevent anything else from being stolen. The vast majority of these things were changed or rolled simply out of an abundance of caution — again, there’s no indication our web servers were compromised — but in this kind of a situation, you change all the locks.

Then, the forensics: we began combing through our logs to try to determine the extent of what was accessed which, to reiterate, we believe is limited to source code and personal data on my Mac. Thanks to good logging (thank you, James) we got a very complete picture. The method the attacker used prevented them from cloning all of our source code — they were making educated guesses at our repo names, one-by-one, which did not expose everything.

The source code theft was confirmed when we received an email from the attacker (with a few source code files attached as proof of the theft) demanding a large bitcoin ransom to prevent the release of the source code, which would “suffocate” our company, in their words. We’re working on the assumption that there’s no point in paying — the attacker has no reason to keep their end of the bargain.

And that brings us to today.

So…

When the dust settled, we sat down for a company all-hands meeting, and the conclusion was a little different than I originally expected.

Someone has a bunch of our source code. But does it really matter?

There are essentially three “worst case” scenarios we considered with our source being out there in somebody’s hands:

  • They build free, cracked version of our apps.
    Guess what — those already exist. You can already pirate our software if you want to pirate our software — but please don’t — so this doesn’t really change anything in that regard. Also, whatever “free” version of our apps that would come from this person are virtually guaranteed to be infected with malware.
  • They create malware-infected builds of our apps.
    This seems likely. Given the person’s entire MO was to infect a well-used Mac app with malware, it seems inevitable. But we will find them, and working directly with Apple, shut them down. To minimize your risk, never download a copy of one our apps from a source that is not us or the Mac App Store. We are going to be hyper-vigilant about the authenticity of downloads on our servers.
  • A competitor obtains this source to attempt to use it to their advantage in some way.
    The many Mac developers we’ve met over the years are fine, upstanding people. I can’t imagine any of them being this unethical, or even being willing to take the risk of us finding fingerprints of our code in theirs. And let’s not forget that — you guessed it — there’s a good chance any stolen source could have malware slipped into it.

Also, one important thought gave us some comfort:

With every day that passes, that stolen source code is more and more out-of-date.

This hack hasn’t slowed us down. That source is already missing a ton of fixes and improvements we committed over the last week alone, and six months from now it will be missing major critical new features. In short: it’s old and getting older.

At this point in our discussion, we even half-seriously considered releasing the source code ourselves — and when that idea was floated, and we realized there wouldn’t be any fallout (other than a lot of code questions!), that’s when we truly felt free.

Assistance

Within 24 hours of the hack, we were on the phone with two important teams: Apple and the FBI.

Apple rallied the right security people quickly to learn all they could about our situation. (They had, of course, already blocked the HandBrake-attached malware for the broader Mac population once it was discovered widely.) They walked us through the best way to roll our Developer ID and invalidate the old one, which we don’t think was leaked, but we’re being overly cautious. And more importantly, the right people at Apple are now standing by to quickly shut down any stolen/malware-infested versions of our apps that we may discover.

The FBI is actively investigating, so I can’t say anything more about that.

Together

We’ll be working overtime for the foreseeable future to keep an eye on this situation.

But we could also use your help.

If you see any cracked or otherwise unofficial versions of our apps in the wild, it’s safest to assume they are infected, and we ask that you please let us know. If you see our source show up somewhere, also let us know. And if you have information that could help with the investigation into this incident, definitely let us know.

The more we know, the more we can use every method available to us — legal, technical, you name it — to fix it.

Feel free to e-mail us or DM us on Twitter anytime — even if you just have questions. We’re here.

And as a reminder, never download one of our apps from a source that is not our website or the Mac App Store.

This has been a hard post to write. I hate that this happened. I kick myself every day for not paying attention to what I was doing; the tells were obvious in hindsight. It’s a good reminder though — no matter how experienced you might be with computers, you’re human, and mistakes are easily made. And even though this doesn’t affect our customers directly, we want to apologize that we’re even having to have this discussion with you.

We’ve been doing this 20 years because you keep us going every day — by buying our software, by giving us your good ideas, by telling your friends about us. You are the good in the world. So we’re going to do everything we can to rise above this and keep going even further — together.

Posted at 10:50 am 47 Comments

Sucks that you got pwned but then again like you said, it’s getting older plus you need the right developers to have Panic-quality apps. Is like meal, you can have the same ingredients but a good chef knows the right way to use them. Keep up the great work guys.

Ps. Looking forward to Transmit 5, my wallet ks ready for you guys

Not a daily Panic app user, but when I need to use FTP to access a server I still happily read for Transmit. It’s great that you’re being open and honest with the community about the breach. Also, thank you for not giving into the attacker’s demands. I think you are correct in that the worse case would be a buch of curious devs just wanting to see how things work. You may even get patches submitted for free a la open source efforts.

Cory Moll

5/17/2017 11:35 AM

This sucks to hear that the seemingly perfect storm of events occurred to create a potentially awful situation, which even I have fallen for at one time. However, the way it was handled – both with an internal conversation as well as external disclosure to the extent you provided, is incredibly commendable. I imagine it wasn’t easy publishing this, but the immediate communication and resolution affirms why I use a Mac and great apps like ones from Panic.

Adam Yanalunas

5/17/2017 11:41 AM

That’s rough, Steven. Sounds like everyone’s doing the best they can with the situation. Hopefully only good can come of this.

Well, it sucks but don’t worry, something like this happens to all of us once or twice in a lifetime.

Jeffrey Goldberg

5/17/2017 11:45 AM

Were the codesigning keys also compromised? Unless you are certain that they weren’t, you should get those revoked and reissued.

Is there something you think it could have helped to prevent this type of intrusion?. Even if you missed the obvious signals like authentication dialog and Gatekeeper warning, I’m wondering if there’s something that could have been done to at least make it harder for them to get your data.

Jeffrey Goldberg

5/17/2017 11:47 AM

Oh, never mind. You answer my question. (I should read carefully before commenting).

> They walked us through the best way to roll our Developer ID and invalidate the old one, which we don’t think was leaked, but we’re being overly cautious.

So, @Paul… people should be pretending to be perfect like you?

and you still live with yourself? Impressive.

Steven and Panic team, sucks this happened. Thanks for giving us the heads up. Look forward to more awesome products from you.

That’s totally awful. So sorry to hear about this. Perhaps it would not have prevented this completely but maybe a workflow where all employees update software via Munki fed by AutoPkg and tested before put in production would have helped. Or a similar setup with other software vendors and more controlled deployment rather than apps updating themselves and users being pestered with dialog boxes. Again, really sad to hear the news. Not fun. Love your apps.

Christian D.

5/17/2017 12:02 PM

Hi there,
thanks for the transparency. The worst thing in such a situation is obscuring and hiding so your open approach is certainly helping building trust. What about Panic releases in the last few days, i.e. i installed Transmit (iOS). Is this already code-audited? Am i safe?
Are external experts involved in forensics and the analysis of the incident, e.g. best practice review of procedures (helps with the transparency thingy)? I dont want to be too picky, but i have a separate VM for development; keeping the system at the same versions all the time.
In any way, so far i am a happy Panic user of a handful of applications and i wish you all the best.

Steven Fisher

5/17/2017 12:05 PM

As a developer (though not a competitor) if you guys had solved a small problem that was bugging me I’d be far more likely to email you and ask how you solved it than dig around stolen source code. My experience with Mac and iOS programmers is they’re generally happy to share little tidbits that are supporting rather than core to their product.

All the best, guys. Sounds like you’re doing the right thing.

Finno Furre

5/17/2017 12:06 PM

Had a belly-laugh thinking about how this sucks for the hackers, thanks!
You guys rock!

Finno Furre

5/17/2017 12:08 PM

Had a belly-laugh thinking about how this sucks for the hackers, thanks!
Talk about not living up to your name (no panic!)
You guys rock!

@Paul A.) I’m genuinely sorry that you work an environment where that’s the expectation. B.) Why would you assume Handbrake isn’t work-related? I use Handbrake for work near-daily.

@Steven Don’t beat yourself up. It’s like a car accident—bound to happen to even the best of people eventually. What counts is how you deal with it once it happens, and by all accounts you guys are doing an excellent job of that. Which is entirely unsurprising considering your track record.

I am genuinely delighted by how many recent ransom-ware demands in the news (Disney, Netflix, Panic) are being met with a shrug and a “Knock yourself out.”

@Christian As you can imagine we immediately looked at git activity logs closely and no code was ever checked in — only checked out. Furthermore, our iOS apps are served and signed by Apple, so no one can “replace” those binaries. We are confident that our currently shipping apps are safe and have not been touched by the attacker. (And at this moment, other than Apple and the FBI, external experts are not involved, but it’s not a bad idea!)

Luke Dennis

5/17/2017 12:18 PM

Thanks for being transparent. Happens to the best of us.

Not gonna lie though: if the source code is leaked, I look forward to indulging my curiosity to peek under the hood for its own sake! :D

Be strong in this difficult moment of life! All my force is with you and surely don’t panic !!
Best from France ;-)

You know… I agree. The best thing of a good piece of software are the people behind it that maintain it and support their users.

Christian D.

5/17/2017 12:35 PM

Thanks, if no code was committed that’s a relief! Again, good luck!
Cheers!

Giles Smith

5/17/2017 12:51 PM

I’m not very developer-savvy, so apologies if this has been answered in the post without me realising, but is it likely that anyone other than Steven will have an infected copy? It’s a fortunate coincidence that it was discovered in this way by someone in-house so quickly, presumably there may be other cases of it reaching the wild?

Kudos for the way this is being handled, the openness and manner of communication are exemplary!

Gregory Naçu

5/17/2017 12:55 PM

Oh my goodness, that’s terrifying. I know what it’s like to have physical property stolen… it’s very annoying! I can only imagine how much more annoying it must be to have secret intellectual property, like source code, stolen. Thieves are despicable, blackmailers are even worse. I am a daily Panic software user. I hope only the best outcome for us all.

Still love you and your apps!! 💐

Panicked

Justin Reese

5/17/2017 1:04 PM

The way y’all have handled this means I am more likely, not less, to remain a Panic customer after this incident. Y’all are awesome. Mistakes are inevitable; good resolution is not. You’ve not failed by being human, you’ve succeeded by doing the right thing. Up high.

Jay Williams

5/17/2017 1:09 PM

It’s honestly like this that makes me proud to be a Panic app user.

This is an amazing article. Ironic on a couple fronts: 1. Panic paniced 2. As a country / society we don’t expect transparency. (No one in politics, especially, would admit accidentally installing malware let alone using the wrong type of email account.)

Seriously, I think one take away for me every time I see an article like this is to do an inventory of my “Applications” folder. Look at the apps, and ask “Am I still using this?” & “If an update was available would I need it or use it?”

I’m not suggesting throwing apps that are seldom used, but definitely: get rid of apps you never used, the ones you tried once and forgot about … the ones you’ve replaced with alternatives. Occasionally I’ll look and see 80+ apps and I know I’m only using half — at most. If you delete an application you can usually go back and get it; but when it’s gone and you get a nefarious Update Notification (or a legit one) you can check your apps folder and see at a glance if the update is needed. If the app is gone, don’t update … which is exactly what the Apple App Store dos automatically.

There are a ton of indie apps & large corporate apps that don’t go through the Apple App Store … installing and updating these can seem easy & if you can go to the Security preference pane in your sleep and allow an exception you’re at risk.

I’m really glad Panic posted this; and glad they realized that their authentic software is not at risk. Hopefully they can develop an Application Inventory Manager app & an Application Distribution platform for developers to fall in love with …

Posts like this make me glad I’m a Panic customer (and have been one for quite some time). I believe a certain very smart person once said that we make mistakes so we can learn to make better mistakes later, and that’s absolutely the case here. I could have just as easily done this myself on a personal or work computer, and caused the same (or worse) havoc on myself and others.

You’re right that the Mac developer community is full of fine, upstanding people. Panic is one of the best examples of that.

Charles Wise

5/17/2017 2:07 PM

Something like Little Snitch or Hands Off might be useful to you. It will catch trojans when they call out. In this specific case you would have been safe since the trojan won’t run if Little Snitch is installed.

Boyd Waters

5/17/2017 2:14 PM

Panic has loyal customers, many of whom are software developers.

I am a customer, I think I am loyal, others might agree or not that I am a software developer. But I hang out where there seem to be many such people.

If we find unauthorized use of your code, we will report it.

Dean Perry

5/17/2017 2:29 PM

Out of curiosity what git service do you use? I guess you self host something as you say you checked the logs? GitLab or something else? :)

Loyd Craft

5/17/2017 3:32 PM

Might make you think twice about putting non essential apps on a development machine, or maybe start coding inside a VM (I know.a lot of devs who do this).. Sorry this happened to you..

Sorry to hear that.

It has been my hunch for a while that source code itself is not as valuable as the people who like it being closed source want us to think. How much genuinely new algorithms are in there? How many »business secrets«?

Hearing your stories (and other similar ones) does make me wonder, however, to which extent Apple is doing a good job. Despite the platform being quite locked down, they seem to have found no good way to make features that protect the user’s data from software work in a way that actually protects you if you want to use software beyond the App Store.

To a certain extend Mac users have »learned« to circumvent some security features / restrictions just to use their computers in a productive way. As a consequence warnings and dialogues aren’t always taken as seriously as they should be. Starts remining me like a small version of what Windows users »learned« in the past.

Thank you for being transparent and sharing your experience. Even the best make mistakes, but like some have pointed out, it is how you handle the situation. Kudos to the entire staff and working quickly to mitigate the situation. Awesome software and proud to be a customer. Best of luck!

Rod Speed

5/17/2017 4:04 PM

So for full disclosure what AntiVirus, and AntiMalware software do you run?

Gilbert Palau

5/17/2017 4:30 PM

Hi, very sorry that happened. What you guys are going through is not something you wish on anyone nowadays.

When I read the article in MacRumors.com I wondered how come they were able to access your usernames and passwords? Don’t you guys use Password Vaults like 1Password or {insert your favorite}? I would also like to know how do you think you guys will prevent something like this to happen again (dunno why on a mac with the Source Code to various important apps, you would need Handbrake, but… only you know that). In the end, it’s a terrible experience, but an experience that doesn’t kill you makes you stronger.

After all, the dust settles do you think you could write a blog post about the steps you took to secure yourself again and how others could avoid having a breach like this?

Wishing you all well.

Shaf Patel

5/17/2017 4:36 PM

Whatever you do, please do not give in to the criminals’ demands and pay them a ransom through Bitcoin.

Ross Lambert

5/17/2017 4:45 PM

I love the transparency, folks. And I think all of your conclusions are correct, especially the part about the source code being less valuable with each passing day.

Thanks for sharing, and good luck on the clean up.

== Ross ==

Excuse me while I go looking for the Audion source code ;)

Seriously though it is amazing the damage one little slip can do. I deal with malware all the time at my job (tech support at a university) and seeing the things some people fall for it is amazing that the phishers ever get anyone to bite. This one was a bit more clever though.

Steve Upton

5/17/2017 4:47 PM

Your transparency and honesty take courage and wisdom.

They also serve as a generous reminder and warning to all of us who could find ourselves in the same situation (Handbrake has been pestering me to update too)

Thanks and keep up the great work.

Allan MacKenzie-Graham

5/17/2017 5:27 PM

I use Transmit with an educational license – I work at UCLA. Thank you for that, BTW.
After reading your blog post I went and bought a copy of Firewatch. Thank you for being so forthcoming with us. It is especially eye-opening to see that even sophisticated computer users can be trapped by our own inattention. Thank you for great software, for the warning, and for the great game I am going to play now.
Cheers,
Allan

I downloaded Handbrake in the same window, but got lucky — no malware.

Super-impressed with the way this was handled at Panic. That’s gotta be a model for others. THANK YOU for sharing!

Source code (perhaps to an old version of an app?) would be interesting from a learning perspective… 8)

Been a customer/fan since Transit (no-m)…

Very well handled—thanks for being open. We own licences for Transmit and Coda 2 and have full confidence in you.

What if .. What if they use the code to study it and create the next malware attack. create a second component that can sniff all your apps. Find a way in and use your app as a forever gate into an os ? Or maybe better, using other techniques, they know how you write your registry and can store malware there, then every laptop that has the software installed can become a host.

@Cabel (unsure if @-highlighting surfaces messages better) – two thoughts.

Posting the leaked source, say, six months from now, could be kind of fun. You mentioned that timeframe yourself, so it’d be “only” as bad as you’ve described. It would be an interesting question to ask whether it would be worth it to release the code: on the one hand it would neuter the value of any copies being shared, on the other hand it would make sharing a lot easier. It’s not really win-win.

Here’s a possibly more interesting idea.

Take the entire leaked snapshot, and write a small script that, every week (?), leaks two or three 5-line excerpts from random locations in random files, onto an obscure page on this website (maybe edit one single link to that page from this article), preferably lines with long words (to surface identifiers or function names), and in such a way that it would be difficult to reconstruct anything useful out of the code (if the leaked code isn’t too large, it could be an idea to go through everything and manually delete certain portions first).

The reason for this, of course, is so that the content gets indexed – which might surface leaked copies of the source elsewhere. (Note that each new leak would need to be additive in order to remain indexed correctly – this is the one major challenge with this idea.)

I wish big corporations and major governments responded to breaches with the same transparency, humility, grace, intelligence, and relevance as Panic just did. The world would be a better place.

Panic Inc.

Required to post: Your real email address and a pleasant demeanor.
Do not post tech support questions. Email those!

pencil