I can see a lot of people trashing on Matrix.org or the "hacker" themselves (the hacker opened a series of issues, detailing how he managed to get in - https://github.com/matrix-org/matrix.org/issues/created_by/m...). However everyone seems to be missing the point - matrix seems like a pretty cool and open project. And someone taking over their infrastructure in such an open way is also great for the community. Even though a little dubious on the legal side of things, I believe it's great it was approached with transparency and a dose of humor.
Some might argue that this is harmful to matrix as a product and as a brand. But as long as there was no actual harm done and they react appropriately by taking infrastructure security seriously, it could play out well in the end for them. This whole ordeal could end up actually increase trust in the project, if they take swift steps to ensure that something like this does not happen again.
> Complete compromise could have been avoided if developers were prohibited from using ForwardAgent yes or not using -A in their SSH commands. The flaws with agent forwarding are well documented.
I use agent forwarding daily and had no idea it contained well known security holes. If that's the case, why is the feature available by default?
SSH agent forwarding makes your ssh-agent available to (possibly some subset of) hosts you SSH into. This is its purpose. Unfortunately, it also makes your ssh-agent available to (possibly some subset of) hosts you SSH into.
I never quite understand why there’s not a confirm version. ForwardWithConfirmation or something. I’m active when I need forwarding - would be happy to simply be prompted before it’s allowed.
Some ssh agent implementations do this, notably the one built into Android ConnectBot can be configured to request confirmation each time it is asked to authenticate. Unfortunately ssh-agent (from OpenSSH) does not as far as I know. It's happy to authenticate as many times as requested without any notification.
Use separate keyboard-interactive 2FA (I recommend google-authenticator) for production ssh access.
Use a key system which requires confirmation or a PIN to authenticate (such as a Yubikey). Use a persisting ssh connection with Ansible (ControlPersist) to avoid unnecessary multiple authentications.
Allow connections only from whitelisted IPs, or Uuse port knocking to open temporary holes in your firewall, or require connections to production infrastructure to go through a VPN.
Access production infrastructure from hardware dedicated for that purpose, never do anything else on it.
I wish there was a way in ssh to tag connections and only allow agent forwarding to keys with the same tag. That would prevent agent forwarding production keys from a dev host.
I'm not sure. A secure, backwards-compatible (with older servers) alternative, which only exposes keys you explicitly choose to expose, should be doable and might help.
another option would be for a SSH client to present a full-screen "$HOST is trying to use your your SSH PRIVATE keys. Press enter, then type "~A" to allow." prompt.
I know. That's why I asked. Chained agent forwarding will serve your keys just the same, so ProxyJump is not "a secure alternative that achieves the same outcome".
Are you disagreeing with the "secure alternative" or the "same outcome"? I thought the difference between ProxyJump and agent forwarding is the following:
Agent forwarding forwards the agent socket to the proxy server. Thus any ssh connection originating from the proxy server can reuse the agent, and with that has the same access to the agent as the originating host.
ProxyJump routes the ssh connection through the proxy host. The crypto takes place between originating host and target host, not between proxy host and target host. ssh connections originating from the proxy host can not access keys from the originating host.
But maybe my understanding of ProxyJump is incorrect?
> ProxyJump proxies your ssh connection, so connecting from A to B via proxy X the connections go A->X and X->B.
No, it rather works like this:
A -> B via X establishes A->X and then, through that connection tunnels a new ssh-connection from A->B.
A->X, then X->B would require forwarding the Agent from A to X, so that the connection from X->B can authenticate using that agent. Proxying the connection does not require X to ever authenticate to B, the authentication happens straight from A->B (1). Thus, no agent (forwarding) needed. You can also chain ProxyJumps: A->X->Y->B tunnels A->B in A->Y which is then tunneled through A->X. In that regard, ProxyJump and ProxyCommand can replace AgentForwarding in most use cases. There are some uses where AgentForwarding is the only solution, though.
(1) Added benefit: X never sees the actual traffic in unencrypted form and all port forwards A<->B work
I think an issue here is we've been told for a long time "always restrict access to the environment through a bastion host" without much implementation detail discussed after that. Agent forwarding tends to show up as the most efficient way to implement this.
Or the hacker could have responsibly disclosed the issue to Matrix then reported their findings like a professional. Besides, we're still defacing web sites? I though that went out of style years ago. Did the hacker make sure to post that on MySpace too?
Responsible disclosure is about not enabling third parties to leverage the disclosure to gain access. In this case the hacker did not disclose the security holes before they were closed for third parties (i.e. the hacker could only still access the hosts because he had access to the them in the past, new access was (hopefully) not possible anymore).
Which of course doesn't mean that the hacker should have just send an email to the matrix team.
> The matrix.org homeserver has been rebuilt and is running securely;
We should have more bounties. Let users donate and put wallets on servers. Attacker will be able to take these funds. It's a reasonable measure of an infrastructure security.
To avoid perverse incentives, you should also build in some reward for the developers/operators. As in: If the server gets hacked, the money goes to the whitehat. If the server does not get hacked for $TIMEFRAME, the money goes to the people responsible for its security.
Also, there is a requirement for the hacker to actually publish the results of how they did it. Otherwise, you run the risk of the hacker just walking away with the funds or giving a bogus reason (after they've already spent the wallet).
Therefore, the wallets should be stored GPG encrypted in some published location. After the hacker has successfully penetrated and retrieved the file, they need to publish a "how I did it" document along with the hash of the GPG encrypted wallet.
Once devs have confirmed the vulnerabilities exist, they respond with the passphrase to decrypt the wallet.
My idea was to not require any explanations, so that blackhat could grab that wallet too. It's just about being able to say "this server is $1k secure". I think it's fantastic that we have a technology to do that.
You still need some trust that private keys to given wallet are on the server, but apart from that, when you know there's $10,000 dollars on the server for anybody who can access it, it says something about how secure this machine is.
Plus you get instant notification when the server is compromised. Not every hacker is kind enough to let you know.
Seems perverse to me as well. Might be a better idea to just fund Matrix enough to be able to have at least someone full time on it. With $3 752 per month on Patreon right now I cannot imagine it's a lot after infrastructure costs and taxes. Certainly not enough to let Arathorn or someone go out of his way to get expensive security training.
checking in as an internet weird person here, any time you have a platform that synthesizes anonymity with collaboration/social interaction, weirdos like us are gonna pop out like clockwork because we find a safe haven for our, uh, weird stuff. a place to not be judged or whatever. i think the sjw type terminology for it is a safe space. and of course due to the human element being so easily corruptible, many people do also tend to use such things for illicit purposes.
I would like for matrix protocol and implementation to be better prepared for such cases.
While I didn't loose access to the encrypted messages, since I used the 'Encrypted Messages Recovery' function of Riot.im, I guess a lot of people have. Maybe allow to store more information on the client side?
I do not really like the fact that this feature can only backup keys server-side, so I did not enable it.
I do however have a keys backup dating back some time, that will hopefully restore some of my encrypted messages. But basically, I understand that every encrypted message was at risk of being lost, so it's not that big of a deal.
there are quite a couple of so called guides (opsec playbooks for crime) that I found specifically on Wall Street Market (a darknet market place like the now defunct Silkroad), available for purchase.
Some of them go beyond just instruction booklets but promise access to their chat systems via invitation (upon purchase of the pdf) and offer some kind of limited coaching. It is essentially the recruiting mechanism to bring in lower ranking soldiers starting out as mules, handlers, or basically move up from re-selling goods.
A couple of these guides point out how much Telegram sucks etc, and that they now have moved to p2p based systems. One praised Matrix heavily for it's good security feature.
The tech-savvy-ness of many vendors has picked up considerably since I first started watching. There is a strong push to re-think and refactor both tools and their processes (yes yes - this happens constantly otherwise they get caught, but never as fast or aggressive than these past months).
It's likely that this is just a (s)kiddy enjoying the attention. Though quite a lot of players have more than just an "academic" desire to ensure these (their) systems can withstand an attack by LE. When I browsed the matrix issues on github I couldn't help but immediately recall the strange emphasis on "we have switched to matrix". It's far fetched but I'd say somebody may have a strong interest in seeing these issues resolved (->or has gotten genuinely fed up and wanted to do something, as opposed to this being just a skid that only did it for the attention)
for a good analysis on how some of these tutorials and the philosophy behind them see: "Discovering credit card fraud methods in online tutorials"
Matrix operational security is a joke and developers understanding of security is a joke. This is 2019, not 1992.
Infrastructure with ssh access without hole punching for currently active authorized connections only? Decrypted signing keys accessible over the network? CI servers and developers having root access?
Though the "we had to revoke all the keys so you lost access to your encrypted messages unless you backed them up" takes the cake.
> "we had to revoke all the keys so you lost access to your encrypted messages unless you backed them up" takes the cake
This is just how it works. It's been well documented and mobile clients got updates that backs up the keys automatically. It's also effectively the same as WhatsApp and some other IMs (they just don't even save your encrypted messages). Either way - backup, or lose your history.
When it comes to criticism about backup I have no problems with things getting wiped from the server. I assume a good p2p design has a "little server" and as much client as possible in it.
Enforcing in the clients to properly back up by default, or otherwise properly educating the user of what happens if they don't back-up would be as important as getting the code right. There is little difference to the user whether they lost data because they didn't understand they really had to do backups, or they got their keys compromised and messages deleted by a malicious 3rd party.
I stand by the assertion that it indicates the Matrix people are clueless.
If this is a design constraint, then the security model needs to accommodate that the user keys are the pot of gold, which means that there needs to be a service provided by a dedicated server which is inaccessible in the course of normal operation via any means other than a well defined braindead simple protocol <keyid>:<command>:<message> providing the message manipulations/key store functions from only other authorized production hosts that need to be able to access this functionality.
The server running the service should have a security policy that would prevent one from running any software that is not supposed to be already present on a server ( use SELinux enforcement policy ) to minimize the attack surface; have its own SSH keys not generally accessible during the normal operation, be accessible only from specific IP addresses, etc etc etc. If it is on AWS, it should probably be in a separate account.
I think you misunderstand why the keys were deleted. The keys get deleted on the client when you log out. This is sensible, because if you log out on a device, you probably don't want to keep the keys around in your browser storage. When the users session is destructed on the server, existing clients get a 403 error and told that their session is logged out. When that happens, they go through the normal logout routine which involves deleting the keys on disk.
Deleting the keys isn't something the matrix.org folks explicitly had to do because of the compromise; it's simply how the riot.im client reacts when you terminate it's session.
It will be updated shortly to reflect the DNS defacement linked here (which was because we failed to rotate a leaked cloudflare API token; we aimed to rotate the master API token but rotated a personal one instead). To our knowledge the rebuilt production infrastructure itself is secure.
We've revoked the compromised GPG keys, and are obviously going to do everything we can to improve our production security to avoid a recurrence in future.
We can only apologise to everyone caught in the crossfire of this incident.
The lame version of port knocking that solves 99.9% of issues.
1. Default policy for access to all of the development environment is deny all.
2. A developer triggers a temporary addition of developers current address to the allow list with an idle timer, punching a hole for developer's edge IP to access the infrastructure.
3. When the idle timer expires or when the developer says "i'm done", the allow rule is removed.
Obviously, a full blown port knocking with keys and policies would be better for a large organization with hundreds of developers and hundreds of hosts but it is the case where 99.9% of the issues can be solved using a very simple system as in order to get to the vulnerable entry point the attacker would need to do it from an IP address used by a developer at that specific time.
IP whitelisting and port knocking are not serious security methods. They're the very-poor-man's version of a VPN and access control policies, and they're not secure.
You are talking about organizations that have GPG private keys used for signing laying around and those that have Jenkins exposed to the outside world.
Dynamic IP white listing and port knocking are perfectly adequate for 99.9% of the organizations.
“Anyways, that's all for now. I hope this series of issues has given you some good ideas for how to prevent this level of compromise in the future. Security doesn't work retroactively, but I believe in you and I think you'll come back from this even stronger than before.
Or at least, I hope so -- My own information is in this user table.”
I think it boils down to the fact that infrastructure for projects (no matter the size) is usually a second class citizen at best.
Either no one is eager to care for it, or the people who are actually focused on developing the software run it because they need to, or worst case - no contributor is trusted enough to handle infrastructure work, with access being given even more sparsely than commit rights to the whole software. Which is fine by itself, but there are so many (big) projects where infra is kind of terrible because 3 out 100 people involved are doing all the work. Or don't.
It's just a problem with the industry that's like "let's just get something out there and achieve product market first..will worry about infra and security later" that later is just pushed into a backlog and forgotten.
As a classically trained sysadmin I've seen this trend going for the last decade.
Some developers seem to be pushing that ops shouldn't exist any longer or should be outsourced to google (who don't hire ops) or amazon (who do).
Managers see this trend and think that hiring only developers is a good way to save costs and do things the "new way".
Traditional ops roles are indeed not as required but security/process/reliability focused people should not be the same people who write new features. They're in contradiction of each other often.
If you're a developer who thinks ops shouldn't exist any longer consider this:
I can write software and design websites as a sysadmin, does that mean I don't need you now, or that I know everything you do?
I argue that it doesn't. A focus on automation is one thing but defenestrating the notion of operations/SRE is going to net you a bad time.
> I noticed in your blog post that you were talking about doing a postmortem and steps you need to take. As someone who is intimately familiar with your entire infrastructure, I thought I could help you out.
> There I was, just going about my business, looking for ways I could get higher levels of access and explore your network more, when I stumbled across GPG keys that were used for signing your debian packages. It gave me many nefarious ideas. I would recommend that you don't keep any signing keys on production hosts, and instead do all of your signing in a secure environment.
RRREEEEEEEE> I noticed you missed a doctype in your html page. In order for web browsers to know what type of html to render you should include a doctype. Thanks!
matrixnotorg> @RRREEEEEEEE Thank you, I will consider that for the next release
I have been asked twice or more why I insisted on not using a Continuous Integration environment for publishing some software releases that are installed by third-parties.
My team was automating the infrastructure to build internal software and naturally they wanted to be able to simplify things.
The idea that was proposed to me was the following: once I push a new version tag to GitHub, the deployment CI server is going to build and release it as an unstable version.
Some important detail here: I use the same key to sign packages regardless if they are released as unstable or stable. That would mean that if someone, somehow, managed to push a tag that was pushed upstream to GitHub, hypothetically they would be able to eventually gain access to consumers machines (basically, developers) when the consumers update it after getting a notification telling them a new version is available. No way I'd allow this to happen, but I would not be surprised if most people just took this as an acceptable risk.
Depending on your threat model I think that signing packages directly from your CI is acceptable, assuming that your CI runs is a reasonably isolated environment (e.g. on your company's LAN) and people who are able to trigger a release are correctly vetted.
If I understand the parent comment correctly they were somehow shipping the release signing key on their production environment which is a whole other level of bad.
> That would mean that if someone, somehow, managed to push a tag that was pushed upstream to GitHub
You have to define what the signature means.
IMHO it is fine for it to mean "this software was built on our build server from a well-defined state of the source code, which is only changable by our employees and contractors, and for which we have the full change log". So I deploy the code signing key to build servers, which is the only place where it is used.
I'm interested in what alternative meaning you would give to a signature. I have considered the possibility of tying it to the QA processes, but then a build can only be signed after checking it manually, which is problematic when many signatures are needed at multiple packaging layers (exe/dll, msi, setup.exe).
The problem is when a malicious package is produced, either because a flaw was introduced in the code, or because a dev machine was compromised, or _when the CI machine sad compromised_; the malicious package will be signed as if it were legit.
One middle point between automated and manual signing is, as usual, key rotation: have the signing keys expire in a short duration of time (say 2 weeks) and manually push them every week, so that the window of attack is as small as possible.
2. If tests pass, the CI sends a notification to the build server. The notification is "prep for release package <hashid>"
3. Build server pulls code from the repo, matches it against "ready, CI passed" notification and builds the package/packages.
Compromise of the entire CI/dev chain would be contained as the builders act as a new pipeline entry point running in parallel of the CI using pull method. To compromise keys located on a build server one would need to either get access to it via whatever the method of remote access the server has ( which should be nearly none ) or figure out how to compromise the code running on the builders using the input from a repo that passed CI.
> Escalation could have been avoided if developers only had the access they absolutely required and did not have root access to all of the servers. I would like to take a moment to thank whichever developer forwarded their agent to Flywheel.
I'd feel so small if I were this developer right now :-|
A couple of his issues appear to have to do with the use of SSH. An Ops-guy whom I worked with had setup a bastion host with ip whitelisting that automatically shut down after 1 hour. He didn't like it as his credo was "if you're using SSH when using a cloud provider you're probably doing something wrong"; meaning to say you should automate and be able to recreate any infra at all times with logs accessible without the need for SSH. I never forgot that.
I have just checked quickly the comments and post-mortem, and I start wondering - it seems that the attack itself would be not really possible if Matrix would not be open source (as this would restrict access to the sensitive data)? Is that right?
This is not right at all, the hack was due to an outdated Jenkins instance and could have happened regardless of what other software was running on the infrastructure.
Absolutely not. There's a zero reason for development infrastructure ( which includes Jenkins ) to have any connection to production outside a well defined "transfer this tarball" to a deploy staging server path during a known deploy window which cannot be controlled via development credentials.
I am highly skeptical when people taking about "rebuilding [the whole] infrastructure" in a few hours. Even more so when restoring all data from breached systems and before a thorough incident analysis. Show me the org which can just pull that off.
It's quite common in enterprise to have a RTO of a few hours, and RPO of a few minutes, even for infrastructure with terrabytes of data. Of course, many moneys are paid for being able to do that.
They didn't host production and the website on the same server. The attacker had access to the whole network. After that was detected and cleared, the attacker was still able to change DNS records. The domain was redirected to an attacker controlled site (https://github.com/matrixnotorg/matrixnotorg.github.io) where some logs of the production servers where posted.
The interesting thing for me here is that none of the other homeservers were affected. Despite the weak security on the largest servers, the ecosystem stays alive.
I might be misreading it, but it seems that the issues you are pointing out related to 3PID, which is still somewhat centralized. Sure, work on this needs to be done yet the system is evolving to be more independent.
If I paid monthly for a server and then spent time configuring it, now I would be able to talk to people who pay monthly for their servers and spent time configuring them.
Alternatively, if you paid monthly to any hosting provider, you could be talking with customers of any other hosting providers.
And if there were any other hosting providers that came doing such shoddy things in their production systems, they would be wiped out of the ecosystem, but the ecosystem would still be alive.
I'm probably really out of the loop, but what is matrix.org? Looks like an open source slack clone? Why do they have >5 million user accounts? Is that everybody who uses that chat tool?
It describes not just Matrix in the French state (like in the title), it also covers the Matrix 1.0 release and what they want to do with the project (e.g. they eventually want to shut down matrix.org once the ecosystem is mature).
Unfortunately I don't have any background context for possible reasons why "actual transparency" on the top line is the issue chosen by the attacker, but makes it seem ideologically driven.
Seems more like a way of showing "I got access to 5493973 passwords and to show that, instead of picking some random users, I'll pick the one responsible for the shoddy security".
Or it might be a clear, concise way of showing that he has access to the entire file without disclosing the information of random users, which also happens to be a particularly short command.
As for motivation I don't know, but I would like to state that I have followed the Matrix project for some time and have found it, and the CEO, to be transparent.
But even if they didn't, "sudo -u root /bin/bash" or similar gives it to you unless sudo is extremely locked down (which, from audits I have done, is "rarely if ever").
There are hundreds of ways to get root prompt even with the root account nominally deactivated.
TL;DR: A collection of inadvertences and suboptimal practices, some (like having GPG signing keys on production systems) more worrying than others. Something that could probably have happened to most orgs without dedicated security resources.
I've been slightly annoyed with matrix ever since they boasted at FOSDEM with the fact that they backdoored their encryption so that the French government could Virus scan sent files. :/
You can use Bash (and I assumed it works in zsh too) variables to save you there. eg
grep something !$
`!$` will be replaced with the last parameter of the previous command. Crude example
cat /etc/hosts
grep 12.34.56.78 $! # will be executed as grep 12.34.56.78 /etc/hosts
I find this extremely useful for when I'm cat'ing a file to get a sense of it's output then wanting to do something more meaningful with it in the next command.
That all said, I have absent-mindedly done my fair share of stuff like this too:
!! | less
# eg will run as `cat /etc/hosts | less`
# if `cat /etc/hosts` was the previous command run
so I certainly wouldn't look down on people who have "abused" cat due to muscle memory.
Thanks, I wasn't aware of that. However it's a different hotkey and only pulls the last parameter. The shell I use, you could hit <alt>+<number> and you would get a parameter of that number completed - not just the last parameter. It was very handy for rebuilding long command lines with different arguments.
> Or you can check “Profiles → Keyboard → Use Option as Meta” for Terminal.app [or just press ⌥⌘O]. And then use option as meta.
I use iTerm2 - which does seem to have similar options but I'm yet to get it working. I know that's down to user error but it's still a real pity that it isn't just the default behaviour (in either iTerm or Terminal).
As someone who doesn't care, my eyes bleed every time someone tries to make others feel small for no reason using improper grammar:
"As na linux/unix sysadmin": typo ('na') aside it is 'As a linux/unix sysadmin'
"...my eyes are bleeding everytime I see" I think you meant 'my eyes bleed', otherwise it means that, coincidentally, you eyes were already bleeding every time you happen to see someone use cat in that way.
"everytime" is wrong, the correct form is "every time"
My point isn't to insult you, it is to show you that everyone has blind spots and we shouldn't give each other such a hard time.
Grammar and spelling blind spots don't compromise the security of thousands of people. I think if you're going to make such glaring mistakes, you should be able to take a bit of guff for it.
I do cat x | grep y, because that way you separate out the primary data being passed around and the secondary instructions for how to process it. Preferring functional programming, this is my bread and butter. It´s superior readability and simplicity is something that gets engraved on the inside of your mind after you do a pipe a few hundred times per day every day. This is not about being terse, terseness is almost never a factor.
The simple principle that emerges is that the first line is for input, not for applying functions. You might guess that someone who does this even in a pipe as simple as cat x | grep y has this muscle memory too, but someone who criticizes it, certainly doesn´t. Maybe that´s not a bad thing, but you do feel a large divide between each other.
There are several benefits to it. "cat filename" is a natural starting point for building pipelines interactively one step at a time. It puts the first input on the left side of the command that consumes it. (The alternative of "cmd1 [<]filename | cmd2 ..." does not quite read left-to-right, and some shells do not support "<filename cmd1 | cmd2 ..."; for example, fish doesn't.) When you are done with preliminary testing and want to run the pipeline on a larger amount of data "cat" is quicker to replace with "pv" (and vice versa). You do incur a performance cost with both "cat" and "pv". The command consuming the input also can't seek on it with "cat", but it doesn't matter if you are processing the input one record at a time. If you like the ergonomic benefits, use "cat" until you are optimizing for performance.
As a sysadmin with 20+ years of experience, I always type `cat | grep`, completely involuntary, probably because "cat = read" is just burned into me, by the time I think about it the command is already written. Also maybe related to catting more than grepping :D
1 - you want to add more filtering/processing before the grep
2 - grep's command line options are confusing (+ globbing + whatever), easier to just use it to grep stdin
3 - It works. Sure 'grep pattern file' works, but here that is inconsequential. I'm not in an 80s machine to worry if I'm opening one more process or pipe than needed, especially in simple cases like this
I have no issue with people who want to prefix grep (nor any other command for that matter) with cat. However I do completely disagree with your 2nd point. It's literally just:
grep [flags] search_pattern [filename]
In GNU grep (eg Linux) it's even easier because your flags can appear wherever in the command you want (even at the end). Though I'd recommend sticking to the habit of having them after the executable name for readability and portability with non-GNU coreutils.
It's really not that hard. There's plenty worse CLI tools to use which we're now stuck with because of historic reasons.
That first line is more or less exactly what I posted, is your typical usage of grep, and is very easy to learn.
Sure, you can list of esoteric examples of grep usage but that's besides the point if it's not how people would typically use grep (in my ~25 years of command line usage, I can't even remember one occasion when I've needed `-f` - not saying it hasn't happened but it certainly isn't something I've needed regularly)
Really ? That's a bold claim. Can this string be hacked ? $6$WqQvgIwr$/BA/19FRJZf.z4buUwX5Sbls07ovv/lVZJ3V2En7VgTR4Skdpz98hFNuq3VX4nIaiSDKObO9jKbkPb5tdt7zY1
Doesn't surprise me that much, Matrix doesn't seem to be too concerned with security, more with security theatre (considering you can still not easily disable read receipts in your client, a major privacy leak IMO, among other issues).
Maybe you should just leave this mentality of "if said thing disapproves of somoene's business, the person saying the thing must be opposed to someone in person." behind.
Because with this mentality of yours means you have a personal dispute with me. Better disclose that before saying such things.
I never talked to you, unless you are the same guy who "operates the single federated independent matrix server" and who does have a personal dispute with matrix.
I've not edited my post in the way you describe, or in any way comparable. All edits I perform either do not change what I wrote (spelling) or I very clearly append it to the post with an "edit:" prefix.
I've not written that I operate "the" matrix server.
There is plenty of other issues with matrix and the reference clients on top of something as simple as mandatory leaking of your presence in a chatroom. I've run a matrix homeserver for almost 3 weeks and it as an utter pain to maintain, despite not a single version upgrade and I was plagued with issues that no chat platform would have if the protocol was remotely sane.
edit: That is on top of the numerous security issues this hack uncovered. Apparently the matrix.org devs kept a users.txt file with a dump of users + passwords on the server. Signing keys for debian packages were stored unencrypted on the production server. People used unsafe SSH settings (SSH Agent Forwarding), ran outdated servers with known root-priv RCEs for months and root privileges for all users on a server. Why should I ever trust a matrix developer with their protocol or reference implementations ever again if they can't be trusted with the simple task of updating a service when a critical CVE comes out?
> I've run a matrix homeserver for almost 3 weeks and it as an utter pain to maintain, despite not a single version upgrade
As a counterpoint, I've been running a Synapse (a Matrix homeserver) for about 1.5 years now and it's been smooth sailing throughout, including the frequent upgrades. Maybe it's different at a larger scale (my userbase is 5-10 users), but if, as you say, you did it for three weeks, I guess you didn't have magnitudes more users than I have.
I've had 12 users I brought over from my mastodon instance. I've received multiple complaints about matrix from each one, half of them stopped using it after a week over the privacy and usability concerns.
While you bring up valid concerns about the Matrix team's security hygiene, the point of an open standard is that anyone can (try to) spot flaws in it, and anyone can (try to) create their own implementation.
I myself am waiting for a healthy ecosystem of servers and clients to spring up before starting to rely on Matrix for anything non-ephemeral - even if it takes years. Perhaps I'll even try my hand at writing a client, if I ever run out of things to do. In the meantime, I will dick around with a throwaway matrix.org account to play with it, and to watch progress happen.
> myself am waiting for a healthy ecosystem of servers and clients to spring up before starting to rely on Matrix
Good luck with that. Right now there's only the centralized matrix.org server, or actually there isn't because it's down. If you want open standards and multiple servers (or your own) use XMPP period.
It's not so much a technical question as it is the attitude of "hey we're implementing our own chat protocol cause XML sucks". Totally not getting the point why users and developers would want to use standard protocols - to save their efforts becoming obsolete, taken over by a single entity, or both. It doesn't help either that scarce development resources are needlessly fragmented between XMPP and matrix.
That said, if the matrix protocol can actually manage to attract users and multiple implementations some years down the road (about 30-40 years after IRC), more power to them.
> It doesn't help either that scarce development resources are needlessly fragmented between XMPP and matrix.
In my experience, there's virtually no overlap between the two groups, and therefore no fragmentation. And for good reason: XMPP is a nightmare to implement, so there's a significant group of developers that just won't touch it, but that might be interested in working on Matrix.
And yes, part of the blame for that lies in the usage of XML. While XML can be useful to represent complex data or documents, it's unsuitable as an over-the-wire format because it doesn't have a directly mappable representation in most languages, due to the combination of attributes and child nodes.
This problem doesn't exist for JSON, because pretty much every language directly supports arrays, objects/maps and primitives. This makes a JSON-based protocol much more pleasant to work with, as there is less data-wrangling complexity involved.
A chat log is rich text with emojis, photos/videos and other binary data, memes, rich inline citations, block quotes, attachments, and endless new formatting practices and gimicks that are unknown yet as digital communication evolves. XML/SGML is made for this kind of application.
End users don't throw arrays and maps at each other on chat, so JSON's affinity to co-inductive datastructures of programming languages (actually just JavaScript) doesn't help all that much when you have to invent ad-hoc markup over arrays, maps, and primitive types, or ad-hoc hex string encodings of binary data. TBH flocking to JSON because JavaScript can represent it as an object literal is a pretty junior attitude and reflects poorly on the matrix effort. It's equivalent to using BASIC or notepad.exe because that's what's installed on a computer out of the box.
> This problem doesn't exist for JSON, because pretty much every language directly supports arrays, objects/maps and primitives
No, JSON will not map directly to a language with advanced type system (with tuples, variants, etc). Even in Elm it's recommended to write a decoder to convert incoming JSON into an internal structure. So in fact the mapping is very poor. And I see no difference in this regard: both XML and JSON is crap.
I can't wait years. I need to pick up a definitive platform right now to push as an alternative to proprietary ones. It would suck to migrate all my friends to something just to ask them to move again to something else a couple years later.
I would agree with that. We need something that works now, not when someone finally manages to reign in the Matrix protocol.As it stands I cannot send a friend an invite to Matrix and expect them to like it one bit (which turns out, is what reality looks like).
Some might argue that this is harmful to matrix as a product and as a brand. But as long as there was no actual harm done and they react appropriately by taking infrastructure security seriously, it could play out well in the end for them. This whole ordeal could end up actually increase trust in the project, if they take swift steps to ensure that something like this does not happen again.
reply