we're starting a policy and regulation blog series over on the Foundation's blog. Over the next few months I'll be covering various pieces of legislation that are already in place, as well as incoming regulation, and what it all means for Matrix.
The new FluffyChat has several performance improvements, including a fix which should speed up the whole app if you have a lot of megolm sessions.
Also FluffyChat v1.21.0 includes a new search and gallery feature for chats. You can now search for specific messages or browse the shared photos in a conversation.
Also please note that the default network request timeout has been changed. Before FluffyChat ran into a timeout if the server needed more than 30 seconds to send the initial sync. This was probably too optimistic so the timeout has now been set to 30 minutes. This should allow much more users with very large accounts to log in. Feel free to share your feedback.
All Changes
feat: Enable download images on iOS, not only share images (krille-chan)
feat: Search feature (krille-chan)
build: Update record package (krille-chan)
build: Use correct pubspec.yaml format for hosted dependency (krille-chan)
build: Use matrix sdk main branch (krille-chan)
chore: Change default timeout to 30 min (krille-chan)
QR Code login works is completed we are just waiting for matrix server to fully support native OIDC login, before rolling out the feature
Message queue work is also almost completed, now messages will be automatically queued for resending when internet connection comes back, without the need of taking any manual action
We are also working on storing and restoring composer drafts for each room, so no need to worry about losing anything that has not been sent yet if you browse another room or close your app
Great news also for element call integration, we are implementing native call notifications (with ringing), and a call log for DMs. Also starting a call will also be displayed as an event in the timeline.
We have added a quick implementation of sharing, so you can now send text and media from other applications.
QR Code login works is completed we are just waiting for matrix server to fully support native OIDC login, before rolling out the feature
Message queue work has been started, messages will be automatically queued for resending when internet connection comes back, without the need of taking any manual action.
Great news also for element call integration, we are implementing native call notifications (with ringing). Also starting a call will also be displayed as an event in the timeline.
Way back in December I TWIM'd I was working on a new project called complement-crypto which aims to write an extenstive, exhaustive set of end-to-end E2EE tests for Matrix clients. The aim of this work is to ensure encryption in Matrix Rust SDK in particular is robust to all kinds of failure modes: from connectivity blips over federation, server restarts and corrupted data, to actively malicious homeservers. Work has progressed significantly in the intervening 6 months:
The test suite runs in both Rust SDK and JS SDK CI pipelines to detect regressions. This required a lot of work enabling conditional compilation (so you don't need to run JS code in rust SDK) and improving test reliability to a sufficient standard that it can be relied upon.
It now has full federation support, spinning up sliding sync proxies for both homeservers if needed.
It has a comprehensive RPC client process to test ungraceful shutdowns (e.g what happens if your client gets SIGKILL'd?)
When things do go wrong, you have powerful debugging tools thanks to using mitmproxy for all communications, in addition to client log files and container logs. The use of mitmproxy is not only useful for debugging, but is also used as a test synchronisation primitive to reduce flakiness (e.g do X when you see the HTTP response to Y), as an assertion tool (e.g ensure the client has uploaded one-time keys by looking for the request), and as an adversarial tool (e.g modify the response returned to the client).
it has allowed us to test the SDKs to reproduce tricky "unable to decrypt" (UTD) scenarios which we couldn't otherwise replicate. The NSE bug was reproduced via a complement-crypto test which semi-randomly performed high-level operations like restarting clients and sending messages in a multiprocess environment to try to trip up the SDK code.
it has allowed us to write regression tests for UTDs such as:
adding targets beyond rust/JS SDK (matrix-bot-sdk, mautrix, and others). A perfect client can still receive UTDs if the sending client didn't encrypt correctly, and the sender may be on a different codebase.
This has been just one part of our work reducing UTD errors. Another significant part has been aggressively analysing bug reports in the wild to identify common failure modes. To that end, please report any 'unable to decrypt' messages you see and include log files with your report. Please also try to get the sender to submit a bug report. I may reach out to you if I need additional information. These bug reports not only guide our priorities, but have also helped us identify obscure failure modes which we cannot reproduce via complement-crypto.
Some minor changes, this week's update will be mostly project/management updates!
Call for help! (Update)
It's been decided to split the event rewrite up into multiple phases/parts. The original vision of the changes turned out to be far too optimistic, and we've run into some language-related challenges trying to implement them.
We'll still try to implement those changes, but this might depend on compile-time source generation or similar endeavours.
If this sounds like a fun challenge to you, or want to discuss and shape the semantics of these changes, feel free to join in!
Changes (Projects & Policies)
Some policy changes, and some new projects have been started within the LibMatrix source tree:
New workflow: Use of feature branches
Keeping up to date with LibMatrix is hard, because the chance of updating to a partially finished commit is too high.
To address this, we're moving to a "main branch is stable" model, with partial implementations on deciated branches so developers can keep up and prepare for those changes.
New subproject: Writing of full documentation
We want to improve the developer experience by offering documentation of features right in your editor, without having to reference the backing source. This should enable users who are less familiar with Matrix' internal structures to still be able to implement their deepest wishes.
New subproject: Workspace and structural cleanup
The LibMatrix source tree became a mess as the project grew. We want to restructure it to be maintainable, and also far more friendly to those who might want to contribute. If this sounds like something you'd like to help with or would love to offer feedback on, please do feel free to reach out at #libmatrix:rory.gay!
Splitting up of subproject: Event handling rewrite
Due to the large amount of complexity and work, we've decided to first start rewriting the inner implementations of events.
This should massively improve the end user experience, as custom event content is no longer lost if edited, as well as no longer having to worry about desyncs between actual event content and what you might see in an editor!
Next to this, you will likely see performance improvements when dealing with accessing the contents of events!
Longer-term changes still include static typing of event collections, especially around single content type collections such as fetching a room's member list, which should also see a large performance uplift when fetching!
Changes (Main release)
All dependencies have been updated. End users should not notice any differences as this is a maintenance task, but this does potentially enable some new usecases and squashes some bugs and security risks originating from outdated dependencies.
We have also expanded our team, in particular on the community growth side, and are about to open another position for a product designer ... if that's you or someone you know, stay tuned!
As of today, 9397 Matrix federateable servers have been discovered by matrixrooms.info, 2860 (30.4%) of them are publishing their rooms directory over federation.
The published directories contain 160776 rooms.
If you have been following the matrix.org blog for some time, you will know that we’ve never been ones to shy away from complex topics like public policy and its impacts on Matrix. With this blog post series, our aim is to introduce a more regular cadence to our regulatory updates and to be more transparent about where we are focusing our efforts in this area.
Each blog post in the series will focus on a given theme or piece of law, as well as its relevant jurisdiction. We will start this series by taking a deep dive into EU regulation, starting with the General Data Protection Regulation (GDPR). Future blog posts in the series will cover the digital services package (DMA and DSA), the incoming CRA and the highly controversial CSAM regulation. These will be followed by a series dedicated to the UK, particularly UK applications of European law such as the GDPR and ePrivacy directive, as well as the Online Safety Act and the IPA amendment bill. Finally, we will conclude the series by looking across the pond and diving into the Cloud Act, as well as KOSA and other existing proposals.
The big one
Over the last decade the most impactful - in terms of grassroots led change, not necessarily enforcement - piece of legislation was probably the GDPR. Although its initial enforcement date (May 2018) was a couple of years before I joined Element and the Matrix.org Foundation, I know that preparing for GDPR was a huge effort that led to a lot of deep thinking about the ins and outs of Matrix.
These have been certainly years of learning, iterating and evolving our approach to this fascinating piece of law, and just like 6 years ago, we keep receiving questions and feedback about how the GDPR applies to the Matrix protocol and how server administrators can remain compliant.
We maintain the view that the GDPR was not built for a decentralised digital world, after all we are (for now!) the exception, not the rule. This does not mean, however, that we don’t make every effort possible to comply with the spirit and the letter of the law. Most organisations using the Matrix protocol will be running strictly closed federations or single servers in closed federation, which they fully control (or appoint others to support them with that control). Compliance is a lot more straightforward in this sense, so for now we will focus on those using Matrix to interact with open federation.
The most difficult part of GDPR compliance for Matrix has always been article 17, right to erasure, or ‘right to be forgotten’ as it is usually known. Our view has always been that this is a relative right, which needs to take into account available technology and other applicable laws in order to be enacted. For example, if an employee leaves a company and asks for all of their data to be erased, in practice not all of that data will actually be erased, due to constraints put in place by other legal instruments (i.e. tax and fraud prevention regulations, employment law, etc.).
Considering this relativity, we looked at what other ubiquitous product offerings already exist in the decentralised space, and how they address erasure. Using email as an analogy was a no-brainer in this regard: one understands that deleting an email account will delete everything from their inbox and sent folders, whilst also understanding that is impossible and not expectable to have the same data deleted from the recipients’ inbox (or the inboxes of those that might have had the same message forwarded to them).
Use of technology is always associated with constant give and take and risk based decisions. Whilst we make every effort to minimise risks for the people using the Matrix protocol, the reality is that one of the main purposes of the protocol is integrity of communications and decentralisation of communications data. This is directly at odds with absolute deletion of communications.
So how do we come to terms with this conflict? Following the email analogy, we address right to erasure from two different standpoints: account data and communications data. As part of the protocol, everyone is able to automatically delete their accounts and select an option to delete all of their data. First, this deletes all ‘external’ data associated with the account, such as e-mail addresses, phone numbers, IP addresses and device identifiers - effectively pseudonymising this data, aside from Matrix IDs which we will address later on. This measure follows recital 28 of the GDPR, which mentions pseudonymisation as a risk reducing measure for data subjects, which helps controllers and processors meet their data protection obligations.
Now, the thornier issue of communications data. How can we apply erasure whilst maintaining the integrity of the service for remaining users? Keeping on with the email analogy, if someone deletes their sent email from their own account, you would still expect an existing thread with multiple users to work and be visible. You just would not expect the same message to leave the original account to new individuals.
That is precisely how the issue is addressed in Matrix - upon deletion of an account, and request for deletion of messages, a flag is applied to the original messages which prevents it from ever being displayed to any new people. This, of course, requires cooperation from the other server administrators who might also have copies of the same message in their server instance. We make this decision by referring to recital 66 of the GDPR, which requires controllers take reasonable steps, taking into account available technology, to inform other controllers of the data subject’s request to have their data erased.
One issue still remains unresolved - Matrix IDs (MXIDs) are always associated with state events (data sent by a user to modify the attributes of a given room - eg its name, its members, its avatar), which means those IDs could still leak over federation. That is a particular concern when one uses their legal name as their MXID. The solution for this is to replace the MXID with a pseudonym, maintaining the integrity of communications and fully removing any last remains of identifiable information in the system.
The Synapse team has also made a lot of progress on message retention policies and media retention, both huge strides to continuously improve the privacy of Matrix users, whilst also helping server administrators comply with their legal obligations.
Things have definitely progressed in the portability space, and chat export is now implemented in multiple Matrix clients, having been first implemented on Element Web in October 2021, in version 1.9.2. We look forward to seeing what the next year brings us in terms of privacy improvements.
For the next post in this series we will continue discussing the EU policy landscape, although from a different perspective. We will be going through the digital services package (which includes the DSA and DMA) and what does it all mean for Matrix now that we have reached implementation stage.
As always, we remain open to your feedback and thoughts. If there is anything you would like to hear more about or have a suggestion for a future blog in this series please feel free to reach out to dpo@matrix.org via email, DM to @gpdr:matrix.org or chat to us in the Office of the Matrix.org Foundation room.
The events of the last week have been utterly terrifying as we’ve seen a highly sophisticated targeted attack on open source infrastructure play out in public, in the form of the liblzma backdoor. Matrix is not impacted by the attack (none of our code or infrastructure is using liblzma or xz 5.6), but it has been a massive wakeup call in terms of understanding the risks posed by overstretched open source maintainership.
The attack particularly resonates as Matrix’s maintainership is distinctly overstretched currently - despite Matrix ending up at the heart of huge amounts of critical infrastructure, ranging from the Ukrainian MOD to NATO and at least 15 other countries and major international organisations that we know of.
Historically, Matrix development has been largely been funded by Element, the company set up by the team who created Matrix in order to fund their work on it. As unpopular as VC funding is in some circles, the Matrix community owes a huge debt of thanks to Element’s investors (Status, Notion, firstminute, Dawn, Automattic, Protocol Labs and Metaplanet) and Amdocs for funding over $50M of work on both Matrix and Element since 2017. Having a large professional team paid as their day job to maintain Matrix has helped enormously against xz-style attacks.
However, this model is simply not sustainable: these days, Element is focused on being able to pay its own costs rather than being dependent on further VC investment. This leaves a massive hole in funding for Matrix, and we’ve already seen the impact of this with projects like Dendrite, Low Bandwidth Matrix, Account Portability, P2P Matrix and Third Room no longer able to be funded by Element (for now). Meanwhile, the remaining core team is stretched.
This feels particularly unfortunate given the number of governments and public sector organisations who rely on Matrix, but in practice it turns out that finding a way for them to fund open source maintenance can be surprisingly challenging - despite the potential impact of Matrix not being able to invest in security (or cryptography, or trust & safety, or performance improvements) being catastrophic, especially as Matrix becomes more and more of a high value target for large scale adversaries.
There seems to be two types of problems: firstly, those who don’t understand why it might be beneficial for a government to pay for open source at all. A particularly amazing real-life example of this came from a certain Ministry of Defence last week, whose procurement department (on being asked to help fund core Matrix development, given their operational dependency on Matrix) said: “You have to understand, we’re responsible for taxpayer money here. We can’t just make a donation to your open source project.” Apparently if we had built the same tech as a proprietary product, paying for it would apparently have been an infinitely better use of taxpayer money. Now, thankfully, organisations like FSFE and EDRi and OSBA have made major strides in educating governments to understand that funnelling taxpayer money into proprietary software licences does not benefit the public in the way that using open source software does - but old views die hard.
Then, perversely, the second problem emerges: FSFE’s well-intentioned “Public Money, Public Code” campaign is often given to us as a reason to insist on funding features rather than maintenance. This seems to be because procurement departments want to have something concrete to procure as a one-off, rather than making an ongoing commitment to keep the project secure, existing and healthy - and so focus on funding new features (or hiring their own staff to build their features) and ignore maintenance. If you ever wondered why Element has so many weird and wonderful features (which are not always maintained as well as they might), this is part of the problem. The problem is captured beautifully in Tobie Langel’s excellent (and highly topical) talk from this year’s State of Open Con:
However: we think there might (just might!) be a long-term solution in sight.
Particularly in the wake of the xz/liblzma attack, it seems that governments may be more aware that they and their societies depend enormously on FOSS infrastructure to operate. Free and open source software has literally become shared digital public infrastructure. And much like shared physical public infrastructure - bridges, roads, sea defences, etc - FOSS maintenance should be funded by governments on behalf of the taxpayer.
This funding should NOT be tied to specific feature development, but simply funding the core maintenance of the infrastructure - paying for the maintainers (and/or letting them or their umbrella org hire trusted ones!) to ensure the core project remains healthy and secure. Otherwise, the pressure just rises on the core project to chase feature development at the expense of maintenance (making maintenance harder) - or, worse, to be pushed away from open source into building proprietary solutions or crippling the open source by moving valuable features into side proprietary products.
Now, the good news is that some organisations are already trying to solve this problem:
OpenSSF is a Linux Foundation project focused on helping solve the problem
Sovereign Tech Fund similarly supports smaller projects to act as a digital public good
Copie Publique takes a different approach to encourage private companies to contribute back directly to FOSS projects
European Digital Infrastructure Consortium (EDIC) is a new strategy from the EU to enable multi-country funding of digital infrastructure projects (but so far seems to be aiming at funding governments themselves, rather than underlying FOSS projects).
However, most of these are not yet operating at the scale of a project like Matrix, and the irony is that the bigger projects need even more financial support than smaller projects to keep alive and sustainable. High level funding does exist in the form of the EU’s Horizon programme for R&D and Innovation (which provides the upstream for NLnet and NGI), with a total budget of €95.5B. However, it’s currently set up to only fund consortiums rather than independent projects - and the last thing a typical open source project needs is to orchestrate and administer an international consortium of vendors and universities in order to get itself funded.
The perfect solution in the EU would probably be a NLnet-style organisation with the remit to route funds in the range of low-millions a year to larger projects like Matrix which have become widespread critical infrastructure - to allow them to thrive in their mission without trying to coerce typical public sector procurement into picking up the bill. Or maybe a tax should be instantiated to force large scale open source projects users to route recurring funding to the project maintainers. So, Governments: please route taxpayer money to support the maintenance (not just features!) of open source projects that your country depends upon, before it’s too late. This also means educating procurement to the topic and updating procurement frameworks to be able to support this.
Meanwhile, we are in the middle of running a fundraising drive to help address the funding gap, which is currently making cautiously positive progress towards its £900K target, having raised £415K since last year, entirely thanks to Individual, Silver and Gold members joining. Our new membership model is working - giving the wider Matrix community a way to join the Foundation in order to participate in the upcoming Governing Board, and help steer the direction of the project, while contributing funding! So while we hope that governments will read this blog post and point out ways to sustainably fund more of the maintenance they depend on - today, you can help too by persuading your organisation (or yourself!) to become a member today and help keep Matrix funded and pointed in the right direction.
Our first ever Governing Board elections are approaching, with the nomination period set to begin in late April. If you want your organization or community to be able to nominate a candidate, or to vote in the election, the time to join as a member is now – same for individuals who want to participate!
This week we’re thrilled to announce three new members: two Ecosystem Members, Trixnity and Nheko-Reborn, and our first Platinum Member, Element!
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/proposals.
A reminder to teams working on Matrix 2.0 features to update their MSCs and send them for review in #sct-office:matrix.org soon to help bring them closer to release in the next couple of quarters.
Thank you to Kévin, Johennes and Rich for their PRs to the spec text this week!
This proposal makes the case for putting the onus on homeservers for tracking whether a user is in a Matrix voice/video call, rather than relying on clients. Currently, clients must update the user's m.call.member state event in the room when they join/leave a call. This can be problematic if a user closes their client without giving it a chance to do so however (such as when a browser is force-quit). This can lead to other clients thinking the client is still in the call, when it left minutes ago!
A homeserver typically continuously remains online. If it were in charge of tracking call membership, then it could notice that a user's device has disconnected from all call streams, and could then update the m.call.member state event in the room itself for that user.
The MSC also proposes two new Client-Server API endpoints which instruct the homeserver to either add or remove a device from the user's m.call.member state event. This would prevent two clients attempting to update the state event at the same time, potentially resulting in the modification of one of the clients being lost.
The intention is that this would lead to more reliable Matrix voice/video calls! If any of that sounds interesting to you, please have a look and leave your thoughts on the MSC.
This week the team released Synapse v1.104.0rc1, in preparation for the full release of v1.104.0. It contains improvements to OIDC support, as well as a few fixes for various bugs and to the Synapse Docker image.
Please test the release candidate if you're able to!
I've released and published iamb v0.0.9! 🎉 Since its last version, the client has gained support for:
Image previews through several different terminal image protocols
Threads, unread indicators, and notifications via terminal bell or desktop environment
Customizing keybindings, sorting for room and member lists, and other parts of the UI
Commands for importing and exporting room keys
Updated to use v0.7.x of the matrix-rust-sdk
SSO login, TOML configuration, fixes for Windows Terminal, improved manual pages, and many other fixes and improvements!
The online documentation has been updated to account for new configuration options and commands, and you can read the GitHub releases page for a full list of changes. I've also added a PACKAGING.md file to the repository to provide a place for notes useful to package maintainers.
Many thanks to all of those who have contributed to this release! :pray:
SchildiChat is a fork of Element that focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.
SchildiChat Next, our fork of Element X for Android, received some new experimental settings for managing your chat overview. You can now:
Sort favorite chats on top
Sort low priority chats on bottom
Filter by DMS/groups, unread chats, favorites, and rooms not added to any space, via dedicated pages from our bottom space navigation
Furthermore, you can now send freeform reactions again, if you want to react to messages with arbitrary text or just want to use your keyboard's emoji picker instead of the app's inbuilt picker.
Spring is here in Fractal land. Birds chirping, flowers blooming, and a new beta for you to try!
Staff’s picks for Fractal 7.beta:
Encryption support has been extended, with server-side key backup and account recovery.
Messages that failed to send can now be retried or discarded.
Messages can be reported to server admins for moderation.
Room details are now considered complete, with the addition of room address management, permissions, and version upgrade.
A new member menu appears when clicking on an avatar in the room history. It offers a quick way to do many actions related to that person, including opening a direct chat with them and moderating them.
Pills are clickable and allow to directly go to a room or member profile.
Many more improvements on the accessibility front, for better navigability with a screen reader.
As usual, this release includes other improvements, fixes and new translations thanks to all our contributors, and our upstream projects.
As the version implies, there might be a slight risk of regressions, but it should be mostly stable. If all goes well the next step is the release candidate!
As always, you can try to fix one of our issues. Any help is greatly appreciated!
This week we’ve done some preparatory work to support permalink navigation. In the meantime, the Rust team is actively working on exposing new API to support this feature.
Also working on a troubleshoot notification screen, as per what was done on Element Android. We will also work on adding a way to switch between available PushProviders in the coming week..
Last note: it will be more obvious that a poll is closed in the timeline, thanks to an enlightenment of the winning answer.
Circles is a secure social network app for families and friends, where every post is protected with Matrix's E2E encryption. We have recent releases of Circles on both Android and iOS.
On Android, v1.0.27 is now available on F-Droid.org and in beta on the Play Store. New features in this release include:
Improved email address management
Re-designed People tab to help you connect with friends of friends
Support for recovering your account when you forgot your password
UI for configuring the default power level in each room
On iOS, v1.0.1 is now available in the App Store, featuring:
New support for push notifications
Cross-signing for an account's 2nd/3rd/etc devices
Several bug fixes
Source code is available from the FUTO Gitlab (Android | iOS) and from our Github mirrors (Android | iOS).
If you're interested in trying the app, stop by and say "Hi" in #circles:futo.org.
It's been a quiet week, with the team merging the LinkedChunk data structure for efficiently storing events in the EventCache, a few bugfixes here and there, as well as a full investigation and fixing issues in integration tests.
The team keeps on working on the event cache (including matching known events against new events received from servers, and prepping for caching on disk) as well as the QR code login feature.
The playbook used to install Redis (and now installs KeyDB in its place) if services have a need for it (enabling worker support for Synapse, enabling Hookshot encryption, etc.) or if you explicitly enabled the service (redis_enabled: true or keydb_enabled: true).
This change is provoked by the fact that Redis is now "source available". According to the Limitations of the new license (as best as we understand them, given that we're not lawyers), using Redis in the playbook (even in a commercial FOSS service like etke.cc) does not violate the new Redis license. That said, we'd rather neither risk it, nor endorse shady licenses and products that pretend to be free-software. Another high-quality alternative to Redis seems to be Dragonfly, but the Dragonfly license is no better than Redis's.
Thanks to Julian Foad, matrix-docker-ansible-deploy can now install the Pantalaimon E2EE aware proxy daemon for you. It's already possible to integrate it with Draupnir to allow it to work in E2EE rooms - see our Draupnir docs for details.
Draupnir v1.87.0 was released, and this will be the final release before we merge our 6month in the making rework of the Draupnir core. We are advising anyone running against gnuxie/draupnir:develop to instead pin to gnuxie/draupnir:v1.87.0 as there is going to a significant drop in user experience as we integrate and gather more feedback on the rework. Those running against gnuxie/draupnir:latest will be unaffected, but we will advise you to read the v2.0.0 release notes as there will be dramatic changes in behaviour. If you're a confident system admin and you're happy with manually intervening then by all means continue to use develop and come talk to us in #draupnir:matrix.org.
Have you seen the weather forecast? Finally we can stop pretending to be interested in matrix and meet up for BBQ again :)
Meet other matrix users, chat about Matrix, the rest, and everything else, discuss your Matrix ideas, sign each other in persona, and maybe spice the evening with a good mate or beer.
And don't forget to wish you brougth your favorite item :)
Every first Wednesday of the month in the c-base at 8pm ('til the next pandemic).
I have for the first time in years pushed a release of my octoprint plugin which sends matrix notifications about the status of your prints with images. Thanks primarily to @Links2004 we now properly handle image transformations, upload images in a background thread and don't send markdown in the plain text body of the matrix message. If you run into any issues please open an issue on the github repo.
PubHubs is a Dutch research project to enable citizen-facing organisations to provide online group communications, value-aligned with their real-world presence. It uses matrix protocol, combined with an interesting and different user identity model involving pseudonyms and selective cryptographic disclosure of identity attributes such as "is over 18" or "is a member of organisation X". Each hub is built around a non-federating Synapse server, with their own identity plugins and custom client. (All public-interest, open-source.)
I have been working on three aspects of introducing initial moderation tooling. Some of it crosses over with general matrix (this week's Pantalaimon role is a side product of the Draupnir part), while some is different (Attribute Disclosure), and the third part is general (research and planning for Civilised Discourse).
My funded stint is coming to an end and I am looking for ways to continue in any related area of work -- please matrix me @julian:foad.me.uk !
Some of you may be familiar with https://codebrowser.dev already. For those of you who aren't, it's a tool that parses compiled code in C++, Rust, or Dart and displays it as a webpage with full syntax highlighting, clickable links to symbols, searchable symbols, and a wealth of other features. It's very useful for browsing a library's source code to see what individual functions do (for example, here's QObject::connect() from Qt 6). The codebrowser software was originally built by Woboq, but as the maintainers have moved on to work on things like Slint, they have passed it on to KDAB.
I have spun up an instance of codebrowser at https://matrix.codebrowser.lorendb.dev, serving sources for mtxclient, libQuotient, and olm so far. If you want to see more projects added, please let me know and I'll add them if possible. I'm focusing on libraries for now, as they are the most useful for developers, but I'm not opposed to adding other Matrix projects either.
Please note that I currently can't support Rust or Dart code yet; while KDAB has implemented support for those languages, they haven't yet open-sourced their changes (though I can confirm that is tentatively in the pipeline). Once they open-source their changes, I'll be upgrading my installation to use that version and adding project like matrix-rust-sdk and vodozemac.
As of today, 9531 Matrix federateable servers have been discovered by matrixrooms.info, 2844 (29.8%) of them are publishing their rooms directory over federation.
The published directories contain 161695 rooms (gitter.im rooms are back)
The wait is over! We are now accepting applications from projects and organizations that want to join the Foundation as Ecosystem and Associate Members.
You may recall that in December we announced two new membership tiers to ensure we have healthy community representation as we introduce more open governance for Matrix. Specifically, we created Ecosystem and Associate Member tiers. Ecosystem Members are open source projects that speak Matrix, and Associate Members are FOSS foundations, nonprofits, and academic institutions.
We’re excited to bring more folks into the fold, and thrilled to welcome Thunderbird, Draupnir, and Conduit as our first Ecosystem Members 🎉 Curious who else has joined as a member recently? Check out our Supporters.
We’d also like to recognize Thunderbird for sponsoring the Foundation, which helps us get a little bit closer to making the Foundation’s activities sustainable 🚀
There is a question we often get asked: why do you operate the Matrix.org homeserver, and why do you keep registrations open? We don't want to become the gmail.com of Matrix after all. We covered those in this post.
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/proposals.
Next Friday, March 22nd, is the release date of Matrix v1.10! Watch this space for when the blog post goes live.
A new release of the spec means it's planning time for the subsequent one. For v1.11 we're looking to focus on Trust & Safety, authenticated media and Extensible Events. See this message in the Office of the Matrix Spec Core Team room for further details. If you have something that you'd like to suggest that isn't already on that list, please do so in that room!
Otherwise there has been lots of activity in the spec-writing side of things this week. 5 new PRs and 2 merged during this week alone! A mix of spec PRs and general fixes/improvements to the spec itself. Thank you to everyone who's pitching in!
The gist of this proposal is that the existing GET /_matrix/client/v1/rooms/{roomId}/timestamp_to_event endpoint (which allows you to jump to a point in a room based on a given date and time), could suffer from some edge cases. Particularly if you're heavily using /timestamp_to_event to form your room timeline (which the Matrix Viewer does).
Synapse v1.103.0 is due out next week, carrying with it some upcoming features and bug fixes:
Stabilise, and enable by default, support for MSC4041 (Retry-After header).
Fix performance issues when joining very large rooms that can cause the server to lock up (thanks to @ggogel for their persistence on this one!)
Fix a bug introduced in v1.76.0 that prevented joining remote rooms when a module uses the on_new_event callback.
...and more!
We have a release candidate out (v1.103.0rc1) if you would like to help test early and shake out any bugs before the full release. Otherwise, expect v1.103.0 to land next week. Thanks!
Element X Android 0.4.5 is now in production. It adds the ability to mark a room as favourite and adds some action regarding moderation (moderation is still under a disabled feature flag).
The next release candidate (0.4.6) can already be downloaded from GitHub here: https://github.com/element-hq/element-x-android/releases/tag/v0.4.6 and will soon be available on the PlayStore if you’re a tester. The loading of the room members is much faster, especially in crowded rooms, and the UX has been tweaked too.
We are working on room filtering, but the feature is not enabled yet on production. You can use the nightly build to be able to enable it from the settings, Developer options.
The room moderation feature is now closer to completion: if you are an admin you can now change the roles of other users, the default power levels for some actions. If you are either an admin or a moderator you can now remove, ban or unban users.
We are also working on a tool to be able to check the quality of Element X Android translations. It is available at https://element-hq.github.io/element-x-android/ If you spot any mistakes / typo / inconsistency in any languages, feel free to reach us! You can help translate the application at https://localazy.com/p/element. We manage the translations in French and German, but other languages are provided by the community. Also, we enable the language only if the translation progress is close to 100%. For layout issues, please report an issue using https://github.com/element-hq/element-x-android/issues/new/choose. Thanks!
We've released a new Android app that's awesome and rebuilt from the ground up instead of being an Element fork. It uses Kotlin and Jetpack Compose for the UI and mautrix-go for all the Matrix things. For more details, check out our blog post: https://blog.beeper.com/p/new-beeper-android-app-open-beta
More logging has been added to the crypto crate, to better understand unable-to-decrypt (UTD) events. In particular, it's possible to listen to the UI timeline's UTDs events, using a special dedicated hook
Back-pagination has been added to the Event Cache, and the UI timeline now makes use of that, in place of its own code.
The event cache now has an experimental fancy data structure to represent events, implemented as a linked list of chunks. It's not used yet, but it's being integrated.
A filter to show only invites has been added to the Room List service.
When fetching all the room members, the computation for disambiguating names has been optimized, resulting in much smaller run times for Room::sync_members() et al. (And a benchmark has been added to prove it!)
And your usual load of refactorings and smaller bugfixes and improvements.
Working on some scope cleanup, moving utilities into subclasses in order to make extensions more cleanly separated, along with some nice bot changes, in order to allow more versatility and customisation.
Changes
Moved named filters to hs.NamedCaches.FilterCache
This also means that any named filter handling functions have been removed from AuthenticatedHomeserverGeneric
Bot utils: AddMatrixBot no longer adds default commands (see additions)
Additions
Added BodyWithoutReplyFallback to RoomMessageEventContent
Added file/media cache at hs.NamedCaches.FileCache
This allows avoiding reuploads of files that are commonly reused, such as reaction images
Added WithCustomEmoji(string mxcUri, string name) to MessageBuilder
You can now add custom emojis to messages, whether you're using a custom emoji pack or not
Added WithCodeBlock(string code, string language = "") to MessageBuilder
Added WithCollapsibleSection to MessageBuilder - sometimes a code blocks or details can get very long :)
Added generic named cache implementation (NamedCache<T>(string name))
You can now easily implement your own caches mapped to account data as Dictionary<string, T>
Language is optional but can be omitted for clients that don't support it
Added hs.NamedCaches in order to group named caches for different data types
AddMatrixBot no longer adds command support or default commands
There's a configuration option (MentionPrefix), to allow mentioning the bot directly, as prefix
Added methods to services.AddMatrixBot():
AddCommandHandler() - Adds command handler
AddCommands(Type[] commandClasses) - Allows adding a list of command classes, without necessarily including builtins like !help
DiscoverAllCommands() - Registers all commands that can be found, including builtins such as !help
WithInviteHandler(Func<InviteEvent, Task> handler) - Allows easy addition of invite handling with custom logic (eg. must be in x room)
WithCommandResultHandler(Func<CommandResult, Task> handler) - Allows overriding command/permission error handling in order to add custom messages, or extra logging
Bot utils: Added Aliases and Unlisted to ICommand, allowing bots to add aliases and hide commands from !help
Bot utils: Added support for multiple prefixes, and prefixes with spaces (needs more testing)
Bot utils: Added support for command names with spaces (needs more testing)
Every first Tuesday of the month, the Digitale Gesellschaft e.V. ("digital society") hosts the Netzpolitischer Abend ("internet politics evening") at c-base Berlin, Germany, with - you may have guessed it already - talks covering digital and internet politics (usually in German). I hope that does not stop you, because the 134th edition from 1709665200 2024-03-05 20:00 CET featured talks on some great topics ranging from Feminist Internet Politics over Freedom of Press to the Latest News on Chat Control. The most interesting topics to you, dear TWIM reader, might be Nadine's update on her Polychat project (#polychat-ambassadors:datanauten.de) and Yan (aka yncyrydybyl aka t) giving a brief introduction on the Matrix protocol and Foundation, an overview how Matrix is being implemented in the German public administration, and answering questions from the in-person attendees. View Yan's talk on YouTube, Vimeo, or watch the live stream of the whole evening here. Here are some links to the things Yan shows in his presentation, I tried to also include links to the English versions where available:
Map of (mostly German and German language) Universities
Tip: If you'd like to follow Digitale Gesellschaft/Netzpolitischer Abend, you can subscribe to their social media or their website feed/youtube feed using your favorite tool; mine is called hookshot.
External videos & posts, reaction images, polls… There are many things I like to see inline in conversations without having to follow a link. emweb is a prototype for a lightweight protocol to solve for all of these and more, while keeping implementation burden low.
I am looking for interested parties that want to help refine the idea, especially with regard to security and UX. There is also a lot more prototypes to build and spread inspiration. A few from the top of my mind: coin flips, games likes tic tac toe or chess, live location sharing…
So far I have two scrappy host implementations: one for the Matrix client Cinny (hosted here) and one for the Mastodon client Elk (hosted here).
The Elk fork is not just an emweb-host but also a guest, so you can send links to Mastodon posts on it within the emweb fork of Cinny, and they will be directly embedded into your chat.
As of today, 9473 Matrix federateable servers have been discovered by matrixrooms.info, 2816 (29.7%) of them are publishing their rooms directory over federation.
The published directories contain 20025 rooms.
Matrix self-defines as the go-to protocol for "open, secure, decentralised communications". The Matrix.org Foundation defines "maximising […] the number of online servers in the open federation" as a key objective in its mission. So why is the Matrix.org homeserver, probably the largest homeserver in the whole federation, still in open registration mode?
Let's dive into the technical, and user experience choices behind it.
A point of centralisation
The Matrix protocol is heavily decentralised. It goes beyond just federation: every single server in a room maintains a local copy of that room in sync with all the other servers participating in the conversation.
Matrix is often compared to email but for instant messaging. The first thing that comes to mind is that email might be federated, but it's no longer that decentralised. According to this article on sellcell.com, Gmail.com provides nearly 50% of the active email accounts in 2023, iCloud follows with 27.8%, and Outlook.com owns 13.1% of them.
The Matrix.org Foundation doesn't want to become the Gmail.com or Outlook.com of Matrix. It doesn't generate any revenue from its users and runs on donations. The costs of hosting, maintenance and support of the Matrix.org homeserver grow with the number of people using it.
A steep and steady growth comes with three significant issues:
The technical maintenance costs do not grow linearly with the number of users. The infrastructure itself costs more, but more time needs to be spent on research and development to make the backend scale.
Moderation becomes more difficult. The more reports the Trust & Safety team has to handle, the less time it can spend on tools to improve the moderation experience.
Support can't be automated and is very costly to scale. The more users on Matrix.org, the less responsive the support team can be (unless we find resources to hire more support team members) and the more frustrating it is for everyone involved.
In short: everyone stands to benefit from Matrix.org not being a supermassive homeserver that is difficult to leave.
Mindfulness of others
One of the goals of the Matrix.org Foundation is to maximise the number of Matrix users in the open federation. Another goal is to make sure Matrix remains decentralised by increasing the number of servers in that same open federation.
One of the fundamental aspects that makes Matrix appealing is decentralisation and (open) federation. Those concepts are not as difficult for the general public to understand as some claim. But we must acknowledge the general public is not regularly confronted with them and their implications when trying new apps.
For Matrix to gain traction outside of technical circles, it needs to be mindful of the time and habits of the general public. Whether we like it or not, people are used to trying apps, not protocols or providers. It is not reasonable to ask a user to pick a homeserver before they even know what a homeserver is, nor what this decision entails. People need to be able to download an app, try it right away, and get some (potential) value from it before making more advanced decisions.
We have made Matrix.org one of the most prominent entry points for people discovering Matrix because we believe inclusivity is one of the fundamental premises of Matrix's global success.
So then, how do we reconcile the need for ease of adoption with the goal of decentralisation?
A way in and a way out
Creating an account on Matrix.org is a typical first step in the Matrix user journey. Since agglomeration around a single instance is against the goals of Matrix and its Foundation, users need to have a way out and incentives to move.
To move to another homeserver, a user needs to be able to pick a homeserver and move there.
Picking a homeserver
Matrix community member Austin Huang created servers.joinmatrix.org to list all the servers with open registrations where people could find a new home. I am working on creating an official and curated version. We need the instance picker to work on Matrix.org's website, and provide data that the client apps can reuse and present to users who want to move away from Matrix.org.
Moving to the new homeserver
The Foundation will implement an account migration tool to allow people to migrate to another server. A proper portable identity mechanism would be better - however, the Foundation had to pause its development due to a lack of funding. Such a mechanism would also rely on a new room version. It would only allow migrating conversations that happened after a room upgrade to that hypothetical new version, which results in data loss.
The most reasonable solution
There are alternatives to a prominent onboarding homeserver like the Matrix.org homeserver. We don't find any of them satisfactory:
Picking a server randomly when a user wants to sign up in the app is confusing and has a lot of security implications. This is not acceptable.
Forcing the user to choose a server when they sign up is a highly cumbersome process that does not respect the user's time and resources.
Portable identities are very desirable but will take time to materialise.
P2P Matrix depends on a portable identity mechanism and is further away in the future.
As you can see, we at the Foundation need to strike a balance as we pursue goals that are occasionally in tension with one another, and the Matrix.org homeserver is a prime example. We are committed both to making Matrix more accessible, and to doing the work to decenter the Matrix.org homeserver.
You can help us achieve both: the Foundation needs your support! If you can afford to, please consider supporting us financially. If you want to contribute to the Foundation’s governance, consider becoming a member.
more good crypto news this week from Canada: the Online Harms Act looks to be explicitly protecting end-to-end encryption and private communications - in contrast to the UK Online Safety Bill/Act. Looks like all attempts to educate around the UK OSB might have not been entirely in vain. You can see it direct from Justice Minister Arif Virani here: https://youtu.be/TjDrAlfhNAA?t=1766
Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://spec.matrix.org/proposals.
Along with the typical rounds of review this week, there was a request on https://github.com/matrix-org/matrix-spec/pull/1739 for client authors to voice their opinion on whether they'd be happy with deprecating the <font> tag in the spec. The tag has been dropped from the WHATWG HTML standard, and replaced by <span>. We'd like to follow this standard where possible in Matrix, though understand that many clients have not made the switch yet.
If you're a client developer that's particularly interested in this bit of the spec, please do join in the conversation!
This one is a real "how did we not think of that" moment. If you aren't familiar with soft logout, it's the mechanism by which the homeserver can sign out a client while hinting that it should NOT destroy the local client state (like encryption keys). This is useful for situations such as resetting your password, where you want your other devices to have to put in the password again, but don't necessary want to have to do an initial sync.
This MSC aims to address this issue, where upon being soft-logged out, the homeserver doesn't tell the client how to log in again. A homeserver can have multiple methods of logging in (via a password, SSO, etc.). Yet a user is typically only tied to one of them. Thus, if you logged in via password login, then were soft-logged out, and tried to log in again via SSO... well you'd end up with a very confused client indeed!
The solution is to allow access tokens that have been soft-logged out (but not yet deleted) to be provided to the /login endpoint. When the server recognises that this is a soft-logged out client, it will tell that client only the one flow that it used to login. Thus there can be no confusion.
This week we released Synapse v1.102.0. This release fixes a fairly annoying bug introduced in Synapse v1.100.0 for mobile users (particularly iOS) in which notification counts were incorrect. Thanks for your patience there!
Other bug fixes and performance improvements as well of course. See the link above for the full changelog.
SchildiChat is a fork of Element that focuses on UI changes such as message bubbles and a unified chat list for both direct messages and groups, which is a more familiar approach to users of other popular instant messengers.
This week, we published the first actual release of SchildiChat Next, our fork of Element X for Android. While previously, you could already download bleeding-edge beta releases from my testing F-Droid repository, this means we're now additionally starting to provide "stable" builds, that you can currently get either from my main F-Droid repository or from Google Play. (Of course, even with the introduction of "stable releases", SchildiChat Next should still be regarded as some kind of preview, while we're still missing functionality from the old client, just like Element X has not replaced the old Element client yet.) So if you prefer less frequently updated but more thoroughly tested builds, or were waiting for a Play Store release, feel free to give it a try!
While there have been lots of theming fixes with the new plasma6 themes, I guess the most interesting change is that Nheko supports intentional mentions now. This means you can explicitly select who you want to notify and who not, when sending a message. This has a lot of weird interactions with the text input, so if you find any, please report how to reproduce them, so that we can fix them!
With this change we should be compatible with Matrix 1.9 now, which means we bumped our maximum supported version appropriately. Before that we were stuck on 1.5, so this is basically a jump forward by a whole year! Such a small change and such a big step, one could probably make a good quote out of that one...
Oh, and there is a new command to make your text glitch... Don't use it... it makes you text look bad!
Anyway, a bit of offtopic in the end: I have seen surprisingly many mentions of people being happy with Nheko recently. I always appreciate those, so thank you for saying that out loud for me to notice!
Have a nice day and I hope your cake will be plentiful!
Moment gains the ability to author spoilers add reactions to messages! As usual, we are focusing on keyboard-use, so in Moment, you add reactions by replying to a message with the /react command. A great idea, contributed by Ellie Clifford!
We have also finally fixed those crazy graphical glitches, previously seen in many menu/settings screens!
Other than that, we're just keeping up with our dependencies, so you shouldn't get compatibility issues when installing Moment on an up-to-date system.
Following last week's RC, version 0.4.5 of the app has been publicly released on the Play Store.
This week has been a really quiet one, focused on progressing through our ongoing work in several features that you can already test in the nightly version of the app:
Room moderation: you should now be able to change the roles of room members and promote and demote them as needed, from a new 'Roles and permissions' section in the room details screen. Note this is behind a feature flag you'll probably have to enable.
It's also possible to kick, ban and unban users from a room as long as you have the right power level.
The work on room filters continues, and we've been focused on tweaking the performance and UX for them.
We're also focusing on improving the localisation of the app: we've been working on an tool to quickly identify UI issues by comparing screenshots of the app in several languages. Please note it's under heavy development, so some parts of it may not be fully functional.
It's now possible to get the unstable features of a server with Client::unstable_features(). Thanks to @hanadi92 for the patch!
One can now set a state store value without loading and returning the previous one. Thanks ben for the patch.
The UI Timeline now supports call invite events, showing such event as the latest room event too, if applicable.
The team keeps on focusing on the event cache, with PRs to extract the back-pagination logic from the UI timeline into the cache, and more preliminary work on storing events efficiently in memory and on disk.
and as usual, a few changes to the FFI, doc updates, and other goodies.
As of today, 9511 Matrix federateable servers have been discovered by matrixrooms.info, 2808 (29.5%) of them are publishing their rooms directory over federation.
The published directories contain 20008 rooms.
