we're starting a policy and regulation blog series over on the Foundation's blog. Over the next few months I'll be covering various pieces of legislation that are already in place, as well as incoming regulation, and what it all means for Matrix.
The new FluffyChat has several performance improvements, including a fix which should speed up the whole app if you have a lot of megolm sessions.
Also FluffyChat v1.21.0 includes a new search and gallery feature for chats. You can now search for specific messages or browse the shared photos in a conversation.
Also please note that the default network request timeout has been changed. Before FluffyChat ran into a timeout if the server needed more than 30 seconds to send the initial sync. This was probably too optimistic so the timeout has now been set to 30 minutes. This should allow much more users with very large accounts to log in. Feel free to share your feedback.
All Changes
feat: Enable download images on iOS, not only share images (krille-chan)
feat: Search feature (krille-chan)
build: Update record package (krille-chan)
build: Use correct pubspec.yaml format for hosted dependency (krille-chan)
build: Use matrix sdk main branch (krille-chan)
chore: Change default timeout to 30 min (krille-chan)
QR Code login works is completed we are just waiting for matrix server to fully support native OIDC login, before rolling out the feature
Message queue work is also almost completed, now messages will be automatically queued for resending when internet connection comes back, without the need of taking any manual action
We are also working on storing and restoring composer drafts for each room, so no need to worry about losing anything that has not been sent yet if you browse another room or close your app
Great news also for element call integration, we are implementing native call notifications (with ringing), and a call log for DMs. Also starting a call will also be displayed as an event in the timeline.
We have added a quick implementation of sharing, so you can now send text and media from other applications.
QR Code login works is completed we are just waiting for matrix server to fully support native OIDC login, before rolling out the feature
Message queue work has been started, messages will be automatically queued for resending when internet connection comes back, without the need of taking any manual action.
Great news also for element call integration, we are implementing native call notifications (with ringing). Also starting a call will also be displayed as an event in the timeline.
Way back in December I TWIM'd I was working on a new project called complement-crypto which aims to write an extenstive, exhaustive set of end-to-end E2EE tests for Matrix clients. The aim of this work is to ensure encryption in Matrix Rust SDK in particular is robust to all kinds of failure modes: from connectivity blips over federation, server restarts and corrupted data, to actively malicious homeservers. Work has progressed significantly in the intervening 6 months:
The test suite runs in both Rust SDK and JS SDK CI pipelines to detect regressions. This required a lot of work enabling conditional compilation (so you don't need to run JS code in rust SDK) and improving test reliability to a sufficient standard that it can be relied upon.
It now has full federation support, spinning up sliding sync proxies for both homeservers if needed.
It has a comprehensive RPC client process to test ungraceful shutdowns (e.g what happens if your client gets SIGKILL'd?)
When things do go wrong, you have powerful debugging tools thanks to using mitmproxy for all communications, in addition to client log files and container logs. The use of mitmproxy is not only useful for debugging, but is also used as a test synchronisation primitive to reduce flakiness (e.g do X when you see the HTTP response to Y), as an assertion tool (e.g ensure the client has uploaded one-time keys by looking for the request), and as an adversarial tool (e.g modify the response returned to the client).
it has allowed us to test the SDKs to reproduce tricky "unable to decrypt" (UTD) scenarios which we couldn't otherwise replicate. The NSE bug was reproduced via a complement-crypto test which semi-randomly performed high-level operations like restarting clients and sending messages in a multiprocess environment to try to trip up the SDK code.
it has allowed us to write regression tests for UTDs such as:
adding targets beyond rust/JS SDK (matrix-bot-sdk, mautrix, and others). A perfect client can still receive UTDs if the sending client didn't encrypt correctly, and the sender may be on a different codebase.
This has been just one part of our work reducing UTD errors. Another significant part has been aggressively analysing bug reports in the wild to identify common failure modes. To that end, please report any 'unable to decrypt' messages you see and include log files with your report. Please also try to get the sender to submit a bug report. I may reach out to you if I need additional information. These bug reports not only guide our priorities, but have also helped us identify obscure failure modes which we cannot reproduce via complement-crypto.
Some minor changes, this week's update will be mostly project/management updates!
Call for help! (Update)
It's been decided to split the event rewrite up into multiple phases/parts. The original vision of the changes turned out to be far too optimistic, and we've run into some language-related challenges trying to implement them.
We'll still try to implement those changes, but this might depend on compile-time source generation or similar endeavours.
If this sounds like a fun challenge to you, or want to discuss and shape the semantics of these changes, feel free to join in!
Changes (Projects & Policies)
Some policy changes, and some new projects have been started within the LibMatrix source tree:
New workflow: Use of feature branches
Keeping up to date with LibMatrix is hard, because the chance of updating to a partially finished commit is too high.
To address this, we're moving to a "main branch is stable" model, with partial implementations on deciated branches so developers can keep up and prepare for those changes.
New subproject: Writing of full documentation
We want to improve the developer experience by offering documentation of features right in your editor, without having to reference the backing source. This should enable users who are less familiar with Matrix' internal structures to still be able to implement their deepest wishes.
New subproject: Workspace and structural cleanup
The LibMatrix source tree became a mess as the project grew. We want to restructure it to be maintainable, and also far more friendly to those who might want to contribute. If this sounds like something you'd like to help with or would love to offer feedback on, please do feel free to reach out at #libmatrix:rory.gay!
Splitting up of subproject: Event handling rewrite
Due to the large amount of complexity and work, we've decided to first start rewriting the inner implementations of events.
This should massively improve the end user experience, as custom event content is no longer lost if edited, as well as no longer having to worry about desyncs between actual event content and what you might see in an editor!
Next to this, you will likely see performance improvements when dealing with accessing the contents of events!
Longer-term changes still include static typing of event collections, especially around single content type collections such as fetching a room's member list, which should also see a large performance uplift when fetching!
Changes (Main release)
All dependencies have been updated. End users should not notice any differences as this is a maintenance task, but this does potentially enable some new usecases and squashes some bugs and security risks originating from outdated dependencies.
We have also expanded our team, in particular on the community growth side, and are about to open another position for a product designer ... if that's you or someone you know, stay tuned!
As of today, 9397 Matrix federateable servers have been discovered by matrixrooms.info, 2860 (30.4%) of them are publishing their rooms directory over federation.
The published directories contain 160776 rooms.
If you have been following the matrix.org blog for some time, you will know that we’ve never been ones to shy away from complex topics like public policy and its impacts on Matrix. With this blog post series, our aim is to introduce a more regular cadence to our regulatory updates and to be more transparent about where we are focusing our efforts in this area.
Each blog post in the series will focus on a given theme or piece of law, as well as its relevant jurisdiction. We will start this series by taking a deep dive into EU regulation, starting with the General Data Protection Regulation (GDPR). Future blog posts in the series will cover the digital services package (DMA and DSA), the incoming CRA and the highly controversial CSAM regulation. These will be followed by a series dedicated to the UK, particularly UK applications of European law such as the GDPR and ePrivacy directive, as well as the Online Safety Act and the IPA amendment bill. Finally, we will conclude the series by looking across the pond and diving into the Cloud Act, as well as KOSA and other existing proposals.
The big one
Over the last decade the most impactful - in terms of grassroots led change, not necessarily enforcement - piece of legislation was probably the GDPR. Although its initial enforcement date (May 2018) was a couple of years before I joined Element and the Matrix.org Foundation, I know that preparing for GDPR was a huge effort that led to a lot of deep thinking about the ins and outs of Matrix.
These have been certainly years of learning, iterating and evolving our approach to this fascinating piece of law, and just like 6 years ago, we keep receiving questions and feedback about how the GDPR applies to the Matrix protocol and how server administrators can remain compliant.
We maintain the view that the GDPR was not built for a decentralised digital world, after all we are (for now!) the exception, not the rule. This does not mean, however, that we don’t make every effort possible to comply with the spirit and the letter of the law. Most organisations using the Matrix protocol will be running strictly closed federations or single servers in closed federation, which they fully control (or appoint others to support them with that control). Compliance is a lot more straightforward in this sense, so for now we will focus on those using Matrix to interact with open federation.
The most difficult part of GDPR compliance for Matrix has always been article 17, right to erasure, or ‘right to be forgotten’ as it is usually known. Our view has always been that this is a relative right, which needs to take into account available technology and other applicable laws in order to be enacted. For example, if an employee leaves a company and asks for all of their data to be erased, in practice not all of that data will actually be erased, due to constraints put in place by other legal instruments (i.e. tax and fraud prevention regulations, employment law, etc.).
Considering this relativity, we looked at what other ubiquitous product offerings already exist in the decentralised space, and how they address erasure. Using email as an analogy was a no-brainer in this regard: one understands that deleting an email account will delete everything from their inbox and sent folders, whilst also understanding that is impossible and not expectable to have the same data deleted from the recipients’ inbox (or the inboxes of those that might have had the same message forwarded to them).
Use of technology is always associated with constant give and take and risk based decisions. Whilst we make every effort to minimise risks for the people using the Matrix protocol, the reality is that one of the main purposes of the protocol is integrity of communications and decentralisation of communications data. This is directly at odds with absolute deletion of communications.
So how do we come to terms with this conflict? Following the email analogy, we address right to erasure from two different standpoints: account data and communications data. As part of the protocol, everyone is able to automatically delete their accounts and select an option to delete all of their data. First, this deletes all ‘external’ data associated with the account, such as e-mail addresses, phone numbers, IP addresses and device identifiers - effectively pseudonymising this data, aside from Matrix IDs which we will address later on. This measure follows recital 28 of the GDPR, which mentions pseudonymisation as a risk reducing measure for data subjects, which helps controllers and processors meet their data protection obligations.
Now, the thornier issue of communications data. How can we apply erasure whilst maintaining the integrity of the service for remaining users? Keeping on with the email analogy, if someone deletes their sent email from their own account, you would still expect an existing thread with multiple users to work and be visible. You just would not expect the same message to leave the original account to new individuals.
That is precisely how the issue is addressed in Matrix - upon deletion of an account, and request for deletion of messages, a flag is applied to the original messages which prevents it from ever being displayed to any new people. This, of course, requires cooperation from the other server administrators who might also have copies of the same message in their server instance. We make this decision by referring to recital 66 of the GDPR, which requires controllers take reasonable steps, taking into account available technology, to inform other controllers of the data subject’s request to have their data erased.
One issue still remains unresolved - Matrix IDs (MXIDs) are always associated with state events (data sent by a user to modify the attributes of a given room - eg its name, its members, its avatar), which means those IDs could still leak over federation. That is a particular concern when one uses their legal name as their MXID. The solution for this is to replace the MXID with a pseudonym, maintaining the integrity of communications and fully removing any last remains of identifiable information in the system.
The Synapse team has also made a lot of progress on message retention policies and media retention, both huge strides to continuously improve the privacy of Matrix users, whilst also helping server administrators comply with their legal obligations.
Things have definitely progressed in the portability space, and chat export is now implemented in multiple Matrix clients, having been first implemented on Element Web in October 2021, in version 1.9.2. We look forward to seeing what the next year brings us in terms of privacy improvements.
For the next post in this series we will continue discussing the EU policy landscape, although from a different perspective. We will be going through the digital services package (which includes the DSA and DMA) and what does it all mean for Matrix now that we have reached implementation stage.
As always, we remain open to your feedback and thoughts. If there is anything you would like to hear more about or have a suggestion for a future blog in this series please feel free to reach out to dpo@matrix.org via email, DM to @gpdr:matrix.org or chat to us in the Office of the Matrix.org Foundation room.
