Release notes for v0.6.2

This release brings a few good things, including a new differential file lists sytem and end-to-end encryption for file transfer (thanks Mr.Alice for that!), and a greatly improved GUI.

Differential file list system

The old system for sharing list of files had originally been designed for a totally different purpose, and was very inefficient for this particular job since it required sending the full list of files every time something changed. We have designed a new system where directories are synced independently using a passive/active system: now friend lists are both passively synchronized in the background at a rather slow speed while being interactively updated while browsing. Random IDs are used when publishing directories to neighbor nodes so that they cannot be used to figure out what a friend is not sharing with you. The whole hashing system has also been improved.

The file sharing manager has been re-designed with less popup windows and more contextual handles. Most data (directories, virtual names, friend groups,…)  can be changed in place by double clicking on it. Changes only take place when the “apply and close” button is hit:

New share manager. A flag as been added to allow files to be searched anonymously (See end-to-end file encryption). The right column shows groups of nodes who can browse the data.

In the example above, “Private stuff” is only visible to friend nodes in the group “myself” and “Holiday rushes” is only visible to Friends and Family. No anonymous access is permitted to these files. “Pictures” is accessible *and searchable* anonymously but not visible in the friend list for any friend node.

Backward compatibility is not fully provided, mainly in order to avoid keeping complicated pieces of code, but an acceptable middle ground has been found. In summary:

• users will need to re-define their shared directories, which is a good opportunity to test the new share manager 😉 and does not take long;
• hashes are recovered from the previous version. So if you share the same directories, the hashing step should be very fast. The previous hash cache “file_cache.bin” is still loaded when found, and renamed to “file_cache.bin.bak”, and then converted into the new one “file_sharing/hash_cache.bin”. If needed, you can still rename this file to force Retroshare to import it again.

All these lists are of course stored encrypted on your disk including friends’ lists, which wasn’t the case before. There is one limitation however: the total number of shared files and directories should stay below 4,194,303 for a maximum number of friends of 1023, because of the storage encoding that has been designed for maximal efficiency (the new system removed all pointers so that Qt can access the hierarchy content both securely and rapidely).

End-to-end FT encryption

There has been a lot of discussion about both the possibility and the reason to encrypt anonymous file transfer tunnels, since anonymity guaranties that whatever the data is, it cannot be attributed to anyone in particular. There are however a number of situations where such a feature is wanted, which can be summarized as: files that are shared in private channels, private forums, or even privately sent using distant chat should not be possible to intercept by a node in the network who’s passively listening for the traffic.

Safely encrypting file transfer along anonymous tunnels is of course a chicken-and-egg problem: you would require a pre-shared secret information from which the encryption key is derived, with someone you cannot identify (by design, since the tunnels are made to be  strictly anonymous), so it sounds impossible to avoid a man-in-the-middle decryption (if you’re a troll, stop reading here and think for a while). We found a middle ground strategy (Partly similar to what GnuNet does, without using asymmetric authentication keys). Let H(f) represent a hash of the file to share:

• encrypted tunnels are opened through tunnels requests based on H(H(f))
• data packets are encrypted and authenticated using Chacha20+HMAC(sha256) with a random key for each packet. That key is derived by combining a 96-bits random initialisation vector with the actual hash of the file H(f). The construction is exactly the AEAD described in the RFC7539.
• files can optionally be non search-able, which keeps their hash secret

Consequently, files shared anonymously using private channels, forums, or securely sent using private chat are guarantied to reach destination using end-to-end encryption that is both anonymous and MITM-resistant, the secret pre-shared information being the hash of the file.

Note that we use our own implementation of Chacha20, which passes all the tests from RFC7539 (Chacha20 is not complicated really). When OpenSSL offers Chacha20 in its mainstream version, we’ll switch to it, without backward compatibility issues.

For backward compatibility reasons, users have now the possibility to change the encryption along file transfer tunnels with two modes (accessible from options->Files):

• accepted: downloaded files will open both encrypted and clear tunnels. Uploaded files accept both modes.
• enforced: downloaded files only open encrypted tunnels.

Obviously, only “enforced” mode totally protects the data, and unless needed, it should be used preferably.

Revised reputation system

The reputation system has been improved so that (1) users that do not give any opinion can rely on their friends to rate post authors; (2) if set, your own opinion is always a priority; (3) anti-spam prevents trolls to create new identities at every post; (4) new comers can easily bootstrap the voting problem.

Reputation now has 5 levels which are figured out by Retroshare combining 2 factors: your own opinion (with absolute priority) and opinions of your friend nodes. The resulting reputation score can be:

1. negative: you personally flagged that identity as bad.
2. remotely negative: your own opinion is neutral, but your friends provide more negative votes than positive votes
3. neutral: no information is available whatsoever for this identity
4. remotely positive: your opinion is neural but your friends globally vote positively.
5. positive: your own opinion is positive

The calculation of reputation has been split into positive/negative votes, and users now have the possibility to decide what is the required balance of positive/negative votes to make an identity flagged as remotely bad or remotely positive (See Preferences->People). Reputation is now displayed in People using a set of bullets:

Reputation is shown as green (positive), yellow (remotely negative) and red (banned) bullets. Grey means neutral.

The reputation system still governs the distribution of data, but the set of rules have been improved so that distribution channels with anti-spam flag ON only distribute posts from authors with a non negative reputation. This applies differently to signed identities and identities signed by a known or friend node.

Forums with anti-spam activated show a special “Distribution” column. The column only displays a yellow (rep. red) flag for those authors who locally have a limited ability to distribute their posts (resp. who are locally banned):

In forums with spam control, a yellow/red sign appears when the post will not be forwarded to friend nodes, because the authors has a bad reputation or is banned by you.

This means in practice that a newcomer to a forum with anti-spam will only spread his posts to his direct neighbors. The posts will only travel further if the nodes around give a positive opinion to their author. This allows new users of a forum to get known and up-voted, while preventing nasty users to create new identities for each (troll) post, since their data will not go anywhere  beyond their direct friends, which in turn can easily decide to fight accordingly.

Improved GUI

The UI has been cleaned up, with new icons, new design. The settings are now totally interactive, meaning that any change you make immediately takes effect. We tried to improve the first time user experience in many ways. Here’s a non exhaustive list:

• New profile creation has been simplified and all complicated options are hidden (while still here, but at least the average user won’t need to figure out what they mean). The look of it has also been improved. The profile (PGP) passphrase is now asked only once and cached for a few seconds which avoids re-asking the PGP passphrase when signing the SSL certificate;
• all explicit mention of PGP keys have been removed, which are now named “Profile” keys, while keeping the ability to see fingerprints and ascii armored key, for advanced users;
• a Home page has been added as a place to start, make friends and grab your own certificate. This should help newcomers to create their network;
• it is now impossible to supply a badly formed certificate when making friends. Some users in the past tried to supply a PGP key instead of a RS certificate, which would obviously not work. This is now automatically detected when inserting the text in the certificate box.

Finally, a few new options will probably help a lot: GXS now auto-cleans unused groups, and offers the possibility to choose sync/storage time. It’s important to note that some of these values may require old nodes connected to your node to keep sending data. So unless you stay on the default values for GXS sync, it’s better to have your friend nodes upgraded too.

Compatibility with OpenSSL-1.1.0

This step has required some effort! OpenSSL-1.1.0 hides a few structures which where previously open. The changes have been tested successfully on Debian Stretch.

However: at the time of writing this post, Debian Stretch continues to distribute a version of sqlcipher (3.2.0) that crashes when combined with openssl-1.1.0. We cannot do much about this beyond waiting for the patch to reach mainstream. This should apparently not take long, so be patient.

What’s next?

We have a clear roadmap ahead of us. Here a list of what’s going to happen:

• A new abstract system to distribute data has already been designed and implemented. It is currently on test phase. It will be used to distribute RS emails in a 100% reliable way, while hiding metadata, which doubly improves over the current system called GRouter. It will also be used to distribute chat messages when the person you talk to goes offline;
• we plan to unify the messaging system. At the moment, the UI exposes chat between nodes using direct links and between GXS identities using tunnels. In the future, chat will only happen between GXS ids, and RS will find the most appropriate way to send the chat, leveraging PGP signatures of the GXS ids to send data using direct links when available;
• we plan to open the GXS API and document it. This is highly advanced already and a technical paper about GXS will be ready within 1-2 months from now;
• the WebUI currently needs a lot of work. We’re planning to make it nice and complete through Google Summer of Code (See this post).
• android client is on its way (compilation works already);
• cleanup the serialization. For now data serialization in RS is a pain for developers because it does not ensure consistency and requires a great deal of copy-paste code;
• clean the indent over the whole bloody code source (no, seriously. That one was a joke).

Summary

A lot of work has been done on this release! As usual, I would like to thanks all contributors. In addition to official developers, a few contributors regularly submit pull requests (Mr.Alice, Phenom, RetroPooh, Felisucoibi, Sehraf, Zener,…), some tenacious testers have spent hours tracking bugs (Jolavillette, Ghibli, AC, …) and all users in connection to the dev team for their (sometimes very critical) feedback. Thanks also to the trolls (that mother nature has created for some obscure reason) for helping us improve the anti-spam system.