Release notes for v0.6.6

This release took even more time than the previous ones. This is for one simple reason: we changed lots of core features of the software (e.g. token system, notifications, etc), which had us re-implement some GUI parts taking the opportunity to make them more efficient or re-designed to a better layout (e.g. channels, boards). In the end, lots of bugs have been introduced in the process and it took some time to reach a perfectly stable product again. While we decided to stop developing new features in November 2020, it took us 4 months to fix all the bugs, mostly thanks to a really good user-feedback. As you will see, it was totally worth it.

1 – New channel and board layouts

Channels benefit from a new layout that is more suitable to their usage. Two different visualization modes are possible (top right blue button, left to the search field): grid or list. Use the double image below to compare both views:

Both in channels or boards, the new visualization system is now based on an abstract item model (instead of manually inserted items as it was in v0.6.5) making it much faster to load and offering a lot more possibilities for how to display the information.

2 – Short certificates

Up to v0.6.5, Retroshare used certificates containing the SSL id, the PGP public key and naming information. It appears that we can afford to only send the PGP fingerprint instead of the full key, without losing any security. Here’s why (“<=” means validated by) :

Full certificate (old method, a.k.a. type-1):

SSL handshake <= certificate PGP signature <= profile being listed as friend.

Short certificate (new method, a.k.a. type-2):

SSL handshake <= SSL Id listed as friend

In both cases the handshake is eventually validated by information provided by your friend using a supposedly secure transmission (hand-to-hand, encrypted email, etc). This part of the security model is still up to the user.

Once a node is connected, it will supply its own PGP key using discovery (as before), which will be checked to match the fingerprint in the certificate and then be marked to be used in type-1 handshake later on for future connections. The new handshake type is indeed only used at the first connection. Whatever happens wrong in the process (mismatch profile keys fingerprint, mismatched SSL id) will prevent further connections.

The security model of Retroshare connections is based on the fact that the original certificate you get comes from the right person. It can be the SSL id (new type) or the profile public key (old type): the security level is the same. If you trust the SSL id, then any information received after connecting to this Id can be trusted as well, which includes the profile public key. Once the profile key is received, all connections (including to other nodes of the same profile) will be validated by the same profile key.

The new home page therefore displays a much shorter certificate, which happens to be small enough to e.g. fit into a QR-code, since it only contains the node and profile names, the SSL Id and profile fingerprint, and the contact local/external IPs and ports (for clear nodes):

With Tor nodes, the certificate may be a bit longer since the Tor v3 onion address is rather longer (56 bytes) than the PGP fingerprint (20 bytes).

3 – Compatibility with Tor v3

Retroshare v0.6.6 supports Tor v3 keys. The change is backward compatible: new Tor v3 nodes will connect to 0.6.5 nodes without any change, provided that the 0.6.5 node you’re connected to uses a version of Tor that supports v3 keys too, which means posterior to v0.3.2.9.

Tor nodes have never been easier to setup: at the node creation time, select “Hidden node over Tor” instead of “Standard node”, and fill your profile name and password. Retroshare will automatically create a hidden service for you and configure itself to use it (as soon as Tor is installed in the system – retroshare only looks for a Tor executable). Using Tor has many advantages:

• Connections through firewalls are extremely reliable and fast since Tor is very hard to filter. A Tor node may therefore be a good way to keep Retroshare working in a heavily filtered network;
• your IP is hidden to everyone including friend nodes;
• a lot of features related to clear nodes are not needed (IP Filters, Relay, DHT, UDP, etc) which makes the software lighter and certainly more stable.

Tor v2 keys will automatically deprecate after July 2021. This is not our decision: it depends on Tor stopping to support these keys. Consequently, it is recommended to upgrade your existing Tor v2 node to Tor v3. Fortunately, this is rather easy: just delete (or move away) the directory ~/.retroshare/HID06_[your SSL Id]/hidden_service/. Then start retroshare. It will automatically create a new (v3) service address. Your friends will still accept you when you connect to them (your profile key hasn’t changed). They will however not be able to locate you themselves (your Tor address has changed), until they receive your new Tor address either by discovery, or by yourself sending your certificate. This mostly happens to clear nodes for which the only direction of connection is from the clear node to the Tor node. In summary:

• delete the hidden_service directory and re-launch Retroshare
• send again your Retroshare certificate (See Home Page) to friend clear nodes

More information: Tor v2 deprecation timeline

4 – Auto-cleaning of unsubscribed channels, forums, boards and circles

All GXS groups (understand e.g. a forum, a channel or even a circle or an identity) now record the last time they are advertised by friends. It is therefore possible, when a group is not subscribed, to safely delete it. The rules are the following:

• unsubscribed forums, channels, boards are auto-deleted after 60 days if not seen at friend nodes either (meaning friends are not subscribed)
• subscribing to a group resets the counter

Some services however have specific rules:

• Identities record the last usage time that depends on how they are used by other services. The People tab comes with a list of usages, which timings correspond to regular sweeps of the databases for signature and consistency checks.
• Circles have specific auto-subscribe rules to consistently benefit from this auto-cleaning strategy:
  1. reminder: circle membership for an identity is based on two necessary conditions: (1) the identity is “invited” by the circle administrator (which is part of the group data), and (2) the identity requests membership (requests are group messages);
  2. a circle is only subscribed (meaning propagated to friends) if a membership request from one of your identities is posted to it, which includes also when actively “leaving” a circle after previously being a member, or if you are the administrator of that circle.
  3. circle membership requests (or leaving requests) are kept for 1 year. A membership request therefore needs to be renewed after one year.

In the early days of circles (previous releases) it was possible to create huge circles inviting all possible identities, which would automatically propagate everywhere. Thanks to the above rules, this is not possible anymore. Such circles that may already be here will be unsubscribed and disappear from your circle list after 60 days provided that none of your friend nodes request membership (The remaining time is visible in the tooltip over unsubscribed circles as “last seen: XX days ago”).

5 – Better notifications in Circles/Channels/Forums/Boards

The previously named “Log” tab is now called “Activity”. It still shows a configurable summary of events in the software (new forum messages, connection attempts, etc). We added a few of them especially for Circles:

• Peer requesting membership or leaving a circle;
• peer owned by your node being invited or rejected from a circle by the circle admin.

…and Forums: moderators added/removed

These notifications are now available because we added a more accurate description of the changes in group and message data in the GXS internal event handler.

6 – Various GUI tweaks

The list of small changes/improvements in the Qt interface is rather large. You will certainly notice the following incomplete subset:

• A new tab for identities has been added in Statistics;
• home page has been improved for more readability and a better compatibility with short certificates;
• forum pinned posts behavior and look has been improved;
• messages layout have been improved with spam box filter, attachments box filter and colored tags folders

7 – Certificates signed with SHA256 instead of SHA1

This is a basic security issue: primary collisions with SHA1 are not very far away and switching to a stronger hash makes SSL handshake more secure. Consequently all certificates created by Retroshare now use SHA256, while keeping compatibility with SHA1 signatures.

For the same reason, on debian Buster, openssl security level defaults to to 2 which precludes the use of SHA1, originally used to sign RS certificates. As a consequence, on these systems, locations created with RS previously to Jan. 2020 will not start unless one reverts the security level to a value of 1, which we only do when loading the certificate with default parameters fails and for OpenSSL versions later than v1.1.1. This backward compatibility tweak is meant to disappear from Retroshare code in 1 year.

SHA256 will be used automatically on new nodes (even with the same profile). Upgrading an existing node is unfortunately not currently possible.

8 – Deprecation of the token system

The existing asynchronous token system has been replaced by a blocking API, that proves much easier to use since it removes the need to have multiple functions calling and receiving events. Still, the blocking calls can take some time and the UI still needs to do them in a thread with the following constraints:

• UI modifications should still happen in the UI thread;
• the UI shouldn’t wait for blocking API calls.

Consequently we use a combination of two new built-in methods (that will simplify even more with C++ async calls): RSThreads::async() which executes a lambda in a separate thread and qtthreadsutils::postToObject() which sets a given lambda to be executed in the GUI thread using the normal Qt event queue. Eventually the code looks like this, as extracted from the People tab:

 void IdDetailsDialog::showIdentity(const RsGxsId& mid) {

 RsThread::async([this]()
 {
     std::set<RsGxsId> ids( { RsGxsId(mId) } ) ;
     std::vector<RsGxsIdGroup> ids_data;
 
     // call the blocking API
     if(!rsIdentity->getIdentitiesInfo(ids,ids_data))
     {
         std::cerr << "failed retrieve identity"<< std::endl;
         return;
     }
     RsGxsIdGroup group(ids_data[0]);
 
     RsQThreadUtils::postToObject( [group,this]()
     {
       /* Here it goes any code you want to be executed on the Qt Gui
        * thread, for example to update the data model with new
        * information after a blocking call to RetroShare API 
        * complete */
 
        displayIdentity(group);
     }, this );
 });
 }

Note that no mutex is needed since everything happens in the UI thread.

9 – New notification system

Adding notifications to a service is now extremely simple: a new entity named rsEvents handles all events. In order to use it, one needs to declare his own type of events, with any private information (hash and event code in the example below), and supply the serial_process method using Retroshare core macros. This way the event can be sent anywhere in libretroshare, plugins, and even to clients connected using the JSON API (e.g. web interface, python scripts, etc):

struct RsFileTransferEvent: RsEvent
{
     RsFileTransferEvent()  
       : RsEvent(RsEventType::FILE_TRANSFER),  
         mFileTransferEventCode(RsFTEventCode::UNKNOWN) {}
     ~RsFileTransferEvent() override = default; 

     void serial_process( RsGenericSerializer::SerializeJob j,
                       RsGenericSerializer::SerializeContext& ctx ) 
                          override 
     {     
         RsEvent::serial_process(j, ctx);
          RS_SERIAL_PROCESS(mFileTransferEventCode);     
         RS_SERIAL_PROCESS(mHash); 
     } 
     RsFileTransferEventCode mFileTransferEventCode; 
     RsFileHash              mHash;
 };

Whenever an event needs to be sent, the code only needs to allocate one event and send it to postEvent (asynchronous through a queue) or sendEvent (direct blocking call):

auto ev = std::make_shared<RsFileTransferEvent>();
ev->mHash = hash;
ev->mFileTransferEventCode = RsFTEventCode::DOWNLOAD_COMPLETE;
rsEvents->postEvent(ev);

Catching the event, for instance in retroshare-gui is pretty simple as well. The client (here the file transfers GUI part) only needs to register an event handler for the given event type (here FILE_TRANSFER):

mEventHandlerId=0;
rsEvents->registerEventsHandler( [this](std::shared_ptr<const RsEvent> event) 
{ 
    RsQThreadUtils::postToObject( [this,event]() 
    { 
        handleEvent_main_thread(event); 
    }) ;
}, mEventHandlerId, RsEventType::FILE_TRANSFER );

Note the use of postToObject (here again!) which allows the event to be handled in the GUI thread, since rsEvent will likely call the lambda that is registered as event handler from a different thread (here, the file transfer loop).

10 – End of retroshare-nogui (don’t panic!)

The former headless retroshare server has been replaced by a new executable named retroshare-service, to which one talks using the retroshare built-in JSON API. It is rather cheap and can be ran on Android phones as well as screenless servers.

Compilation simply requires the appropriate flags:

> qmake CONFIG="rs_jsonapi"

An example of application to this interface is the new WebUI developed during Google Summer of Code 2019. See https://github.com/retroshare/RSNewWebUI, and add “rs_webui” in the compilation flags.

While retroshare_service can be used to create new nodes, the web UI doesn’t currently offer to do it. Setting up a retroshare server with retroshare_service+WebUI therefore needs to copy the full ~/.retroshare/ data directory prior to launching retroshare service:

> ./retroshare_service/src/retroshare-service -U list -W

While ‘-U’ would normally be followed by the node ID you want to start, using ‘list’ instead will show up the list of available nodes and ask for the profile password before starting it. Another limitation is currently that retroshare_service does not use the automatic Tor configuration code (which depends on Qt) and setting up a headless Tor node requires some manual tweaking in order to (1) configure Tor system-wide to start a hidden service, then (2) create a Tor node with the Qt UI, choosing the manual Tor configuration option and setting the correct ports according to what was supplied to Tor. Option ‘-W’ asks retroshare_service to set a new password for the webinterface.

The webinterface is now entirely supplied by Retroshare. In particular its html and JS files do not need anymore to be installed on the client where the WebUI runs.

11 – Other backstage improvements

Some backstage improvements are also worth mentioning:

• A new cache for metadata allows a much faster loading of forums/channels/boards;
• GxsId tracking has been improved, which can be seen by more accurate usage statistics in People->Person;
• RTT/bandwidth management and service priorities has been improved after lots of careful tests. The RTT now between two friends keeps low values;
• memory usage has improved, but is still not perfect. Some systems tend to allocate too much memory on threads and RS uses lots of them;
• improved debug output thanks to new RsDbg(), RsError() and RsWarn() macros;
• consistently switched licence to AGPLv3 (GUI) LGPLv3 (libretroshare) in prevision to application for Debian packaging.

Last words

As usual, only congratulating all the testers and pull-requests providers will not make them true justice. This release would not be here without them. So many thanks to all of them!

Since we reached some rather stable status, we would now like to

• Apply for debian packaging (we will need some sponsoring!)
• cleanup deep indexing (currently using xapian) so that it can be shipped in the next stable version
• improve the reactivity of GXS sync
• make our Tor automatic system Qt-free. This is not so easy because the current code heavily depends on signal/slot connections.