US researchers have developed a system called Similarity-Enhanced Transfer (SET), which if put into use with existing file sharing networks, can potentially improve transfer rates by up to 500%. It works by looking for segments of files that either match or closely match the request file. At present, when someone downloads a file on a typical file sharing network, such as BitTorrent, the client looks for multiple sources sharing the exact matching files. However, if the number of sources is scarce, chances are that that most, if not all the parts to make up the download taking place are still readily available, but only as parts of other shared content, usually as a result of people sharing out files under different file names or as part of a compilation.
The researches were quite surprised when they discovered how many files being shared have identical pieces of data between them, although it does makes sense once they looked more closely at what causes this. The SET system looks for similar source files using a technique called handprinting, a method currently being used for filtering spam e-mail and then requests chunks from files that the same or similar to what the client is downloading. If someone downloads a song and tries adding or correcting its tags, their file sharing program will notice that the hash of the file has changed and thus share it as a completely different file, making the share useless to clients trying to download from sources containing the matching original. As a result, SET will help a lot for music, since it will detect songs shared out as just slightly different to what the client is downloading, where the only part of the file that is likely different is the segment containing the song tags or more likely, just the file name being different..
In tests carried out by the researchers, SET helped improve the transfer rate of a 55MB movie trailer by 30%, where content was obtained from shared movie trailers that were found to be 47% similar. Music files saw a significant improvement, with a 71% improvement on a particular MP3 file. The researchers reckon it would be quite straight forward to retrofit BitTorrent to make use of SET.
Further details can be read in this BBC News report. Thanks to GristyMcFisty and DamnedIfIknow for letting us know about this news. DamnedIfIknow added: "This is a technique that I would like people to steal," - How appropriate!! 😉
