Rodi - anonymous P2P

Originally Posted by f%$#MPAA/RIAA

Does anyone know @ what speed your connection has to be to be considered "Broadband"?

2Mbps (from B-ISDN), but I guess in practise anything counts to make stats look prettier.

see lesson 6 in the User Manual for a couple of screenshots http://larytet.sourceforge.net/userManu ... sson%206.0

or run it from http://larytet.sourceforge.net/tryRodi.shtml
Flush cache in your browser if you run Rodi before.
you can also download it directly from http://larytet.sourceforge.net/rodiTimple_shtml.jar
and http://larytet.sourceforge.net/rodi_shtml.jar

it does not make lot but it gives some impression how it is going to look like.
Size of GUI is 160K


any comments, suggestions ?

Originally Posted by carpefile

I'm sorry tm, I have to disagree with you on broadband. While it my be true that in the US, broadband to rural areas is not a big priority to the government, in many many other countries it is.
In both Asia and Europe, broadband is quickly becoming the norm for everyone. If you look at nations like S. Korea and Germany, they have made it a priority to bring broadband to everyone.

Supplying any utility in remote, low-population-density areas is never economically feasible if left up to private enterprise, so this might require a government mandate for it to ever happen. There are a wide range of possible outcomes, so none of us can claim to predict the future with any certainty. I would like to point out the history of cable television as an example. Cable TV has been available in the US cities for decades, and by the early 1970s nearly all metropolitan areas - cities and suburbs - had cable service. However, today - 30 years later - most areas that did not have cable then, still do not have service today, and quite possibly never will. The fact remains that much of the USA is without cable service. Many people without cable service installed their own satellite dish, and the same might be required for them to access broadband. But currently, the cost of a 2-way satellite dish and its monthly service cost is simply too much for most people.

The railroads, multilane highways, telephone, and electricity services were pushed by government-funded programs that sought to make it accessible to everyone nationwide. Maybe the same thing will happen with broadband internet. Or maybe it will follow the same model as cable TV. (I hope not, but it's a possibility that we must consider.) Anyway, broadband upload speed is generally not increasing, so a "broadband" service having 3Megabit downstream bandwidth may have only a 128K or 256Kbit upload, putting it closer to dialup speeds in upload bandwidth.

Originally Posted by larytet

Is block size the only problem you see which prevents you from using the network ?

Because having a 4MB non-resumable block size is going to kill any hope of dialup users using the network, that is the only critical factor that I can see. Everything else such as java bugs, lack of GUI, etc. will be worked out in time.

Originally Posted by larytet

I will change publishing in a way that client will choose reasonable block size automatically as a function of file size. for smaller files publisher will tend to make blocks smaller.

The TigerTree hash used in some protocols such as G2 seems to do this automatically and creates very small blocks when hashing small files. BitTorrent also seems to use a variable block size. The obvious advantage of using small blocks is that file releases can spread much faster due to partial uploading.

Originally Posted by larytet

think that fixed size of hash in the eDonkey "url" is a file hash not blocks hashes, but i am not sure.

Larytet, here is a whitepaper on eMule you might be interesting in reading:

http://leibniz.cs.huji.ac.il/tr/acc/200 ... _emule.pdf

"When calculating the file ID the file is divided in parts each 9.28MB long. A GUID is
calculated separately for each part and then all the hashes are combined into the unique file
ID."

Rodi's hashing tables seems to be more complicated than most other P2Ps, which use only a single (final) hash value that contains all the sub-hashes and (if applicable) the sub-sub-hashes hashed together into a single value.

I think the important thing is that there be a standard of some kind established so that everyone sharing the same file will be sharing it in identical blocks. That way resuming interrupted downloads from different users, multisourcing, and file swarming become possible
.


The new screenshots look good. I plan to be testing this out next.

Rodi hashes are very simple in reality. I do not try to calculate hash of hashes, but simply hash of every block and hash for the whole file.

Type of the hash is part of XML string publisher sends to the downloader. It is going to be fairly easy to make a change in the scheme of calculation of the hashes. Because i use XML string when deliver block and file description such change most probably will be backward compatible.

more screeshots here - http://larytet.sourceforge.net/images/userManual/
It takes about 6-8 hours to add a table to the GUI. To reach basic functionality i need 4 more tables or may be 3 days of work. + i need two additions in core, related to the hashes, may be a day or two.

Then i will try to run the limited set of test cases i developed so far - hopefully another day or two, and beta is out.

I am well ahead of my initial time table - end of Apr, 05

This GUI is nothing more but another proof of concept. I am in no way going to compete with Azureus or eMule. I hope that people who knows to do this better than me will join the project at some point.

This is looking very nice. Having an easy-to-use GUI in the beta release should really help spread its popularity to the mainstream P2P users. That's great news that Rodi will be finished very soon.

With any open source project, the key to getting other developers onboard seems to be completing it in an advanced stage where it is fully functional. Far too many SourceForge projects - even those being based on excellent ideas and innovative concepts - seem to fade away and lie dormant because they never got completed to the point of being consumer-ready standalone applications.

Rodi hashes are very simple in reality. I do not try to calculate hash of hashes, but simply hash of every block and hash for the whole file.

What I meant was that other P2P hashlinks use a single hash that many other hash values can be extracted from - I guess one of the advantages of using a treehash - so the file hash (containing all the subhashes) can be posted as a single 32 digit string and yet contain very many values of separate individual subhashes within it. The very small size of the complete hash 'package' means that it can easily be distributed to many other users in a single small UDP packet.

The P2P "backup" model

Besides IP spoofing - which many people cannot perform - another P2P concept that I think can be very useful is the "backup" model as demonstrated by Scatterbrain Backup and the upcoming Peerio Data backup. In this model, network users will automatically (involuntarily) "download" files and part-files from unknown users, so that a publisher's files get automatically distributed throughout the network and cached by users, similar to the way peers in anonymous networks cache data from other (3rd party) users that pass through them.

One advantage of this push-process file distribution is that on a small, infant network - which tends to remain idle much of the time - files will get distributed without any user interaction and everyone who is connected will be automatically uploading and downloading 100% of the time. Another advantage is that because users are sharing files that they neither purposefully downloaded (or are even aware of their existence) they cannot be held legally liable for them.

What I meant was that other P2P hashlinks use a single hash that many other hash values can be extracted from - I guess one of the advantages of using a treehash - so the file hash (containing all the subhashes) can be posted as a single 32 digit string and yet contain very many values of separate individual subhashes within it. The very small size of the complete hash 'package' means that it can easily be distributed to many other users in a single small UDP packet.

Hash package is relatively small when comparing with the file size and depends on the block size. Also computation times for such hash tree can be significant.
Apparently small hash in eDonkey links does not contains lot of usefull information, because block size is too large.
You think that hash trees is some magic trick where using two 128 bits hashes you can check hashes of 10 or 100 blocks. In reality this is not the case and not even close.

this is good explanation for Merkle Trees http://www.open-content.net/specs/draft ... ex-02.html

And again it does not come without cost. Somebody somewhere should calculate and eventually deliver all or most of the hashes to all participating peers.

Not to say that i am against it. There is pros and cons.

I think can be very useful is the "backup" model as demonstrated by Scatterbrain Backup and the upcoming Peerio Data backup

I have to think about it. I am not sure that there is a scalable solution. Peers should somehow find out what content is rarest and peers should be able to discover all content automatically.

I am not sure also that you are going to be happy about keeping something on your disk, besides files you personally need and use.

In the current design Rodi network is expected to evolve into multiple clusters. In the center of the cluster (or hub) one or more crawlers making data mining for some speicific data, for example, fantasy books. Around it identification or key server(s) belonging to one or more Rodi Houses whose members are trusted publishers or content distributors.
In such cluster search can be done rather quick and search results are going to be reliable.

Originally Posted by larytet

I am not sure also that you are going to be happy about keeping something on your disk, besides files you personally need and use

I wonder - all the self-proxying anonymous P2Ps - Freenet, Mute, Winny, etc. - keep a cache of other user's files. They let each user set the maximum size of the encrypted cache, so users with a small, full HDD can set it small. The idea seems to work for those networks. I was not suggesting that Rodi needs to include 'push-publishing' features, only noting that I think it is one good idea that has not yet been fully developed in P2P clients.

There is an old thread on ScatterBrain backup here:

http://www.p2pforums.com/viewtopic.php?t=4342

Unfortunately Braintech (Owen) had to put the project on hold while attending college. Although Scatterbrain Backup is not (yet) opensource, Braintech said this in the last post:

The project is on hold for the moment while the designer and programmer - (that would be me) - completes his master's degree... I may have a chance to work on it over the summer, but I am hoping someone will see the technology and write something more user-friendly, contact me

So if BrainTech would be interested in having someone else complete Scatterbrain v.2 - or use the concepts to incorporate into another project, that might look promising

Originally Posted by larytet

Hash package is relatively small when comparing with the file size and depends on the block size. Also computation times for such hash tree can be significant.
Apparently small hash in eDonkey links does not contains lot of usefull information, because block size is too large.
You think that hash trees is some magic trick where using two 128 bits hashes you can check hashes of 10 or 100 blocks. In reality this is not the case and not even close.

this is good explanation for Merkle Trees http://www.open-content.net/specs/draft ... ex-02.html

One thing I do not understand from that paper is why it states:

"Thus the authors recommend a segment size of 1,024 bytes for most applications, as a sort of "smallest common denominator", even for applications involving multi-gigabyte or terabyte files. "

Because even for a dialup connection, 128KB or so is (in my opinion and experience) really as small as the block size ever really needs to get - considering the additional overhead of small blocks. (a 1024 byte block would get transferred in a flash)

I don't quite understand all the details of the mechanics of file hashes. In most P2Ps that use sub-filehashes, the download starts as a zero-filled file of the same size as the original, and the blocks are gradually filled in. In a system like Scatterbrain, the file's blocks are broken up and distributed to different users taking up an amount of disk space required for the block only, not for the entire file - so there are no zero-part filled sectors, and I assume that separate hashes would need to be kept of all the blocks, rather than just the final (summation) hash.

there is no direct link between zero fill and hash. p2p clients tend to create (preallocate) disk space because it reduces disk fragmentation and removes chance of file write failure because of the "disk full" error.
Rodi client preallocates disk space too.
p2p client after restart checks the data and does not attempt tp recalcuate hash for zero only blocks, because hash of the zero block is known value. I have yet to implement this in Rodi - there is no support for download resume after power up today.


P2P applications you mentioned (Freenet, Mute, Winny) have very specific niche - anonymous networks. They are not designed to be scalable or extremely fast.
Rodi is designed around BT ideas. I only removed (made them optional) tracker and index file.

I couldn't get Timple to connect to Rodi core on local machine. Shouldn't there be somekind of connect button in the connect settings?

-edit-

Oh it seems you need to "touch" the port setting for it to try connect even if the port is correct by default. But evene then it gets stuck on Attempting to connect...

save files http://larytet.sourceforge.net/rodiTimple_shtml.jar
and
http://larytet.sourceforge.net/rodi_shtml.jar
on your disk in the same directory. makes sure that version is 0.2.01 (look on the top of the window). start both in any order
It should connect automatically

if still any problem please in Rodi Core window
type command dbg pp and send me output. something like this

Code:

/dbg pp
Look Req             = 0          Look Ack total       = 0          Get data req         = 0          
Get data ack         = 0          Header size err      = 0          Packets total        = 0          
Look notify          = 0          Parsing failed       = 0          Unknown IE           = 0          
null pool            = 0          Null IE reference    = 0          IE parsing failed    = 0          
NAT                  = 0          Data transfer        = 0          Unknown msg          = 0          
DT no protocolID     = 0          DT Unkwn protId      = 0          DT notify            = 0          
chat                 = 0          mng                  = 0          signatures           = 0          
pub key found        = 0          signature ok         = 0          No msg processor     = 0          
Illgl prot id        = 0          Unkn prot id         = 0          
/

[edit]Or forward port 31211 on your firewall, run Rodi core and send me your external IP address as reported by http://larytet.sourceforge.net/ipRange/ipRanges.php
(email mailto:larytet.6103180@bloglines.com), .keystore file, and keystore password if different from default. i will try to figure out what is going on.

Did you run setup wizard ? you have to (see lesson 6.0 http://larytet.sourceforge.net/userManu ... sson%206.0) create keystore file first, then restart Rodi Core and Timple. private key in the keystore file is the only thing protecting your client. If you don't want to restart Rodi Core, you can use command

Code:

conf key addPub rodiMng mng yourPublicKey

This command will register public key yourPublicKey and give the owner of the private key mng (management) access rights.

P.S. i put more screenshots in the first section of lesson 6 and attached your question down there. hopefully the answer is right

Ok got it working now, I think my version of rodi was obsolete. You might want to add a connect button because you need to "touch" the port setting before it connects, even if the port is already correct number. Like, you need to put a wrong port, then back the right port before it connects...

Ok what's next. I did the 6.1 and published a file for testing. Then in 6.2 I added host 127.0.0.1 255.255.255.0 31211 (this section needs a button for removing hosts).

When I click the Look link the window remains empty?

there is a bug - i am looking for it this very moment.
first of all reporting of connection status is not always correct and there is another issue with setting different port numbers.
fix is comong soon

thanks for try you give it

Originally Posted by larytet

thanks for try you give it

No problem. If someone sees all that trouble to develop a new toy for filesharers, the least we can do is to help testing it, right?

Sorry if this has been asked before, but what would you say is the key strength of Rodi over other P2P applications?

Why should one choose to use Rodi instead of say, DC++ or Bittorrent?

What does Rodi offer for an average filesharer?