Linode Voluntary distcc Community

@tierra:

What do you think about a voluntary distcc network where anyone can connect up Linodes for sharing CPU time for compiling.

If someone else wants to work out the details of working with Chris so that the bandwidth doesn't count in a DC and work out the details of ensuring two hosts on the same machine are not working on the same problem then I would be happy to contribute my CPU towards the goal. I'm having a couple of problems on my machine right now but when/if you get this going feel free to contact me about joining in. I could even host a webpage or something that would contain instructions/software to show others how to join in.

This would also help out greatly with any hosts with more Gentoo users than average… even out the load between hosts so to speak. The only people that should be complaining about this idea should be the ones that haven't been affected by Gentoo users, but still have a low number of Gentoo users on their host (basically, the ones that will presumably be taking on more of the load from the high profile, high load hosts, not that it would be taking away anything since your gauranteed a minimum amount of CPU time).

Technically speaking though, this is Linode, where your system resources are divided up using UML, and your resources are yours to use however you want. And technically speaking, if I and any other Gentoo user wanted to link up hosts with distcc, nothing about Linode restricts us from doing that already (except that it would count against our bandwidth, which wouldn't be entirely fair since we wouldn't be charging that bandwidth on the TP/HE bills).

As a Gentoo user, I was thinking about this too. I'd be happy to set up distcc on my config and share CPU power.

I've 2 questions :

• wouldn't you need the same gcc version on all nodes in a distcc farm (distcc newbie here) ?

• is the bandwidth between linodes really counted by caker : I would have guessed that the bandwidth counter was setup on the network borders (maybe on a firewall). Technically it can be tricky to monitor bw between hosts in the same DC…

We could organize ourselves without bothering caker if this is the case :

• we already know on which host each linode is hosted (the linode owner's knows),

• we already know in which DC each host is installed (this is on www.linode.com).

The only thing left is to setup a page (Wiki ?) with :

• two lists of linodes (one for each DC) willing to share computing power grouped by hosts (we might need one list per gcc version too),

• groups of linodes already setup to cooperate.

In the list, just add the group a linode is in to help people choose free linodes.

When a new linode wants to use distcc, it adds itself to the appropriate list, choose an existing linode group if there's one with no linode on the same host as its one or just create another group with only itself waiting for others to come in.

There should be a "note" section below signaling linodes being unresponsive in order to clean the lists and the groups when people stop using distcc or go away, have temporary problems, …

What we lack could be an automatic way to firewall distcc.

Distcc users should setup their firewall to allow only hosts in the same group.

This can be bothersome as new users should wait for people to update their firewall rules before using distcc (is distcc able to ignore non responding hosts cleanly ?). Using a Wiki's ability to send notices when a page is updated could help greatly. You could setup a separate page for each group to make things easier.

Are you sure you want code on your system being compiled by other (untrusted) linode users?

I'm a Gentoo user with 2 linodes, and I don't.

@gyver:

wouldn't you need the same gcc version on all nodes in a distcc farm (distcc newbie here) ?

Yes, from what I understand, you have to have the same major and minor version, so for example, mines is currently 3.3.4, I could use distcc with anyone that has version 3.3.x. With most everyone using Gentoo though, most everyone's would be the same stable version (I hope no-one is using masked ebuilds of gcc, shouldn't be right now anyway as 3.3.4 is I believe the latest version out including masked ebuilds).

@gyver:

is the bandwidth between linodes really counted by caker : I would have guessed that the bandwidth counter was setup on the network borders (maybe on a firewall). Technically it can be tricky to monitor bw between hosts in the same DC…

I assumed it was a per Linode BW count… I figured UML per host handled that, but only Chris knows.

@gyver:

The only thing left is to setup a page (Wiki ?) with :

• two lists of linodes (one for each DC) willing to share computing power grouped by hosts (we might need one list per gcc version too),

• groups of linodes already setup to cooperate.

In the list, just add the group a linode is in to help people choose free linodes.

This is a little more complicated than initially thought with GCC versions coming into play, thanks for mentioning it. It would require a little more scripting to automatically assign Linodes to groups so groups were fairly balanced, and it ensured both that Linodes were on different hosts, and also that they have compatible versions of GCC, which brings me to another point: With GCC in play, it almost makes it required (well, it would make it a lot easier to solve a number of our issues all at once) that another service be coded up to run on each participating Linode that would handle a number of things you pointed out.

The first and main reason for a service, would be to handle any firewall configurations that would need to be change/updated with new hosts or remove any non-responsive (whether it be that they stopped using the service, or are trying to cheat the system by blocking access to their distcc, but using everyone else's). Second, it could inform the main control of what version of GCC it's using (which I also just realized that even if distcc has support to check those versions and not use them if they aren't compatible, that it would still be necessary to make good judgements on which groups would work best together). Third, it would listen for group changes from the main control for which hosts it needs to configure itself with. And finally, a fourth reason, to handle any optimizations with GCC settings depending on host settings, like -j#.

Where ever and how ever the main control would be run (whether it be a simple web script, or otherwise), it could also handle reports on service status, keeping information on reports of Linodes abusing the system, etc.

Ok, now that I'm done theorizing on solutions, back to thoughts going on in the back of my head…

This is starting to look a little more complicated than what it's worth. I'm still up for the challenge, but that's a lot of work for a small benefit. I don't even compile that much to make this worth it on my Gentoo Linode. And like dmuench said, I haven't even covered the issue of security with a system like this. It was well worth scratching the itch though and satisfying my curiosity by bringing up a debate on it though.

@tierra:

@gyver:

is the bandwidth between linodes really counted by caker : I would have guessed that the bandwidth counter was setup on the network borders (maybe on a firewall). Technically it can be tricky to monitor bw between hosts in the same DC…

I assumed it was a per Linode BW count… I figured UML per host handled that, but only Chris knows.

Bandwidth between linodes within the same DC is not counted, anything which leaves the linode switches is counted.

Adam

… and Adam I guess… =)

Thanks for the info.

That means it would be possible to setup distcc between two (trusted) Linodes on different hosts without Chris's help, and it would be beneficial for all involved including the people on those hosts.

Was this idea dropped by the wayside or is there a wiki (as was suggested) somewhere?

There is a new Linode wiki now, but it was never setup for this purpose, and this idea was just dropped.

http://www.linode.com/wiki/

If someone comes up with some good ideas for the major problems discussed earlier in this thread, you could start some work on this page:

http://www.linode.com/wiki/index.php/Gentoo

Whatever work that gets done on it could turn around and turn into a pretty big open source project to handle a big secure distcc network, which would be very cool, but would require a lot of work and time that I don't have.

It has been discussed before and there is apparently no secure way of using distcc apart from using it between trusted nodes (which is not what we are talking about). Sorry guys but your linodes will not be in my whitelist. I really think that due to this major problem implementing a nice distcc network is just a Bad Thing.

I can't think of any method to prevent bad binary data being sent back to requesting machines apart from compiling locally and comparing the resulting file, but this would render distcc useless ;) Even if the linode users supplying distcc are really as nice as they say, their linodes can be compromised because of misconfigured firewalls/services/trojans, etc.

Some proposed to send the same source to two or more distcc servers and compare the results. It would certainly decrease a little the risks. But it is not a serious solution as it would eat resources for nothing and not help that much in terms of security (the servers can be all compromised/maintened by "bad boys"). Working with one non-trusted system or hundreds of untrusted systems is the same.

I still think this is really a pity but all we can do is to focus on this big issue before anything else (but as I said, I think there is no real solution). :cry: