mult clients

log in

Advanced search

Message boards : Number crunching : mult clients

Previous · 1 · 2
Author Message
Profile STE\/E
           
Avatar
Send message
Joined: 28 Mar 10
Posts: 642
Credit: 3,866,375
RAC: 462
Total hours: 20,095,425
Message 2105 - Posted: 21 Mar 2014, 13:53:51 UTC

I'm okay on 18 of 19 Box's, the 19'th one won't work for whatever reason ...
____________
https://signature.statseb.fr/sig-1323.png
https://stats.free-dc.org/badgesbanner.php?cpid=13a87c3a303bcdca4ba0ed600daebb6b

EmSti [BlackOps]
 
Avatar
Send message
Joined: 3 Dec 13
Posts: 4
Credit: 68,710
RAC: 0
Total hours: 150,617
Message 2106 - Posted: 21 Mar 2014, 13:59:26 UTC - in response to Message 2039.

Hmm, ok... I guess if that's the way you want to run it. I've never really had any problems running multiple WUs within BOINC, the max I've got is 6 (4 CPU, 1 GPU, 1 NCI) on one host. I manage the heat on systems where that's a problem without any incident.



I too have run into situations that 2 clients solve. Mainly around supporting multiple gpus. Some projects have run into issues with 2 or more NVidia in the pc and not being able to run on all of them from one client. Also video cards with 2 engines (e.g. gtx 690) are some times a problem for projects and I split them off to different clients, when I want them both on the same project. Finally multi gpu issue that additional clients solve is when you have cards of different ability. For instance you have a titan and want to run 3 wus/gpu and you have another NVidia card that can't handle it. Simply put the titan in one client, the other card in another and setup up different app_config or app_info xmls as needed.

There are even more reasons I can think of in addition to the ones Nate1 listed below, but I think we have painted a sufficient picture of why. Only adding my thought hear in the hope it helps someone out.

Profile NATE1
 
Avatar
Send message
Joined: 17 Dec 11
Posts: 14
Credit: 56,733
RAC: 0
Total hours: 35,791
Message 2107 - Posted: 21 Mar 2014, 19:30:25 UTC

also you ever seen the size the client state can get?
take a look next time after the client has been running for a while with a number of projects attached...

every time the client state file needs to be updated it has to create what is called client_state_next.xml which has all the updates to it
it then deletes the client_state_prev.xml
renames the client_state.xml to client_state_prev.xml
then renames the client_state_next.xml to client_state.xml
most people never see the client_state_next.xml
but you let the client_state.xml hang on to all the information that is in there and then create another client state next file the same size or larger.
your killing your ssd, hdd don't suffer that much. but the ssd will die an early death with all the changes...
mult clients keep the client_state.xml down to a min. size
it also allows the projects apps to run faster, not having to dig into all the other information about all the other project before it finds what it needs to update. the smaller the client state the less time it takes to create the client state next and the faster it can get back to processing the project tasks
lower overhead.

Link
 
Avatar
Send message
Joined: 20 Jun 12
Posts: 114
Credit: 327,572
RAC: 87
Total hours: 170,954
Message 2109 - Posted: 23 Mar 2014, 18:54:30 UTC - in response to Message 2107.
Last modified: 23 Mar 2014, 18:59:03 UTC

it also allows the projects apps to run faster, not having to dig into all the other information about all the other project before it finds what it needs to update. the smaller the client state the less time it takes to create the client state next and the faster it can get back to processing the project tasks
lower overhead.

The BOINC client writes the client_state.xml, not the project apps. And it also does not need to dig into anything, it reads the client_state.xml once at startup, than it just writes new versions of it when needed and deletes the old ones.
____________
.

Profile NATE1
 
Avatar
Send message
Joined: 17 Dec 11
Posts: 14
Credit: 56,733
RAC: 0
Total hours: 35,791
Message 2110 - Posted: 23 Mar 2014, 21:29:49 UTC - in response to Message 2109.

it also allows the projects apps to run faster, not having to dig into all the other information about all the other project before it finds what it needs to update. the smaller the client state the less time it takes to create the client state next and the faster it can get back to processing the project tasks
lower overhead.

The BOINC client writes the client_state.xml, not the project apps. And it also does not need to dig into anything, it reads the client_state.xml once at startup, than it just writes new versions of it when needed and deletes the old ones.


that has always been one of my biggest problems, I am unable to get my point across
no it does not write the app.
it may read the client_state at startup but it has to create a client_state_next file with all the updated information for the client_state. I looked into it. had to do a little digging myself.
when the boinc client goes to update the client_state file it creates a client_state_next file, in which it places all the updated information.
to put it simply it copies the client_state to the client_state line for line only changing the lines that need to be updated or added to.
when done it deletes the client_state_prev file, renames the client_state to the client_state_prev and then renames the client_state_next to the client_state
if you only have one project attached to the client and don't keep 30 days results and whatnot in it, I have some client_state file less than 10 K
now a client that has 12 projects attached to it with 30 days of results the client state can reach well over 10 million bytes.
now coping 10 milling bytes from one file to another and changing whatever needed to be update then doing the delete and renaming takes a heck of a lot of time...
also for a period of time you have 3 10 million byte files on your mass storage device..
so which is quicker? 10 k bytes copied and updated or 10 M bytes copied and updated?

well whatever...

Link
 
Avatar
Send message
Joined: 20 Jun 12
Posts: 114
Credit: 327,572
RAC: 87
Total hours: 170,954
Message 2111 - Posted: 23 Mar 2014, 22:30:07 UTC - in response to Message 2110.

now a client that has 12 projects attached to it with 30 days of results the client state can reach well over 10 million bytes.
now coping 10 milling bytes from one file to another and changing whatever needed to be update then doing the delete and renaming takes a heck of a lot of time...

30 days cache is very unusual, so if that's the issue, than simply set it to a lower value, with that many projects you don't need such a big cache, if one project runs out of work, others take over.

Also I remember before we had WU limits at SETI, people were reporting client_state.xml sizes of over 40MB, the only issue they had was uploading the 10+MB scheduler requests to the servers.



also for a period of time you have 3 10 million byte files on your mass storage device..

Having three 10MB files on the hard drive should not be an issue since at least two decades and for sure not today. So what's your point with "10 million bytes"?

It also doesn't matter, if you split the one big client_state.xml into many small ones, the total size is still the same or actually many small client_state.xml files even a bit larger than one big because the host/user information has to be stored in each of them.
____________
.

Profile NATE1
 
Avatar
Send message
Joined: 17 Dec 11
Posts: 14
Credit: 56,733
RAC: 0
Total hours: 35,791
Message 2112 - Posted: 23 Mar 2014, 23:30:33 UTC

12 clients 1 ci project attached to each client (12 projects)
each client has a client_state.xml of say 10 k
each client running on pref.s only 4 run at time for 8 hours each.
then those 4 stop and the next 4 and then the next 4
from 00:00 to 08:00 projects a,b,c, and d run
from 08:00 to 16:00 projects e,f,g, and h run
from 16:00 to 24:00 projects j,k,l, and m run
cycle repeats.
when 4 of the projects are running then only the client_state for those 4 projects needs to be updated and not all of the client_states will update at the same time.
so at most your only copy/update/rename 10 k per project or around 40 k if by chance all 4 projects just happen to need updating at the same time.

1 client 12 ci projects attached
client state 100 k, no repeated header/footer information
every time one of the projects need to have something updated in the client_state you will have to create another 100 k file each and every time.

messing with 40 k of data is a lot faster messing with 100 k of data.
maybe total bytes = 120 k but your only having to deal with at most 40 k at a time.

point is not all of the clients are running the projects apps at the same time.
and no I wasn't talking about 30 days cache.
and before you say it, why would you run it that way, that's dum.
I didn't say I did, just trying to provide an example.

so, I try to explain what I'm talking about, but like my last post, nobody really gets what I am trying to say 99.9999% of the time..

Link
 
Avatar
Send message
Joined: 20 Jun 12
Posts: 114
Credit: 327,572
RAC: 87
Total hours: 170,954
Message 2113 - Posted: 24 Mar 2014, 7:48:33 UTC - in response to Message 2112.
Last modified: 24 Mar 2014, 7:50:13 UTC

when 4 of the projects are running then only the client_state for those 4 projects needs to be updated and not all of the client_states will update at the same time.
so at most your only copy/update/rename 10 k per project or around 40 k if by chance all 4 projects just happen to need updating at the same time.

1 client 12 ci projects attached
client state 100 k, no repeated header/footer information
every time one of the projects need to have something updated in the client_state you will have to create another 100 k file each and every time.

messing with 40 k of data is a lot faster messing with 100 k of data.
maybe total bytes = 120 k but your only having to deal with at most 40 k at a time.

What's missing in your equation is the load generated by 12 WUProp apps (or 4x WUProp running at the same time). Also running more than one the BOINC client at a time costs additional CPU resources (I think we don't need to talk about HDD space here). From your describtion I understand, that 12 BOINC clients are running 24 hours a day and suspending and resuming their project apps according to their settings. You really think, that this way you save some resources? No, you might save some cycles when writing the client_state.xml, but you'll use even more somewhere else, it's not like all those BOINC clients are doing nothing just because their project apps are suspended. They will for example send scheduler requests to the WUProp servers once an hour, that involves not only writing a new version of the client_state.xml, but also sched_request_wuprop.boinc-af.org.xml and sched_reply_wuprop.boinc-af.org.xml + of course some CPU cycles. And that's just one thing they would be doing.



and no I wasn't talking about 30 days cache.
and before you say it, why would you run it that way, that's dum.
I didn't say I did, just trying to provide an example.

so, I try to explain what I'm talking about, but like my last post, nobody really gets what I am trying to say 99.9999% of the time..

Well, maybe you need better examples than. Examples, where it is clearly the user's fault, that something isn't working properly, are simply showing, that it's possible to misconfigure BOINC. And that's a well known fact.
____________
.

Profile NATE1
 
Avatar
Send message
Joined: 17 Dec 11
Posts: 14
Credit: 56,733
RAC: 0
Total hours: 35,791
Message 2114 - Posted: 24 Mar 2014, 10:01:26 UTC - in response to Message 2113.

From your describtion I understand, that 12 BOINC clients are running 24 hours a day


no

Well, maybe you need better examples than. Examples


yes

BobCat13
 
Send message
Joined: 6 Apr 10
Posts: 27
Credit: 520,202
RAC: 47
Total hours: 545,161
Message 2431 - Posted: 22 Oct 2014, 19:25:44 UTC - in response to Message 2088.

I authorized multiple clients on the same host.
Please, notice you must use an app_info.xml file for client which communicates on port different than 31416.
Please, find below an example for linux 64bits.
<app_info>
<app>
<name>data_collect_v4</name>
<user_friendly_name>Data collect version 4</user_friendly_name>
<non_cpu_intensive>0</non_cpu_intensive>
</app>
<file_info>
<name>data_collect_v4_4.13_x86_64-pc-linux-gnu__nci</name>
<executable/>
</file_info>
<app_version>
<app_name>data_collect_v4</app_name>
<version_num>413</version_num>
<plan_class>nci</plan_class>
<avg_ncpus>0.01</avg_ncpus>
<max_ncpus>0.01</max_ncpus>
<cmdline>-p 31416</cmdline>
<file_ref>
<file_name>data_collect_v4_4.13_x86_64-pc-linux-gnu__nci</file_name>
<main_program/>
</file_ref>
</app_version>
</app_info>

You should replace 31416 by guirpc port.

Is it still necessary to use app_info.xml? I'm not running multiple clients, but am running boinc on a port other than 31416 and found it only needed the following in an app_config.xml:

<app_config> <app_version> <app_name>data_collect_v4</app_name> <plan_class>nci</plan_class> <avg_ncpus>0.01</avg_ncpus> <cmdline>-p 31417</cmdline> </app_version> </app_config>


This requires a boinc client version of 7.2.39 or higher.

Profile marmot
     
Avatar
Send message
Joined: 13 Dec 15
Posts: 174
Credit: 2,240,572
RAC: 556
Total hours: 5,165,651
Message 3559 - Posted: 30 Dec 2015, 6:55:42 UTC - in response to Message 2431.
Last modified: 30 Dec 2015, 6:59:18 UTC


Is it still necessary to use app_info.xml? I'm not running multiple clients, but am running boinc on a port other than 31416 and found it only needed the following in an app_config.xml:

<app_config> <app_version> <app_name>data_collect_v4</app_name> <plan_class>nci</plan_class> <avg_ncpus>0.01</avg_ncpus> <cmdline>-p 31417</cmdline> </app_version> </app_config>


This requires a boinc client version of 7.2.39 or higher.


Thanks!
This is just what I needed for splitting into two clients on my 6-core and getting WUProps to read the second client.

The fetch algorithm doesn't read app_config files and so doesn't calculate in the <project_max_concurrent>2</> for ATLAS on an 8GB RAM machine or the 2 core max that the project at vLHC dictates. Nor does the work fetch seem to account for high resource setting project that is out of work for days, which is much more common than I expected. DENIS is reworking their servers, Citizen's grid seems to be off for the last month, etc.

Looks like the federal funding for BOINC is gone so the three dedicated coders (Anderson + others) are off while volunteer coders are taking over. I'm hoping they go back to the times of version BOINC 5 and 6 when it was quite easy to running multiple clients and the Windows registry wasn't relied upon and each client had it's own data directory along with a command line switch to tell the client where to find it's data directory. At least the windows installation engine should be fully aware of installing multiple copies and allow startup of multiple managers if they continue to rely on the Windows registry.

Oh I dream of a BOINCMGR.exe GUI that has commands on the left which allow you to assign a project particular cores of your computer, a resource allocation system that doesn't care about RAC but only the amount of hours a project gets on it's assigned cores and a work fetch that never idles a core when work is unavailable on a dedicated project but suspends the projects and then goes down the list from highest resource share to lowest and eventually grabs 1 temporary WU from the next project with work available to keep the cores from idling.

Previous · 1 · 2
Post to thread

Message boards : Number crunching : mult clients


Home | My Account | Message Boards | Results


Copyright © 2024 Sebastien