(cheap) renderfarm config

[TomasEsperanza]

I appreciate the input guys (I guess I ransacked the thread somewhat )

 

Nikolaos, I was thinking along similar lines. The CPU cooler fan is in the way, and a major reconfiguration of this system this is not something I need to get into. The 6000 viewport performance is excellent actually, but for GPU rendering which I don't employ at present, I predict that the burden of calculations may cripple the displays somewhat. Also, I don't know how much difference a new GPU will make to the display performance, this is something I'm going to find out. If the Slate material editor is faster, that'll be worth every penny.

 

I have a Dell U3011 (30") and a Dell 2412M (24"), scenes can be heavy (vegetation etc) and materials complex.

 

I currently distribute the rendering to the dual CPU node (a laptop gets some buckets sometimes too), which combined with the workstation is pretty quick. This can work with RT too (in CPU mode), but...

 

If I am going to use RT more extensively (instead of regular test renders), whilst developing materials and lighting, then I presume that moving to GPUs for feedback may be more efficient an investment than more CPU nodes. I can still make use of the various CPUs I have for overnight renders and animations (laptop included), but to have a really responsive RT window open all the time would smooth the workflow in the development stages (lighting and materials particularly).

 

Yes, hopefully the three fans on the strix will do. The case has three 60mm square grills near to GPUs, so that's auspicious. Finally, looking at the Workstation Specialists choice of components I'd say the quality of hardware all looks pretty decent. I Googled FSP-group.com, and have confidence in the PSU for now. No probs so far anyway, but if performance is throttled once I've added the GPU then... we'll see.

 

Best I get on with the dusting then ! ;p

[Nikolaos M]

Voila, then!

Keep the 6000 for viewports, since it is adequate for your jobs, and put the load of the Real Time Rendering on the 980Ti. Just check the P9X79WS's manual to see what's the next pciex16 slot you should use to install the 980Ti. In this generation of motherboards things were more free concerning the right order of gpu installation, I think. I suppose you could choose any slot you like (the WS version has plenty of them).

 

I guess you're familiar with Vray RT's capabilities (gpu), because it doesn't fully support the Vray and Vray RT's (cpu) features.

http://docs.chaosgroup.com/display/VRAY3MAX/V-Ray+RT+Supported+Features

 

FSP are good psu's in general, with a low failure rate and a golden 750W should be enough for an oc'ed 3930K and a 980Ti doing gpu RT. But you'll see things in action.

 

(A good dusting will probably do your system good in overall temps too. I usually see a 1-2°C improvement if I happen to dust my case after a long period).

[TomasEsperanza]

Hey, the Strix just arrived! (that was quick!) Awesome

[TomasEsperanza]

OK, I've got a problem, and seeing as you've been so helpful thus far, I'm wondering if you can make sense of this for me?

 

 

It looks to me like:

 

1. The Quadro is running off the only available power cable, which splits into two 6+2pin PCI-E plugs.
   
2. The Quadro only uses one of these two 6+2pin plugs.
   
3. The Strix requires two 8pin PCI-E plugs.
   
4. So I could run the Strix off the two 6+2pin plugs, if I unplugged the Quadro.
   
5. Problem is: I want to leave the Quadro as is, and the free adapter that came with the Strix seems useless in this situation.
   
6. So I guess I need to buy a cable to go from the PSU to the Strix.
   
7. Looking at the PSU, the outputs are not ideal, but I'm not sure.
   
8. From what I can see, the PSU has one 8pin EPS 12V output, and four 5pin Molex outputs.
   

 

Any idea what cables I'm going to need?! (' Hope I don't need a new PSU).

 

Thanks in advance

[Nikolaos M]

We must know the exact model of the psu to see if it's a single railed psu and how many Amps it can provide.

 

http://www.nvidia.com/content/quadro/pdf/quadro-power-guidelines.pdf

The Quadro needs ~18A and the 980Ti ~38A (these are average system requirements, not psu only), i.e. you need a psu that can provide enough amperage for both cards plus the cpu. The wattage is more than enough (assuming it's a 750W in deed) but it's not the only factor to consider when powering a system like that. A single 12V rail psu with at least 50A capability should be OK, I think.

 

If the psu specs are OK, then you can try this solution. The Quadro can run with either a single 6+2 (=8)pin cable or dual 6pins (check the manual to be sure). So, take the Strix adapter that splits into two 6pins and power the Quadro with it. You can use the free 8pin 12V connector on the psu to do that. Then, connect the two 8pins (6+2) to the 980Ti. And you should be OK (I hope so... )

 

I checked for FSP 750W gold, but the models I found don't match yours (they have 4 pcie connectors, so...).

 

I sense you have a problem in deed. And that's my fault. I should have checked the psu specs before giving advice for a gpu addition.

Edited January 15, 2016 by nikolaosm

[TomasEsperanza]

I sense you have a problem in deed. And that's my fault. I should have checked the psu specs before giving advice for a gpu addition.

 

No my Friend, you were simply being helpful, and that's very much appreaciated. HUG

 

My detective work tells me this is an "FSP Aurum CM Gold 750 W" (I'm 95% sure). This Review seems to tell all, (if only I understood it better!)

 

http://www.techpowerup.com/reviews/FSP/Aurum_CM_750W/2.html

 

I am studying these documents now. In the meantime if you can advise, you're certainly more au fait with this than me. Ta!

 

If possible I'll go to a PC store and buy whatever cable/s I need, today.

Edited January 15, 2016 by TomasEsperanza

[Nikolaos M]

I see. This psu has 4x12V rails and each one of them contributes with 18A. Only two of the rails (12V3 and 12V4) are destined to power the pcie devices, so that gives a combined 36A exclusively for the gpus. This could be enough, but I'm not sure. You'll need a cable like this:

 

 

It's an 8pin to 2x8(6+2)pins. Connect it to the free 8pin port on the psu and with the other two 8pin ends you can power the 980Ti.

 

Good think is that ATX (24pin), EPS (4+4pin) and all the other peripherals use the other two 12V rails without interfering with the power that goes to the gpus.

 

I hope it works. If not, I'm afraid you should buy a new psu that suffices for the task.

[marcinwuu]

Hi Joel,

 

My specs are currently:

 

Workstation - "Workstation Specialists WSX6.1" (ATX)

CPU: i7-3930K Six-Core @ 4.20GHz

MOBO: Asus P9X79 WS

RAM: 32GB Ram

GPU: Quadro 6000

PSU: 750W 80Plus Gold

OS: Win 7 Pro on SSD

 

Render Node - "Workstation Specialists RS-D2600" (Stackable)

CPUs: DUAL Intel Xeon X5650 Six Core 2.66GHz (Intel® Turbo Boost @ up to 3.06GHz)

RAM: 48GB Ram

PSU: 500W 92%+

OS: Win 7 Pro on SSD

 

The Quadro 6000 is pretty dated now (Cuda 448, plus I'm running two large monitors), hence adding GPU/s seems appealing. But I can't afford to be frivolous so... ?

Maybe a 980ti (Cuda 2816) for RT, and just use the Quadro for displays? (they're both 6GB).

 

Can you recommend a course of action?

 

I'm not likely to purchase and return, that is too much hassle (for me).

 

Hey,

 

I am sorry for possible thread hijack, but i have noticed you are using dual xeons x5650 in render node. Now and then i see some servers popping up with dual xeons and something like 60 or more GB of RAM.

 

Do you think guys that it would be a good solution to get server like that for the render node (vray cpu)? One example could be:

 

Dell PoweeEdge R710 Dual Proc Intel® Xeon® CPU X5660 @ 2,8Ghz

4x 1TB disks + 2x 200GB disks

Proc Intel® Xeon® CPU X5660 @ 2,8Ghz

RAM 44GB

iDrac 6

DVD

2x PSU

 

Thanks in advance

[Nikolaos M]

It depends on the cost. If you try to find used parts in good prices, then it would be a decent solution. I wouldn't justify anything more expensive than 500-600€ (400-450£) for a dual 1366 Xeon node, since a modern render node based on a 5820K processor can be built with aprox. 1000€ and it would be faster than a dual 5660 node and more upgradable in the future:

1 x OCZ Trion 100 240GB, SATA (TRN100-25SAT3-240G)

1 x Intel Core i7-5820K, 6x 3.30GHz, boxed ohne Kühler (BX80648I75820K)

1 x Crucial DIMM Kit 32GB, DDR4-2133, CL15-15-15 (CT4K8G4DFD8213)

1 x ASRock X99 Extreme4 (90-MXGVL0-A0UAYZ)

1 x Raijintek Themis Evo (0P105245)

1 x Fractal Design Define S, schallgedämmt (FD-CA-DEF-S-BK)

1 x Super Flower Golden Green HX 550W ATX 2.3 (SF-550P14XE (HX))

Total cost: 1009€ (it needs a cheap gpu too, a second hand card would be ideal).

 

The old Xeons could be oc'ed too with base clock oc, but I don't know if this was possible on 2P motherboards of the time. A single 5660 could reach 1000cb in Cinebench with an oc of ~4.0GHz, but building two separate nodes with it could be slightly more expensive than what I could accept for a system so old and with no upgrade perspective in the future.

[Nikolaos M]

No my Friend, you were simply being helpful, and that's very much appreaciated. HUG

 

My detective work tells me this is an "FSP Aurum CM Gold 750 W" (I'm 95% sure). This Review seems to tell all, (if only I understood it better!)

 

http://www.techpowerup.com/reviews/FSP/Aurum_CM_750W/2.html

 

I am studying these documents now. In the meantime if you can advise, you're certainly more au fait with this than me. Ta!

 

If possible I'll go to a PC store and buy whatever cable/s I need, today.

 

Hey Thomas. Have you managed to resolve this issue?

[TomasEsperanza]

Well, the cables weren't available in any local stores, so I just settled with using the Strix, (and don't want to get into replacing PSU for this system).

 

Currently I'm experiencing "Buyer's Remorse". (My own fault entirely). I can't get RT to work now, so I am experiencing no practical performance gains using Max, V-Ray, etc. and... this card is much noisier than the Quadro 6000; (to be expected, but I overlooked that).

 

Lesson re-learnt: CG is time consuming even if you invest in new hardware, so best to appreciate current set up and keep working if you can.

 

I think I might need to re-install V-Ray to get it to work properly. (Or I could re-install my old card and smash the Strix with a hammer! lol) Seriously though, the new GPU may yet redeem itself, if I can figure out why RT isn't playing nice.

 

RE: Renderfarm config:

 

I will probably eventually re-purpose current WS system as an additional CPU render node, once it gets too old, and is time to build a new WS. (This way all the WS components are of similar generation/age).

 

Marcin - A couple of second hand xeons, with an SSD in a render node, sounds promisingly efficient to me; I don't regret mine.

[marcinwuu]

Hey,

 

They are efficient imo as well.I really like this video

 

What i am thinking here also is licensing. With dual xeons you would save 1 vray licence, since you don't need second pc for similar performance. Anyway, will be looking for opportunity

[Nikolaos M]

I don't know if this is helpful, but check it out:

http://docs.chaosgroup.com/display/VRAY3MAX/Set+Up+V-Ray+RT+GPU

 

"Driver versions for V-Ray RT GPU CUDA

Driver version 350.12 has been tested to work most reliably. If possible use that version

Driver versions 352.86 and 353.06 DO NOT WORK with V-Ray RT GPU CUDA. If you cannot use the recommended driver version (350.12) you can use the following workaround:

Use the following environment variable:

VRAY_CUDA_DISABLE_PTX_OPTIMIZATION=1."

 

Thomas, try a clean installation of the driver version mentioned above. I think it's a driver's issue.

 

As for the noise thing, install this software (almost all Nvidia users do, me included) https://gaming.msi.com/ru/features/afterburner and create a custom fan curve to suite your needs (and ears...). For low loads you can make the gpu to run almost inaudible. THe Strix is famous for its silent operation. I can't believe it's more noisy than the old Quadro with the stock cooler.

Edited January 19, 2016 by nikolaosm

[TomasEsperanza]

As for the noise thing, install this software

 

Aha, Yes, in order to download the lastest driver I ignored the CD, that's why.

 

Top tips! Cheers

[alenagic]

Hi Nikolaos. Maybe u can help me.

I buy 980ti for RT.

And i have some problem.

vray3.2+max2015

When i try RT. with CUDA ENGINE, RT not working, but working with OpenCL. Im using latest nvidia driver 361.43, is this maybe unsupported driver for cuda ? How to use this "VRAY_CUDA_DISABLE_PTX_OPTIMIZATION=1." where to put this if i want to use latest driver ? or i need to downgrade driver to 350.12 ?

Im new in RT

 

PC system: WIN 8.1

CPU 4790K

RAM 16GB

GPU 980Ti

PSU 750W

[Nikolaos M]

Hi Alen. I'm not a regular Vray RT user. I've just done some testing with it, out of curiosity. My gtx 780 worked just fine, so I didn't have to try different drivers etc.

 

The "VRAY_CUDA_DISABLE_PTX_OPTIMIZATION=1.", as far as I know, is an environment variable that one could add inside the Advanced System Settings of the OS (Control Panel>System>Advanced System Settings for W7). I think the first part is the variable's name and the second part is the value. But I'm not sure for anything of the above. You should get help from someone who has already tried these tweaks. Can't help more with this, sorry.

[alenagic]

I try this no luck.

I have some error in vray rt log window.

"error: .src/ocl_tracedevice.cpp : CUDA error 700

Do u know maybe what is this error ?

I dont try to downgrade driver to 350.12, i need to use latest driver for other things.

[TomasEsperanza]

I buy 980ti for RT.

When i try RT. with CUDA ENGINE, RT not working, but working with OpenCL. Im using latest nvidia driver 361.43, is this maybe unsupported driver for cuda ? How to use this "VRAY_CUDA_DISABLE_PTX_OPTIMIZATION=1." where to put this if i want to use latest driver ? or i need to downgrade driver to 350.12 ?

 

Same here; I tried the various drivers but only the OpenCL engine works, not CUDA. Weird.

 

I thought the same thing about "VRAY_CUDA_DISABLE_PTX_OPTIMIZATION=1."; what is it? How does it work? etc. I just used OpenCL instead.

 

Would CUDA be quicker?, maybe a bit? ...I guess we can't test that at present.

 

 

(My conclusion is that: the 36 CPU threads I was using until now, are (approximately) twice as quick as the single 980ti using OpenCL, and so even if I bought a second 980ti I wouldn't get faster renders than I had with the CPUs (Damn!). Therefore, for such a tiny "renderfarm" (as a lot of freelancers have), buying a GPU for rendering is only rewarding if you were previously only using a single CPU workstation to render.

 

For example: in my case, I could have bought three GPU's to improve my overall times by about 30%, or I might have used that same money for another CPU render node. There wouldn't be much difference in it, until you start scaling it up significantly, and then other factors come into play so.... I guess it's best to just pick a side and invest. Starting from scratch I suppose GPU rendering would be a bit cheaper.)

Edited January 25, 2016 by TomasEsperanza

[Nikolaos M]

Cuda would be quicker than OpenCL for these Nvidia gpu's. That's what Chaosgroup recommends for Vray RT:

It is recommended to use the CUDA engine on nVidia GPUs.

There must be a solution to this. Officially, the biggest Maxwell gpu that's supported is gtx 980, but I see rendering tests with TitanX's and 980Ti's here and there, so someone must have figured this out.

Just one naive question. You both use 64-bit OS, don't you?

[TomasEsperanza]

You both use 64-bit OS, don't you?

 

Yes, 64-bit Win 7 Pro here.

 

When I hit render, if RT is set to CUDA, the frame buffer comes up black and then a windows pop-up says that "V-Ray Standalone has stopped working". So with the 980ti, the CUDA engine crashes V-Ray.

 

I'm sticking with CPU for now, but if anyone reads this and knows what the problem is, it would be good for people know.

[Simon Moir]

I have two 980tis in my workstation (driver 358.91), Max 2016 with V-Ray 3.3 works fine with either or both cards using Cuda...

[Nikolaos M]

Maybe a clean installation of this driver could resolve these issues. When i say "clean", I mean that all files associated with previous drivers must be deleted first. Sometimes the systems gets confused when there is a conflict between new and older driver files. And one more thing. It's important that the ram amount that's needed for the rendering is not larger than the gpu's vram.

 

Simon, are your rendering times satisfying with the 2x980 tis?

Edited January 26, 2016 by nikolaosm

[Simon Moir]

They're OK... I use it mainly for V-Ray material previews etc. I prefer rendering on our small farm for finals. If RT output was identical to regular V-Ray I'd use it more often, I personally find it slightly limited, but it does keep getting better and better. We're also starting to look at Unreal / Unity / Lumion etc.