8 processor system

[Devin Johnston]

I'm looking at XI for my next system, but Boxx also sells their APEXX system and I'm not sure which one would be best. I want to use the Opteron 870's to power it, any recommendations as to video card or hard drive? Right now I've got the Nvidia GeForce 7900 GTX listed, but I'm not sure which one would be best when using 3D Studio. I've also got a 74GB 10,000 RPM SATA drive onboard with an optional 160GB 7200RPM SATA as a backup drive.

[ihabkal]

is it cost effective in our business to get a $35000 system?

 

 

a dual dual core 285 costs around $5000, you can get a good render farm with 7x4=28 cores (physical cpus) instead of 8

28 or 8, what do you think?

 

will vray and brazil use all 8 cores in the Appex? this system seems to me more like a marketing jig from boxxtech than a cost effective machine...

 

besides when you spend all that moeny on the APpex, one extra grand for the Quaddro 4500 is nothing.

[Devin Johnston]

I've decided to go with the 4 processor system instead of the 8 because it's about half the price. One thing they don't tell you is that the 8 processor systems require a different operating system called Enterprise, and it adds about $3000 to the price tag. I also learned that the bus speed of the Athlon processors is twice as fast as the Opterons and there cheaper. I'm still going with the Opterons because they don't make a dual processor board for the Athlon chips but I'm going to keep my eye on them.

[tecton3d]

I've decided to go with the 4 processor system instead of the 8 because it's about half the price. One thing they don't tell you is that the 8 processor systems require a different operating system called Enterprise, and it adds about $3000 to the price tag. I also learned that the bus speed of the Athlon processors is twice as fast as the Opterons and there cheaper. I'm still going with the Opterons because they don't make a dual processor board for the Athlon chips but I'm going to keep my eye on them.

 

i have nothing against boxx as I love mine, but after reading your other post and realizing that you're expanding your studio... getting a quad socket (8core) system doesn't make sense either technically or finaincally ~ IMO. Again, boxx is a great company but I'm not convinced that such a system would be any better than a couple of fast workstations and a rack of dedicated nodes.

 

depending on how many people you hire for your work, build at least two screaming workstations (RAID arrays, quadros, 4gb ram... etc) and have the "farm" for all to acess and run test renders...

 

here is another option for a rendering solution: http://tyan.com/products/html/typhoon_b2881.html

basically a boxed system comprised of 4 - dual CPU Opteron mobos (16cores) each with thier own hd, psu, ram,... etc. Its more like a condensed rack server than a workstation, but if you have a decent workstation this is a viable option.

 

edit: I realize your bosses are probably anxious to start cutting checks for this expansion, but AMD is to release Socket F sometime within the next couple of months and that would allow for QUAD CORE cpu's. So if you hold of until they come out, when quad core opterons come you - a simple cpu swap will net you 2x rendering power... FWIW

[Devin Johnston]

i have nothing against boxx as I love mine, but after reading your other post and realizing that you're expanding your studio... getting a quad socket (8core) system doesn't make sense either technically or finaincally ~ IMO. Again, boxx is a great company but I'm not convinced that such a system would be any better than a couple of fast workstations and a rack of dedicated nodes.

I would normally agree with you but after looking at my options and the software available to me I've decided to try and use Maxwell as my core animation program. That may seem stupid to many of you but I've been testing it for quite a while and I think I can make it work. One big drawback is of course speed, this system would give me a massive increase in speed and in conjunction with my network I'd be able to produce some of the very first Maxwell animations.

[tecton3d]

I would normally agree with you but after looking at my options and the software available to me I've decided to try and use Maxwell as my core animation program. That may seem stupid to many of you but I've been testing it for quite a while and I think I can make it work. One big drawback is of course speed, this system would give me a massive increase in speed and in conjunction with my network I'd be able to produce some of the very first Maxwell animations.

 

all eggs in one basket?

 

that system will not give you anthing a dual dualcore workstation + dual dualcore rendernode will... actually the more CPU's you put on a mobo, the less efficient it becomes and the 2.0ghz speed of the 870's isn't really all that fast and once you get into poly-heavy scenes 2.0 will be a little restrictive.

 

When you start to scale your pipeline up and since you're using Maxwell, you want to get the most out of every processory in the pipe... you'll have to look at it this way esp since Maxwell is so slow. All those cores in a single workstation won't do anything but sit there and idle until you're ready to render and with maxwell being so slow the best thing to do would be to aution-off the render to a farm so you can continue to work. A 4-way (4cpu/8core) system isn't giving you the ideal efficiency... of course you have 8cores sitting next to you @ at your desk but still... just hotglue a nodes to your current workstation and VIOLA ~ boxx apexx4/8... I digress but I hope you get my point. With the cost of the apexx4 (23k), you can buy 6 dual dualcore opteron Boxx renderboxx (24 opteron cores + other network comps + workstation) and still have 6k to invest in a workstation or workstations (I'd go with 2 midrange 5400 or 7400's). With the apexx, you will have 16 less cores to render on than going the rendernode route, which with MW is not a good idea plus if something happens to the Apexx you are totally F&*%$@ and your pipeline stops!

 

when it comes down to it, the more processors/cores the better, no matter how they are configured, don't let the WOW factor of having 8 cores in a system mislead you into thinking it's going to make Maxwell suddenly a fast renderer.

 

... if Maxwell is going to be the horse you go riding into battle on, you better have a hell of a rendering army behind it

 

good luck

 

edit: just out of curiousity ~ is the choice of the 8core system the convenient result of NL's 8 core license limit?

[Devin Johnston]

My question to you would be have you ever used Maxwell?

 

It's not like Final Render or Vray in that you can use a bank of render nodes for distributed rendering, with Maxwell you either do everything on your main system or send it out for cooperative rendering where you get no feedback until that operation is over. You’re correct that a 4 or 8 CPU system wouldn’t be as fast as 4 or 8 individual computers, but the difference here is the application and how it works. If I'm going to be using Maxwell for animations I'm going to need as much horsepower in a single computer as I can afford in order to clear the noise as fast as possible. Using a rendering farm in this case would make the job harder and much less convenient even though they may be faster in the end it's really about interactivity with the application that I'm worried about.

[tecton3d]

My question to you would be have you ever used Maxwell?

It's not like Final Render or Vray in that you can use a bank of render nodes for distributed rendering, with Maxwell you either do everything on your main system or send it out for cooperative rendering where you get no feedback until that operation is over. You’re correct that a 4 or 8 CPU system wouldn’t be as fast as 4 or 8 individual computers, but the difference here is the application and how it works. If I'm going to be using Maxwell for animations I'm going to need as much horsepower in a single computer as I can afford in order to clear the noise as fast as possible. Using a rendering farm in this case would make the job harder and much less convenient even though they may be faster in the end it's really about interactivity with the application that I'm worried about.

 

that's pure rubbish of NL to not offer DistRend with such a slow renderer... I guess there's always hope that an update could provide such a feature:confused:

 

since no DR is the case (which still has me floored!!), it will be worth upgrading to 875's or 880's plus your viewports and non-smp actions will be a little snappier

 

hell, I was even thinking of getting maxwell assuming DR was included...

 

all I can say is good luck to you Devin

[Devin Johnston]

The reason DR doesn’t work with Maxwell is because the image can't be split up into buckets, the way the MLT engine works makes that imposable. They do have cooperative rendering which will use X number of computers to render out a scene. The drawback to this is that the final images aren’t created until the end so there is no visual feedback.

[tecton3d]

The reason DR doesn’t work with Maxwell is because the image can't be split up into buckets, the way the MLT engine works makes that imposable. They do have cooperative rendering which will use X number of computers to render out a scene. The drawback to this is that the final images aren’t created until the end so there is no visual feedback.

 

I gotcha ~ I remember something like that being posted a while back but it still seems a bit crazy... no wonder Maxwell hasn't been fully adopted by the industry yet. You'd think there could be a feature, even if a seperate application, that could reassemble the rendered info from all the DR nodes and present it to you visually for feedback giving you the chance to tweak....

 

any mentioning of such a feature in upcoming releases? I'm all ears:D

[mtutaj]

maxer,

 

Have to talked with Boxx? any chance that they could do a sample rendering on one of those systems using M~R so that you could see how much it will help you? using one of your scenes, so that you can make a better decision. I have to sort of agree with tecton3d. that is a lot of money to invest in one solution with no guarnteed results.

 

Let us know,

 

Mike

[Devin Johnston]

It's been talked about but so have a bunch of other things that have never happened. That would be a feature that is low on the priority list for NL; they still haven't finished implementing all the promised features from a year ago.

[AJLynn]

Warning: longwinded technobabble rambling follows.

 

The more I think about it, the more I think the distributed render time issues Devin posted on already are inherent to the technology. Bucket renderers have a discrete unit that the software can define many of and pass them around, using more CPUs as needed. A Maxwell node must operate on the entire scene at once. It distributes computation over the pixels in the scene, either randomly or in a pattern, and each pixel gets a certain amount of sampling done on it.

 

I would venture that the beta used a random distribution, while 1.0 either used a patterned distribution (presumably one advantage to this is that it gives an order of execution and multiple computers can begin at different stages, but in the end they all come around to the same points in the cycle so you're going to lose anyway) or had a broken random number generator (it happens). 1.1 is, if I'm reading the material on it correctly, back to randomness.

 

When the scene is recombined, the host computer would need a way to combine the samples together - either by comparing each pixel in each render and taking the one with the highest sampling value or (and this would be smarter if it can be made to work) combining the samples from each render to get a higher value than any one of them. Even if it does this, inevitably some work has been repeated - more than one computer generated duplicate information, and you can't combine two copies of a duplicate to get more information than either one had - so you lose efficiency. The more computers, the more likely you get duplication, the less efficient. So your increase in render speed falls off as some logarithmic function of the number of CPUs and the probability of duplication (which increases at higher sampling levels - this is also true on single computers, hence the falloff in render speed as SL increases).

 

And the more data being combined, the more computation is required to combine it, something that so far probably only concerns Devin and Daros.

 

So the only way to properly leverage an enormous Maxwell farm is to send your jobs in frames, one per computer, not as distributed single frames. The more total CPU power you can get, the faster you can animate, but the more power you can get in a single box, the faster you can do still frames.

[Devin Johnston]

combining the samples from each render to get a higher value than any one of them. Even if it does this, inevitably some work has been repeated - more than one computer generated duplicate information, and you can't combine two copies of a duplicate to get more information than either one had - so you lose efficiency. .

 

This is exactly how it works and without cooperative rendering it would be almost imposable to generate high rez images unless you had a super computer like this 8 processor monster.

[AJLynn]

You've probably had more opportunity to test Maxwell scalability than anybody else, so what do you think - with a sufficiently powerful single machine, can you do high-res renders in reasonable time? Have you done any comparisons with 1.1?

 

There's probably more study that could be done on Maxwell scaling before you buy such an expensive system. Comparisons between 4 hour renders on one thread and 1 hour renders on four threads on a dual-dual, or one that would be interesting would be one thread vs two on a hyperthreading CPU - does it speed it up by 10-20%, or actually slow it down? My concern would be that falloff in performance due to threads duplicating each other's work would make an 8-core machine not much faster than a 4-core, but I have no way of knowing.

[tecton3d]

There's probably more study that could be done on Maxwell scaling before you buy such an expensive system. Comparisons between 4 hour renders on one thread and 1 hour renders on four threads on a dual-dual, or one that would be interesting would be one thread vs two on a hyperthreading CPU - does it speed it up by 10-20%, or actually slow it down? My concern would be that falloff in performance due to threads duplicating each other's work would make an 8-core machine not much faster than a 4-core, but I have no way of knowing.

 

Guiness induced technobabble to follow:

 

but thanks for the post Andrew and Mike... that's my whole issue is perhaps Maxwell isnt made to be used for production yet... I mean think about it, the scaleability you get with Final Render, Brazil, Mental Ray, V-Ray (cough ~ greatest since sliced bread right!) with DR is amazing... with such a potentially good renderer (maxwell) and to HAVE TO BUY A 23,000 computer to run it makes me stop for a second and rethink my intentions as an artist. If each core (at the same speed) is simply recalculating the same thing and running in parallel with the render then what's the point of a multi-core system? Get the fastest P4 system, overclock, and that would by far be your best rendering system... i.e. no duplication of results with each core.

 

they've promised you everything else under the sun... are you sure you belive this:

Maxwell can exploit all of the processors available on your system and can make the work simultaneously on the same render to provide faster renders. For example, on a multiprocessing platform with 8 processors a user can expect up to an eightfold increase in render performance/speed.

 

Maxwell still has me stumped... NL must not want to take our industry seriously to force us to buy 4/8 socket computers to get anything out of thier squiemesh software:confused:

 

I've e-mailed my contact at boxx to see what they say about Maxwell just to make sure you'll get what you're paying for with the Apexx... They'll be happy to sell you 20k worth of computers no matter in what form it comes... : )

 

I just can't get over the idea of scaling 23k worth of computers over a VRay or FinalRender or Mental Ray network... it would be truly amazing! I just hate to see you fight with your chosen technology when you have an opportunity to really break out, if even it is perhaps with another software.

 

i'm confused : (

[Devin Johnston]

You've probably had more opportunity to test Maxwell scalability than anybody else, so what do you think - with a sufficiently powerful single machine, can you do high-res renders in reasonable time? Have you done any comparisons with 1.1?

This issue has been covered on the NL forum many times, the conclusion is that you get about 90-95% scalability when using cooperative rendering. My own tests have shown that for every additional CPU you throw at Maxwell your image will improve proportionally. With a 8 CPU machine (16 processors) it should be possible to produce a high rez interior image over nigh.

 

I know what the Vray people are saying, why spend so much time and money on an engine that's so slow. This is a good question and one that I've been struggling with for quite some time. My conclusion is that the MLT method of rendering will never be as fast as the other methods used by Vray or Final Render. You have to come to terms with that and accept that the software has limitations that only more power will remedy. Once you understand that and look at what’s available now it is possible to use Maxwell in production and with a large enough farm it can be usable. When I think about it 5 or 6 years ago we were all struggling to produce animations with shadows and reflections, which took a lot of time. As computers have gotten better so have the animations and their complexity and only now are they fast enough to take advantage of and engine like Maxwell. I've already invested loads of time and money in this project and it's either going to work for me or fail, at this point I'm going to say that I can make it work and I'm going to give it my best shot.

[AJLynn]

You're making sense, I hope you can get it to work out. I'd love to see what you can do with it given plenty of work and resources.

[Devin Johnston]

Well I might have hit a wall; apparently some of the software I want to run on this machine isn't supported on Windows Enterprise. That means that it might or might not run at all, and if it does there might be problems. This is really getting aggravating, I think I find a solution to a problem and two others pop their ugly heads up.

[mtutaj]

thats always the problem at being at the cutting edge of technology. Most of the bugs and problems have been fixed for the items used by the masses. and will take a while to filter down to other users. and of Course when Vista comes out, there will be new server programs as well then.

 

MIke

[manta]

Well I might have hit a wall; apparently some of the software I want to run on this machine isn't supported on Windows Enterprise. That means that it might or might not run at all, and if it does there might be problems. This is really getting aggravating, I think I find a solution to a problem and two others pop their ugly heads up.

 

What software ?

[Devin Johnston]

I was told by the developer that Vue 5 Infinite probably wouldn’t run on Windows Enterprise. He suggested that I try running it on Windows 2003 Server and if it ran on that then I'd have a chance of it running on Enterprise. I'm going to test that today and see what happens.

[Devin Johnston]

We have decided not to go with the 8 processor system due to the questionable compatibility of the programs we run and the operating system. We're going to go with AMD Athlon 64 X2 5000+ dual core chips because the have a 2 Ghz bus speed as opposed to the Opteron with the 1 Ghz bus speed. They also use the DDR2 ram which operates at 1Ghz instead of the Opteron ram that runs at 400Mhz. I'd say 10 of these baby's will make quite a nice farm.

By the way I’ve been getting quotes from Boxx and Xi, Boxx is about 40% more expensive for the same unit which I find outrageous. They keep telling me that you get wonderful customer service but were talking about $1000 more per unit here! How can anyone justify buying form them when their prices are so out of wack?

[radii]

I use a Boxx dual Xeon 3.2mhz, 3 BG ram, ATI Firegl at work, and a Xi P4 3.4 mhz, 1.5 GB ram, Quadro FX 540 at home. The Boxx is obviously faster, it is not a quantum leap though. Hardware wise, the Boxx feels a lot more solid and well made (Xi case can easily be twisted by my 2 year old son :-) ) The Xi though, has been running day and night flawlessly for more than 2 years and still feels "spiffy".

I'm not shure if the added price in a Boxx system is justified. A lot of their costs seem to come from advertisements and these "bussiness alliances", aka "... such and such software and renderer is certified on a Boxx.." Nothing but a big bunch of BS, if you ask me.

For private use (and maybe even at work), I will definetly go with an Xi system again (and not Boxx).

[Devin Johnston]

Yea that has been my conclusion as well, there trying to make up the cost of advertising and R&D by jacking up the prices of their products. I've got a feeling that there not going to be in business for long if they keep that up.