Local vs. Network setting for rendering

[Jan Havlicek]

Hello everyone,
what is in your learned opinion better for studio of 2-3 people who want to share library of their assets (textures, 3D objects, etc, used via Connecter or Project Manager) - CLOUD (e.g. dropbox) or own SERVER machine?
We were almost ready to buy some NAS server, but then it occured to me (and please correct me if i'm wrong), the files sync'ed via dropbox on my workstation are faster option for rendering, than detached files on some other machine, where network bandwith comes to play. I have only supericial understanding of network data exchange, so i'm not sure how factors like "100Mbit or 1Gbit adapter/modem/router", coaxial cable quality, etc would manifest themselves during for instance real time rendering (where i assume is any potential delay caused by slower loading of textures visible the most). Am i missing something? We work primarily in 3dsMax with Corona.
There are articles on NAS vs Cloud subject, but it is the connection with everyday rendering use, that interests me here. Thanks anyone for helpful comments.

[Scott Schroeder]

Local files are always going to be faster that files pulled from a remote cloud server. And by remote, I mean something that isn't on your local network so if you lose internet, you cannot access them. I have a NAS storage that is local to my home network with a Diskstation machine. I have it hooked into a pretty robust router as I was planning on being a permanent remote employee before this work from home experience started.

As far as transfer speeds go, yes, you want the fastest you can get with a router without going too overboard. You also want to use CAT5 or CAT6 cables in your network. Those speeds will show up when you are trying to copy/load files. In terns of real time, this could either be longer scene load times or drops in FPS if the scene needs to load the file while you are exploring it. In terms of standard offline rendering like corona, this will be longer scene prep times as the files are pulled across the slower network.

For me, the advantage of the NAS system is that storage for that is dirt cheap. I have 10 terabytes right now of the typical platter type hard drives. Eventually, the plan is to go all SSD in there, but that's not a huge priority over sheer storage size. Also with the NAS system, is that backups are easier and if a single drive goes down, it can switch stuff over to the other working ones in there. Your 3D assets are the lifeblood of your work, so to keep them all local as the same machine you work on is a little too risky for myself.

If your studio is all working from home, you could all use Dropbox or something similar to sync but that will require one of you to upload all of your assets to the cloud. Which that could be a while depending on the size. Then, other people will need to download the same amount of files which again could be a long time. If you all have fast internet that doesn't have low data caps, then this should be negligible. Another option is to purchase large external storage drives for them all, do a copy of your asset directory on all of those drives, then give them to your team. This way, you avoid the initial upload/download and you only use Dropbox to sync changes.

[Francisco Penaloza]

First, it is incredible how this Corona Virus experience has changed many thing related on how we work.

If your team will be under the same roof, having a small or medium server is the way to go. Small servers not only can store your projects and assets files but also could for example manage your Network licenses or render farm administrators. With any networking you need to plan a head, 100 Mb wont cut it for today standards, 1Gb or more if you can, should be the way to go, thankfully equipment is not that expensive, so a robust router and a good switcher should give you a great start. The benefits of working with centralized assets in a team environment is priceless so as your team grow, the network system will pay by it self. it also simplify the way you do backups.

NAS system also are great for small teams, now the big difference with 'Servers" is the CPU capacity and software, if it is a small team a NAS may be enough, but if there is much demand from everybody the little CPU in the NAS may become your bottle neck. 

You mentioned 3D Max and Corona render, so what real time tool do you use?? any ways any off line render or real time render will load the assets in to memory, so networking lag is only at the beginning, after app is rendering or running then the load time is small.  But that depends on the app.  For example if you are rendering an animation, Backburner will take longer to load the scene the first time, then each frame will check for updates only so the other frames will process faster. Now you need to consider the writing time too, and here is other difference between a Server and a NAS system.

Having said all that, all these only apply if you all are under the same roof. If you guys are remote, then things get a little more complicated. My self and team been testing many ways to share and keep up to date all our work and assets, and it been hard, I haven't found any app that I can be 100% satisfied.

Dropbox, One drive and similar, they all work about the same, a copy in the cloud and a copy in your computer/ computers the main problem is capacity, when you pass a size threshold you get charged a lot more or the app just can't handle.   Also there is that small detail that for example Dropbox and One drive like to create their own folder structure as a root, so if you are sharing a project with a co-worker the path will be different,  there are work around but it it gets messy very quick.

As Mr Elvis  mentioned having a local drive copy for everyone who works from home may be a good options, because you could map it with the same letter and everyone won't have problems, but keeping everyone up to date and synced files then it get complicated.

We are testing now using local storage with Sync app. this is not 100% bullet proof but so far is what it worked for us.

So we all have a external local HHD/NAS with all projects and all assets, then using the Resilio Sync app to keep all assets always up to date and sharing some projects, so everyone in our team has exactly the same files, this also create a nice redundancy in case someone's house explode. For large rendering we are testing could companies.

Hope this helps.

and I would be curious how other people are dealing with this.

I haven used Connecter I would like to know how it really work, the times I tested never could figure out how to add all out assets, without having to re organize all folders and icons.

 

 

[Jan Havlicek]

Thank You both for comprehensive answers.

We are preparing for remote model of work, thus the more troublesome setting. Regarding the real time tool - corona has "Interactive rendering" option:

https://youtu.be/IhIoQ-F1X9E

It's vital tool for us during material tweaking and lightning selection. And while I wouldn't mind potential few seconds delay caused by network transmission during "big" X-hours long final render, these same seconds would be pain during this interactive rendering.

I do appreciate the easily expandable capacity of own NAS/server, thats a big plus. Safety-wise (backup) I hope the whole idea of cloud (dropbox,..) storage means, that in event of losing all our workstations, we still have our assests somewhere online - thus safe as well.

@Francisco PenalozaI shall investigate the Resilio App, thank You. If I understand correctly, every one of your workplaces has its own NAS machine, be it so even in cases where that particular workplace has only one workstation? And then there's local connection between NAS (which is constantly syncing online with others) and workstations (which loads assets from that NAS)? If is it so, in that already mentioned scenario with only 1 workstation in 1 workplace, wouldn't the NAS be unnecessary? (though it provides posibility for adding more data space and potential more workstations there in future).

Connecter seems nice so far, but tricky in one regard. It appears to be designed for cloud synced libraries of your assets. That you can do for free. However the settings files for such established "connecter workspace" cannot be placed in these (or any) cloud synced folders. To achieve that without having the settings placed on your own 24/ running server (which a) kind of defeats the purpose of being cloud only, b) seems relevant to my original question) you have to subscribe to the monthly plan. We are considering it. Organization of your folders shouldn't be an issue. We have for example wood textures from multiple sources (thus in different folders), so we use connecter's tags. Requires some labeling at the beginning.

 

[Francisco Penaloza]

@Jan Havlicek

Interactive rendering should not be affected by network or external storage, as I mentioned, you load the scene in 3D Max, then Corona will process the assets before it display your scene, while working in interactive mode there should not be any load from external devices everything is in RAM already.

Regarding workflow, this is particular for each person and company.

I work for an Architectural firm, so everything is network based. Before Corona, we used VPN or Citrix for any remote work , even some Virtual machine technologies and of course BIM 360.

For visualization purposes VPN wasn't working for us. that's why we started to test others options.  Personally I work with 3 computers, one Desktop and one laptop and a render node. In my case having an external device storage works better than having everything inside one computer. That external store is accessible to any computer at my home and is always sic to our Office server, at the office we also have some render nodes, so if we need to use those we can submit our work from home to render at the office network.

Personally I freelanced for many years, and I also learned that having everything in one computer wasn't the bets workflow in my case. Personally I had a small server computer I build, then I changed by a NAS and many other external HHD. Overall that workflow give me the flexibility to not depend on one machine and easy and redundant backups.  I also have a cloud backup service. redundancy is never enough ?

 

Cheers.

 

[Thomas Edward Allen]

Get the NAS and get some redundancy. Synology is great.

[Tommy L]

One other thing, if you're buying hardware from scratch, be sure to set up a 10gig network, its a little more expensive but much faster. 

As has been mentioned already, working remotely requires a different kind of planning. We went from a 10gig network in the office to using Dropbox as a server. Its workable. Small things have changed, but its mainly been an easy switch. 

[Kevin Häfeli]

I'm doing some market research and stumbled across this thread.
Our cloud company develops a solution for the ArchViz industry to support artists during the shift to remote work caused by Covid-19 (and later on..)

We develop a product with the following features:

• syncing project files (textures, 3d objects etc) automatically between the cloud and your workstation (to collaborate with your co-workers)
• attaching the render power of the cloud (>10'000 servers) to your local machine. Enabling you to get the results much faster (without investing in a powerful and expensive workstation at home)
• maintaining plugins, 3d render engines, licences, versions 

what is needed?

• a decent internet connection (> 20mpbs)
• some local storage for the files you're currently working on

would that be something interesting?
what reservations come into your mind when you hear of such a solution?

[Jan Havlicek]

Hello Kevin,

sorry for quite delayed reply, for some reason I forgot to respond immediately.

The idea sounds great, even more so now, when my country has one of the worst developments regarding second covid wave, so I expect the home office will be new normal now.
Questions that pop to mind are:
a) how much this service aims to be primarily cloud storage and how much it is asset manager (like mentioned Connecter, Kstudio's Project Manager, etc)
b) pricing of this service, compared to today's options

Our current setup is: each team member has the same dropbox paths on their workstations. Dropbox folder contains assets. Thus when we send eachother scene files, the paths to textures, external objects, proxies are the same. Since the disk space on our workstations isnt infinite, I expect we will soon apply the same technique, only with dropbox route not on local disk but on external device, as suggested by gentlemen in this thread. As for asset managment, currently Connecter.

[Cisco Freaks]

On 8/3/2020 at 3:12 PM, Jan Havlicek said:

Hello everyone,
what is in your learned opinion better for studio of 2-3 people who want to share library of their assets (textures, 3D objects, etc, used via Connecter or Project Manager) - CLOUD (e.g. dropbox) or own SERVER machine?
We were almost ready to buy some NAS server, but then it occured to me (and please correct me if i'm wrong), the files sync'ed via dropbox on my workstation are faster option for rendering, than detached files on some other machine, where network bandwith comes to play. I have only supericial understanding of network data exchange, so i'm not sure how factors like "100Mbit or 1Gbit adapter/modem/router", coaxial cable quality, etc would manifest themselves during for instance real time rendering (where i assume is any potential delay caused by slower loading of textures visible the most). Am i missing something? We work primarily in 3dsMax with Corona.
There are articles on NAS vs Cloud subject, but it is the connection with everyday rendering use, that interests me here. Thanks anyone for helpful comments.

https://easypresales.com/easy-backup-compare//

[Kevin Häfeli]

On 10/21/2020 at 12:40 AM, Jan Havlicek said:

Hello Kevin,

sorry for quite delayed reply, for some reason I forgot to respond immediately.

The idea sounds great, even more so now, when my country has one of the worst developments regarding second covid wave, so I expect the home office will be new normal now.
Questions that pop to mind are:
a) how much this service aims to be primarily cloud storage and how much it is asset manager (like mentioned Connecter, Kstudio's Project Manager, etc)
b) pricing of this service, compared to today's options

Our current setup is: each team member has the same dropbox paths on their workstations. Dropbox folder contains assets. Thus when we send eachother scene files, the paths to textures, external objects, proxies are the same. Since the disk space on our workstations isnt infinite, I expect we will soon apply the same technique, only with dropbox route not on local disk but on external device, as suggested by gentlemen in this thread. As for asset managment, currently Connecter.

Hey Jan,

Somehow I missed to respond too. Happy to answer the questions - hope they're still relevant.

1. We're mainly a cloud rendering service. But the cloud journey starts usually with storage. That's why we support file sync to improve the "time-to-render" (upload waiting times).
2. Rendering price is ~20% below render farms: https://render.helio.exchange/

We can also hook into Dropbox files, which would be faster to access. But as we do a simple delta sync each rendering (where you e.g. just need to sync the updated .max file anymore) it's also fast with a local storage / NAS solution.

Cheers,
Kevin