Sorry for being lengthy, but I pretty much concerned myself with QA on the art production side for the last 3 - 4 years…
As I said a pipeline is a process, defined and carried out by humans and (often) supported by pipeline tools. But it’s totally possible to have a pipeline without any scripting involved. It’s just more cumbersome. But you have to start from there to understand how you can fit tools into the process.
First when defining a QA process you have to answer the questions WHO, WHEN, WHAT?
WHAT are the steps your QA process must carry out?
- technical checks
- artistic checks
both decide if an asset passes or fails - other: defining checklists, usage monitoring, auditing for correct use/abuse, etc.
(side note: think about exceptions. What if your art brief consists of 100 assets, each with a distinct and defined poly-count? is your checklist handling system flexible enough for such things?)
(side note: if you allow e.g. ADs to override assets despite failed checks, you need to monitor and audit, to avoid abuse by ADs/Producers who want to take shortcuts which compromise quality)
WHO carries these steps out? This depends on your studio org and the people you have available. For most tasks the roles are pretty set though: you usually have someone who does the planning, someone who is technically knowledgable, someone who is artistically knowledgeable and the artist. Often there’s an overlap. In your process you need to define who makes which decisions, who becomes active at which stage and who needs which information and when they need this information.
WHEN? This defines the checking process. E.g. when do reviews take place? When do technical checks happen? e.g. after each production step? before export? daily? Depending on the assets, and the project’s focus on quality, this can vary a lot! You could work on a character that takes 30 days to produce, or on small scale props where your artists churn out a dozen a day. Assets can be very high poly or low poly.
Once you have answers to all this, your process is defined.
Next: how can you support this with tools? And at this point the re-usability of most pipelines break, because they’re not flexible enough.
Let’s break down the WHAT some more:
Remember, ALL checks must pass for the asset to pass (at least in most cases )
- technical checks which can be automated
- technical checks which cannot be automated and must be checked manually
- artistic checks which cannot be automated and must be checked manually
(without point 2 and 3 your checking will be incomplete)
point 3 is obvious. You cannot check if the asset conforms to a concept or an art style.
point 1 are checks which are generally applicable to a whole group of assets: like naming conventions, freeze transforms, etc.
point 2 are checks which cannot be automated due to things like:
a) time/skill/budget constrains: your programmers just do not have time/skill/budget to implement this check (and trust me this WILL happen, we got over 100 different checks and still clients can think of new stuff! )
b) technical limitations: your check framework doesn’t support (or integrated with) whatever tech it needs for the checking
c) technicalities which need human judgment: e.g. the arrangement of edge-loops for good animation/rigging, or anything that needs to be manually checked because of a) and b)
The manual way:
- after the workflow is defined, people know whom to inform, and when to inform.
- they know who needs to attend reviews, what the review standards are, and who makes the decisions
- decisions are hopefully(!) recorded in writing (e.g. Excel, e-mail) and not just verbally
- new tasks will be created (for the next production step, or for any fixes), e.g. in Excel, or also just verbally
The automated way:
Your pipeline can only hope to fully automate (i.e. no human intervention) point 1.
However, a bad pipeline would stop here, and totally forget about the 2 other checks that need to be carried out by humans (point 2 and 3)!
A good pipeline would now offer a system to collect the decisions of these 2 other steps as well, and integrate with whatever data it collected by itself.
To do this, the pipeline must still rely on human being to manually enter the results from the manual checks. This is where the pipeline must work with people like the AD/Lead for art check, a TA/art-QA for tech check, and maybe a producer to schedule follow up tasks (if your pipeline tools are flexible, you could also update/spawn tasks in systems like Jira or Hansoft).
For this the pipeline tools must know the WHO and WHEN. I.e. you need a way to define your process and actors and make it understood to the pipeline tools (another thing where most pipelines become inflexible and bound to one certain way of doing things)
You see, the actual pipeline tools do very little: They do automated checks and provide glue/communication between the human parts of the pipeline.
However, especially with the communication a pipeline can contribute tremendously to the success of art QA!
Another issue to think about is how to interweave QA with production steps. QA at the end is bad. The later QA happens the more you have to undo. Optional QA is usually as useful as no QA. Checking results which are not monitored and taken into account in decision making are useless. A pipeline and its process must ensure these problems do not happen. This is one of the few times where I think it is actually okay for a pipeline to be inflexible.
Summary: for QA focus on the entire process, and ensure your tool is aware of it and supports it by taking information, generated outside, into account.
Unfortunately I haven’t come across much good literature about software QA, and most if it is geared for very formalized environments dealing with code (also be warned, it’s a super dry subject! which explains the bad reviews), but yet I found these books quite helpful when understanding that QA is really about process, and what methods it uses: http://www.amazon.co.uk/Software-Testing-Continuous-Quality-Improvement-ebook/dp/B000Q36ELK/ref=sr_1_3?ie=UTF8&qid=undefined&sr=8-3&keywords=software+quality+crc
http://www.amazon.co.uk/Software-Quality-Assurance-Implementation-Alternative/dp/0201709457/ref=sr_1_1?ie=UTF8&qid=undefined&sr=8-1&keywords=software+quality+assurance (even though this one gets better review, I feel it’s even more abstract)
Finally, there is another reason why your pipeline/QA tools must be flexible: Process improvement (think CMMI & Co). If the QA process needs to change, your pipeline needs to change (because people shouldn’t be slaves to a pipeline ) I know a lot is very hard to accomplish, especially the desired (required) flexibility. But that’s imho one thing where pipelines need to improve most. And that’s also what will decide if your pipeline is generic enough to be re-used and shared by many projects.