The plugin works in the way that it simply looks up the location of the DirectX SDK from the environment variable DXSDK_DIR and executes the utility fxc.exe on the given asset.
My problem in all this was when something goes wrong during shader compilation, I need to propagate the error message up to VACP.The fxc.exe utility outputs information to the standard error stream (stderr), so how would I do to get this information into a std::string or similar?
The solution was to use anonymous pipes to redirect the streams.
First, create your anonymous pipe:
SECURITY_ATTRIBUTES sec_attr;
sec_attr.nLength = sizeof(SECURITY_ATTRIBUTES);
sec_attr.bInheritHandle = TRUE;
sec_attr.lpSecurityDescriptor = NULL;
if(!CreatePipe(this->fxc_stderr_rd, this->fxc_stderr_wr, &sec_attr, 0))
{
return -1;
}
Here, fxc_stderr_rd is the HANDLE to the read "end" of the pipe, and fxc_stderr_wr is the HANDLE to the write "end" of the pipe.
Note that bInheritHandle of the SECURITY_ATTRIBUTES struct needs to be set to TRUE, or else you'll get smacked in the face with ERROR_BROKEN_PIPE once you try to read/write from the pipe. Not nice.
After the pipe handles has been created, call SetHandleInformation to make sure the read handle for the stderr is not inherited by the child process:
if(!SetHandleInformation(this->fxc_stderr_rd, HANDLE_FLAG_INHERIT, 0))
{
CloseHandle(this->fxc_stderr_rd);
CloseHandle(this->fxc_stderr_wr);
return -1;
}
Time to create our child process and tell it to redirect its stderr stream to our pipe:
STARTUPINFO startup_info;
ZeroMemory(&startup_info, sizeof(STARTUPINFO));
startup_info.cb = sizeof(STARTUPINFO);
startup_info.dwFlags |= STARTF_USESTDHANDLES;
startup_info.hStdError = this->fxc_stderr_wr;
PROCESS_INFORMATION proc_info;
ZeroMemory(&proc_info, sizeof(PROCESS_INFORMATION));
if(!CreateProcess(
const_cast<LPCWSTR>(fxc_path.c_str()),
const_cast<LPWSTR>(fxc_args.c_str()),
NULL,
NULL,
TRUE,
0,
NULL,
NULL,
&startup_info,
&proc_info))
{
CloseHandle(this->fxc_stderr_rd);
CloseHandle(this->fxc_stderr_wr);
std::wstringstream msg_stream;
msg_stream << L"CreateProcess failed. Return value: " << GetLastError();
ContentCompiler::set_last_error_msg(msg_stream.str());
return -1;
}
The interesting part here is what we do just in the beginning of this code snippet.
In the STARTUPINFO struct, setting the STARTF_USESTD_ERROR_HANDLE flag on the dwFlags member will tell CreateProcess to examine the hStdError, hStdInput and hStdOutput members for handles to redirect the streams to. In this case I am only interested in the stderr stream.
The next line effectively redirects the standard error stream to the write handle of our pipe!
Once the fxc.exe has been invoked as our child process, we'll just wait until it has finished compilation by using WaitForSingleObject, after which we examine the exit code of the process aswell as close our handles:
WaitForSingleObject(proc_info.hThread, COMPILE_TIMEOUT);
DWORD exit_code;
GetExitCodeProcess(proc_info.hProcess, &exit_code);
CloseHandle(proc_info.hProcess);
CloseHandle(proc_info.hThread);
CloseHandle(this->fxc_stderr_wr);
This effectively closes the handles to our child process and its main thread, aswell as the handle to the write end of our anonymous pipe, which we do not need anymore. It is not time to dispose the read handle just yet...
fxc.exe, following the norms of exit codes, returns non-zero on failure. If exit_code is non-zero, we should examine the contents of the pipe.
if(exit_code != 0)
{
std::string error_info = this->get_pipe_data();
ContentCompiler::set_last_error_msg(error_info);
return -1;
}
Where get_pipe_data is defined as:
const std::string HLSLCompiler::get_pipe_data()
{
std::string error_info;
char buffer[READ_BUFF_SIZE];
DWORD bytes_read;
while(ReadFile(this->fxc_stderr_rd, buffer, READ_BUFF_SIZE, &bytes_read, NULL) && bytes_read > 0)
{
error_info.append(buffer, bytes_read);
}
return error_info;
}
And there you go!
There will probably be an angry mob screaming at me (or not so much!) for only showing how to redirect stderr when the title of this post promised oh so much more. It would be practically the same with stdin and stdout, I promise.
The code snippets are taken directly from my project so i apologize if the look like they're taken a bit out of context ...because they are! The code is also fairly fresh so I haven't gotten proper error handling or cleanup yet. Do not despair, it'll be there soon! And it'll be checked in to the repository too, oh fun!
