* [[http://openhome.cc/Gossip/Programmer/Encoding1.html|程式人專欄: 亂碼 1/2(上)頻遭模糊的字元集與編碼基礎]]
* [[http://openhome.cc/Gossip/Programmer/Encoding2.html|程式人專欄: 亂碼 1/2(下)常見字元編碼處理方式]]
* [[http://www.luanxiang.org/blog/archives/1271.html|浅谈编码]]
* 寬字符 ([[wp>Wide character]], [[http://www.cplusplus.com/reference/cwchar/wchar_t/|wchar_t]])
* 多字節字符 (multibyte character)
* 字符/字元 (character)
* 碼位/碼點 (code point)
* 字符對應的編號。
* 字符集/字元集 (character set)
* 字符和碼位的對應關係。
* 編碼 (encoding)
* 碼位實際儲存在內存或是磁盤上的內容。
* UTF-8 就是 Unicode 字符其中一種編碼。
* [[wp>ASCII]] 是一種編碼。
* 碼位 (code point) 和編碼 (encoding) 是兩種不一樣的概念。
* [[wp>Byte order mark]]
* 指名後續編碼為大端或是小端。
* 代碼表/頁碼表 (code page)
* 字符對應的編碼 (encoding)。
* byte string。即 char*。
* character string。即 wchar_t*。
* 小結:
* Unicode 是字符集 (character set)。
* Unicode 有底下幾種編碼 (encoding):
* UCS-2、UCS-4、UTF-8、UTF-16 (Windows 採用) 和 UTF-32。
====== Windows ======
* [[http://unicodebook.readthedocs.org/en/latest/operating_systems.html#operating-systems|Programming with Unicode (Windows)]]
* [[http://msdn.microsoft.com/en-us/library/06b9yaeb.aspx|Text and Strings in Visual C++]]
* [[http://msdn.microsoft.com/en-us/library/cwe8bzh0.aspx|Unicode and MBCS]]
* Unicode is a 16-bit character encoding, providing enough encodings for all languages. All ASCII characters are included in Unicode as widened characters.
* Windows 採用 UTF-16 做為 Unicode 字符集的編碼。
* Unicode: Windows ME/Windows 98 以前的平台不支援
* MBCS: Unicode 的替代品,在所有 Windows 平台皆支援。新開發的軟件不建議採用 MBCS,直接採用 Unicode。
* SBCS: 即 ACSII
* [[http://msdn.microsoft.com/en-us/library/c426s321.aspx|Generic-Text Mappings in Tchar.h]]
* 巨集 _TCHAR 的對應如下:
* Unicode (UTF-16): wchar_t。此為 Windows 規定。
* MBCS: char
* SBCS: char
* 小結:
* 工程設置一律採用 "Use Unicode Character Set"。
* 使用 TCHAR 宣告字符串常量和變數。
* 使用 _t 開頭的函式。
* [[http://utf8everywhere.org/#how|How to do text on Windows]] 不建議上述兩項作法。
===== 轉換 =====
* [[http://msdn.microsoft.com/zh-tw/library/ms235631.aspx|How to: Convert Between Various String Types]]
* A char * string (also known as a C style string) uses a null character to indicate the end of the string. C style strings usually require one byte per character, but can also use two bytes. In the examples below, char * strings are sometimes referred to as multibyte character strings because of the string data that results from converting from Unicode strings.
* [[http://stackoverflow.com/questions/8032080/how-to-convert-char-to-wchar-t|How to convert char* to wchar_t*?]]
* 多字節字符串 (multibyte character string) 轉寬字符串 (wide character string)
* [[http://msdn.microsoft.com/zh-tw/library/eyktyxsx.aspx|mbstowcs_s]]
// Create and display a C style string, and then use it
// to create different kinds of strings.
char *orig = "Hello, World!";
cout << orig << " (char *)" << endl;
// newsize describes the length of the
// wchar_t string called wcstring in terms of the number
// of wide characters, not the number of bytes.
size_t newsize = strlen(orig) + 1;
// The following creates a buffer large enough to contain
// the exact number of characters in the original string
// in the new format. If you want to add more characters
// to the end of the string, increase the value of newsize
// to increase the size of the buffer.
wchar_t * wcstring = new wchar_t[newsize];
// Convert char* string to a wchar_t* string.
size_t convertedChars = 0;
mbstowcs_s(&convertedChars, wcstring, newsize, orig, _TRUNCATE);
// Display the result and indicate the type of string that it is.
wcout << wcstring << _T(" (wchar_t *)") << endl;
errno_t mbstowcs_s(
size_t *pReturnValue,
wchar_t *wcstr,
size_t sizeInWords,
const char *mbstr,
size_t count
);
* mbstowcs_s uses the current locale for any locale-dependent behavior; _mbstowcs_s_l is identical except that it uses the locale passed in instead.
* [[http://msdn.microsoft.com/en-us/library/windows/desktop/dd319072(v=vs.85).aspx|MultiByteToWideChar]]
// 計算輸入(欲轉換)字串 pszValue 的字數 (character)。注意! 非字節數 (byte)。
size_t n = ::MultiByteToWideChar(CP_ACP,0,(const char *)pszValue,-1,NULL,0);
// 配置輸出(欲轉出)字串緩衝區。
wchar_t* buffer = new wchar_t[n];
// 轉換輸入字串 (pszValue) 至輸出字串緩衝區 (buffer)。
::MultiByteToWideChar(CP_ACP,0,(const char *)pszValue,-1,buffer,int(n));
// 將緩衝區資料另存起來。
m_strValue = tstring(buffer);
// 釋放緩衝區。
delete [] buffer;
int MultiByteToWideChar(
_In_ UINT CodePage,
_In_ DWORD dwFlags,
_In_ LPCSTR lpMultiByteStr,
_In_ int cbMultiByte,
_Out_opt_ LPWSTR lpWideCharStr,
_In_ int cchWideChar
);
* CodePage
* 輸入字串的編碼。
* lpMultiByteStr
* 輸入字串指針
* cbMultiByte
* 欲處理輸入字串多少個字節數 (byte)。
* 若為 -1,代表輸入字串為空字符 (NULL) 結尾。MultiByteToWideChar 返回值代表輸入字串的字數 (character),包含空字符。
* lpWideCharStr
* 輸出字串緩衝區。可為 NULL。
* cchWideChar
* 輸出字串緩衝區大小,以字數計 (character)。
* 若為 0,MultiByteToWideChar 返回值代表輸出字串緩衝區 (lpWideCharStr) 所需字數 (character)。
* [[http://www.cplusplus.com/reference/sstream/wstringstream/|wstringstream]]
#include
#include "tstring.h"
std::wstringstream wss;
wss << pszValue;
m_strValue = tstring(wss.str().c_str());
* [[http://blog.163.com/tianshi_17th/blog/static/4856418920085209414977/|也谈C++中char*与wchar_t*之间的转换]]
* 尚未驗證其正確性。
* 寬字符串 (wide character string) 轉多字節字符串 (multibyte character string)
* UTF-8 一般可以用 char* 表示,因為彼此皆為 8-bit 編碼。char 是 signed 或是 unsigned 不影響。
* [[http://stackoverflow.com/questions/148403/utf8-to-from-wide-char-conversion-in-stl|UTF8 to/from wide char conversion in STL]]
* [[http://llvm.org/svn/llvm-project/llvm/trunk/include/llvm/Support/ConvertUTF.h|ConvertUTF.h]]
* [[http://stackoverflow.com/questions/8818652/how-can-char-represent-an-utf-8-string|How can char[] represent an UTF-8 string?]]
* 小結:
* 編碼在編譯器之應用,編譯器內部應採用寬字符串 (wide character string),即 Windows 內部支援的 Unicode; 外部輸入或輸出一律視為多字節字符串 (multibyte character string),可能是 ASCII 或是 UTF-8。
* 編譯器使用的 Lex 應視輸入為 UTF-8,再轉換成寬字符,交給後續程序處理。
===== 其它 =====
* [[http://stackoverflow.com/questions/11107608/whats-wrong-with-c-wchar-t-and-wstrings-what-are-some-alternatives-to-wide|What's “wrong” with C++ wchar_t and wstrings? What are some alternatives to wide characters?]]
* [[http://stackoverflow.com/questions/13087219/what-exactly-is-the-l-prefix-in-c|What exactly is the L prefix in C++?]]
* [[http://stackoverflow.com/questions/17103925/how-well-is-unicode-supported-in-c11|How well is unicode supported in C++11?]]
* [[http://stackoverflow.com/questions/16167305/why-does-my-application-require-visual-c-redistributable-package|Why does my application require Visual C++ Redistributable package]]
* [[http://www.codeproject.com/Articles/76252/What-are-TCHAR-WCHAR-LPSTR-LPWSTR-LPCTSTR-etc|What are TCHAR, WCHAR, LPSTR, LPWSTR, LPCTSTR (etc.)?]]
* [[http://www.cppblog.com/seahouse/archive/2011/01/13/137571.aspx|VSVC编译选项/MDd与/MTd]]
* [[http://stackoverflow.com/questions/604484/linker-errors-between-multiple-projects-in-visual-c|Linker errors between multiple projects in Visual C++]]
* char* 轉換成 LPCTSTR
* [[http://blog.chinaunix.net/uid-11143705-id-90369.html|char 转wchar_t 及wchar_t转char]]
* [[http://blog.csdn.net/fengshalangzi/article/details/5815073|wchar_t与char转换(总结)]]
* [[http://www.cnblogs.com/gdutbean/archive/2012/03/31/2427609.html|char 转wchar_t 及wchar_t转char]]
====== Linux ======
* [[http://unicodebook.readthedocs.org/en/latest/index.html|Programming with Unicode]]
* [[http://www.cprogramming.com/tutorial/unicode.html|Unicode in C and C++: What You Can Do About It Today]]
* [[http://linuxprograms.wordpress.com/tag/unicode/|C: Using scanf and wchar_t to read and print UTF-8 strings]]
* [[http://www.codeproject.com/Articles/38242/Reading-UTF-with-C-streams|Reading UTF-8 with C++ streams]]
====== 中文編碼 ======
* [[wpzh>中日韓統一表意文字]]
* [[wpzh>大五碼]]
====== 參考資料 ======
* [[http://www.utf-8.com/|UTF-8 and Unicode]]
* [[http://www.joelonsoftware.com/articles/Unicode.html|The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)]]
* [[http://www.csie.ntu.edu.tw/~p92005/Joel/Unicode.html|每個軟體開發者都絕對一定要會的Unicode及字元集必備知識(沒有藉口!)]]
* [[http://stackoverflow.com/questions/20942469/clarification-on-joel-spolskys-unicode-article|Clarification on Joel Spolsky's Unicode Article]]
* [[http://www.utf8everywhere.org/|UTF-8 Everywhere]]
* [[https://docs.python.org/release/3.2/howto/unicode.html|Unicode HOWTO]]
* Unicode 編碼速查
* [[http://www.scarfboy.com/coding/unicode-tool|Unicode lookup/search tool]]
* [[http://www.fileformat.info/info/unicode/char/search.htm|Unicode Character Search]]
* 自製編程語言第五章