N00b's Programming Machine
    Share your code. npm Orgs help your team discover, share, and reuse code. Create a free org »

    codepagepublic

    js-codepage

    Codepages are character encodings. In many contexts, single- or double-byte character sets are used in lieu of Unicode encodings. The codepages map between characters and numbers.

    Setup

    In node:

    var cptable = require('codepage');

    In the browser:

    <script src="cptable.js"></script>
    <script src="cputils.js"></script>

    Alternatively, use the full version in the dist folder:

    <script src="cptable.full.js"></script>

    The complete set of codepages is large due to some Double Byte Character Set encodings. A much smaller file that just includes SBCS codepages is provided in this repo (sbcs.js), as well as a file for other projects (cpexcel.js)

    If you know which codepages you need, you can include individual scripts for each codepage. The individual files are provided in the bits/ directory. For example, to include only the Mac codepages:

    <script src="bits/10000.js"></script>
    <script src="bits/10006.js"></script>
    <script src="bits/10007.js"></script>
    <script src="bits/10029.js"></script>
    <script src="bits/10079.js"></script>
    <script src="bits/10081.js"></script>

    All of the browser scripts define and append to the cptable object. To rename the object, edit the JSVAR shell variable in make.sh and run the script.

    The utilities functions are contained in cputils.js, which assumes that the appropriate codepage scripts were loaded.

    The script will manipulate module.exports if available . This is not always desirable. To prevent the behavior, define DO_NOT_EXPORT_CODEPAGE.

    Usage

    Most codepages are indexed by number. To get the Unicode character for a given codepoint, use the dec property:

    var unicode_cp10000_255 = cptable[10000].dec[255]; // Ë‡

    To get the codepoint for a given character, use the enc property:

    var cp10000_711 = cptable[10000].enc[String.fromCharCode(711)]; // 255

    There are a few utilities that deal with strings and buffers:

    var ćą‡ć€ť = cptable.utils.decode(936, [0xbb,0xe3,0xd7,0xdc]);
    var buf =  cptable.utils.encode(936,  ćą‡ć€ť);
    var sushi= cptable.utils.decode(65001, [0xf0,0x9f,0x8d,0xa3]); // đŸŁ
    var sbuf = cptable.utils.encode(65001, sushi);

    cptable.utils.encode(CP, data, ofmt) accepts a String or Array of characters and returns a representation controlled by ofmt:

    • Default output is a Buffer (or Array) of bytes (integers between 0 and 255)
    • If ofmt == 'str', return a binary String (byte i is o.charCodeAt(i))
    • If ofmt == 'arr', return an Array of bytes

    cptable.utils.decode(CP, data) accepts a byte String or Array of numbers or Buffer and returns a JS string.

    Known Excel Codepages

    A much smaller script, including only the codepages known to be used in Excel, is available under the name cpexcel. It exposes the same variable cptable and is suitable as a drop-in replacement when the full codepage tables are not needed.

    In node:

    var cptable = require('codepage/dist/cpexcel.full');

    Rolling your own script

    The make.sh script in the repo can take a manifest and generate JS source.

    Usage:

    $ bash make.sh path_to_manifest output_file_name JSVAR

    where

    • JSVAR is the name of the exported variable (generally cptable)
    • output_file_name is the output file (cpexcel.js, cptable.js, ...)
    • path_to_manifest is the path to the manifest file.

    The manifest file is expected to be a CSV with 3 columns:

    <codepage number>,<source>,<size>
    

    If a source is specified, it will try to download the specified file and parse. The file format is expected to follow the format from the unicode.org site. The size should be 1 for a single-byte codepage and 2 for a double-byte codepage. For mixed codepages (which use some single- and some double-byte codes), the script assumes the mapping is a prefix code and generates efficient JS code.

    Generated scripts only include the mapping. cat a mapping with cputils.js to produce a complete script like cpexcel.full.js.

    Building the complete script

    This script uses voc. The script to build the codepage tables and the JS source is codepage.md, so building is as simple as voc codepage.md.

    Generated Codepages

    The complete list of codepages can be found in the file pages.csv.

    Some codepages are easier to implement algorithmically. Since those character tables are not generated, there is no corresponding entry (they are "magic").

    CP# Source Description
    37 unicode.org IBM EBCDIC US-Canada
    437 unicode.org OEM United States
    500 unicode.org IBM EBCDIC International
    620 NLS Mazovia (Polish) MS-DOS
    708 Windows 7 Arabic (ASMO 708)
    720 Windows 7 Arabic (Transparent ASMO); Arabic (DOS)
    737 unicode.org OEM Greek (formerly 437G); Greek (DOS)
    775 unicode.org OEM Baltic; Baltic (DOS)
    808 unicode.org OEM Russian; Cyrillic + Euro symbol
    850 unicode.org OEM Multilingual Latin 1; Western European (DOS)
    852 unicode.org OEM Latin 2; Central European (DOS)
    855 unicode.org OEM Cyrillic (primarily Russian)
    857 unicode.org OEM Turkish; Turkish (DOS)
    858 Windows 7 OEM Multilingual Latin 1 + Euro symbol
    860 unicode.org OEM Portuguese; Portuguese (DOS)
    861 unicode.org OEM Icelandic; Icelandic (DOS)
    862 unicode.org OEM Hebrew; Hebrew (DOS)
    863 unicode.org OEM French Canadian; French Canadian (DOS)
    864 unicode.org OEM Arabic; Arabic (864)
    865 unicode.org OEM Nordic; Nordic (DOS)
    866 unicode.org OEM Russian; Cyrillic (DOS)
    869 unicode.org OEM Modern Greek; Greek, Modern (DOS)
    870 Windows 7 IBM EBCDIC Multilingual/ROECE (Latin 2)
    872 unicode.org OEM Cyrillic (primarily Russian) + Euro Symbol
    874 unicode.org Windows Thai
    875 unicode.org IBM EBCDIC Greek Modern
    895 NLS KamenickĂ˝ (Czech) MS-DOS
    932 unicode.org Japanese Shift-JIS
    936 unicode.org Simplified Chinese GBK
    949 unicode.org Korean
    950 unicode.org Traditional Chinese Big5
    1010 IBM IBM EBCDIC French
    1026 unicode.org IBM EBCDIC Turkish (Latin 5)
    1047 Windows 7 IBM EBCDIC Latin 1/Open System
    1132 IBM IBM EBCDIC Lao (1132 / 1133 / 1341)
    1140 Windows 7 IBM EBCDIC US-Canada (037 + Euro symbol)
    1141 Windows 7 IBM EBCDIC Germany (20273 + Euro symbol)
    1142 Windows 7 IBM EBCDIC Denmark-Norway (20277 + Euro symbol)
    1143 Windows 7 IBM EBCDIC Finland-Sweden (20278 + Euro symbol)
    1144 Windows 7 IBM EBCDIC Italy (20280 + Euro symbol)
    1145 Windows 7 IBM EBCDIC Latin America-Spain (20284 + Euro symbol)
    1146 Windows 7 IBM EBCDIC United Kingdom (20285 + Euro symbol)
    1147 Windows 7 IBM EBCDIC France (20297 + Euro symbol)
    1148 Windows 7 IBM EBCDIC International (500 + Euro symbol)
    1149 Windows 7 IBM EBCDIC Icelandic (20871 + Euro symbol)
    1200 magic Unicode UTF-16, little endian (BMP of ISO 10646)
    1201 magic Unicode UTF-16, big endian
    1250 unicode.org Windows Central Europe
    1251 unicode.org Windows Cyrillic
    1252 unicode.org Windows Latin I
    1253 unicode.org Windows Greek
    1254 unicode.org Windows Turkish
    1255 unicode.org Windows Hebrew
    1256 unicode.org Windows Arabic
    1257 unicode.org Windows Baltic
    1258 unicode.org Windows Vietnam
    1361 Windows 7 Korean (Johab)
    10000 unicode.org MAC Roman
    10001 Windows 7 Japanese (Mac)
    10002 Windows 7 MAC Traditional Chinese (Big5)
    10003 Windows 7 Korean (Mac)
    10004 Windows 7 Arabic (Mac)
    10005 Windows 7 Hebrew (Mac)
    10006 unicode.org Greek (Mac)
    10007 unicode.org Cyrillic (Mac)
    10008 Windows 7 MAC Simplified Chinese (GB 2312)
    10010 Windows 7 Romanian (Mac)
    10017 Windows 7 Ukrainian (Mac)
    10021 Windows 7 Thai (Mac)
    10029 unicode.org MAC Latin 2 (Central European)
    10079 unicode.org Icelandic (Mac)
    10081 unicode.org Turkish (Mac)
    10082 Windows 7 Croatian (Mac)
    12000 magic Unicode UTF-32, little endian byte order
    12001 magic Unicode UTF-32, big endian byte order
    20000 Windows 7 CNS Taiwan (Chinese Traditional)
    20001 Windows 7 TCA Taiwan
    20002 Windows 7 ETEN Taiwan (Chinese Traditional)
    20003 Windows 7 IBM5550 Taiwan
    20004 Windows 7 TeleText Taiwan
    20005 Windows 7 Wang Taiwan
    20105 Windows 7 Western European IA5 (IRV International Alphabet 5)
    20106 Windows 7 IA5 German (7-bit)
    20107 Windows 7 IA5 Swedish (7-bit)
    20108 Windows 7 IA5 Norwegian (7-bit)
    20127 magic US-ASCII (7-bit)
    20261 Windows 7 T.61
    20269 Windows 7 ISO 6937 Non-Spacing Accent
    20273 Windows 7 IBM EBCDIC Germany
    20277 Windows 7 IBM EBCDIC Denmark-Norway
    20278 Windows 7 IBM EBCDIC Finland-Sweden
    20280 Windows 7 IBM EBCDIC Italy
    20284 Windows 7 IBM EBCDIC Latin America-Spain
    20285 Windows 7 IBM EBCDIC United Kingdom
    20290 Windows 7 IBM EBCDIC Japanese Katakana Extended
    20297 Windows 7 IBM EBCDIC France
    20420 Windows 7 IBM EBCDIC Arabic
    20423 Windows 7 IBM EBCDIC Greek
    20424 Windows 7 IBM EBCDIC Hebrew
    20833 Windows 7 IBM EBCDIC Korean Extended
    20838 Windows 7 IBM EBCDIC Thai
    20866 Windows 7 Russian Cyrillic (KOI8-R)
    20871 Windows 7 IBM EBCDIC Icelandic
    20880 Windows 7 IBM EBCDIC Cyrillic Russian
    20905 Windows 7 IBM EBCDIC Turkish
    20924 Windows 7 IBM EBCDIC Latin 1/Open System (1047 + Euro symbol)
    20932 Windows 7 Japanese (JIS 0208-1990 and 0212-1990)
    20936 Windows 7 Simplified Chinese (GB2312-80)
    20949 Windows 7 Korean Wansung
    21025 Windows 7 IBM EBCDIC Cyrillic Serbian-Bulgarian
    21027 NLS Extended/Ext Alpha Lowercase
    21866 Windows 7 Ukrainian Cyrillic (KOI8-U)
    28591 unicode.org ISO 8859-1 Latin 1 (Western European)
    28592 unicode.org ISO 8859-2 Latin 2 (Central European)
    28593 unicode.org ISO 8859-3 Latin 3
    28594 unicode.org ISO 8859-4 Baltic
    28595 unicode.org ISO 8859-5 Cyrillic
    28596 unicode.org ISO 8859-6 Arabic
    28597 unicode.org ISO 8859-7 Greek
    28598 unicode.org ISO 8859-8 Hebrew (ISO-Visual)
    28599 unicode.org ISO 8859-9 Turkish
    28600 unicode.org ISO 8859-10 Latin 6
    28601 unicode.org ISO 8859-11 Latin (Thai)
    28603 unicode.org ISO 8859-13 Latin 7 (Estonian)
    28604 unicode.org ISO 8859-14 Latin 8 (Celtic)
    28605 unicode.org ISO 8859-15 Latin 9
    28606 unicode.org ISO 8859-15 Latin 10
    29001 Windows 7 Europa 3
    38598 Windows 7 ISO 8859-8 Hebrew (ISO-Logical)
    47451 unicode.org Atari ST/TT
    50220 Windows 7 ISO 2022 JIS Japanese with no halfwidth Katakana
    50221 Windows 7 ISO 2022 JIS Japanese with halfwidth Katakana
    50222 Windows 7 ISO 2022 Japanese JIS X 0201-1989 (1 byte Kana-SO/SI)
    50225 Windows 7 ISO 2022 Korean
    50227 Windows 7 ISO 2022 Simplified Chinese
    51932 Windows 7 EUC Japanese
    51936 Windows 7 EUC Simplified Chinese
    51949 Windows 7 EUC Korean
    52936 Windows 7 HZ-GB2312 Simplified Chinese
    54936 Windows 7 GB18030 Simplified Chinese (4 byte)
    57002 Windows 7 ISCII Devanagari
    57003 Windows 7 ISCII Bengali
    57004 Windows 7 ISCII Tamil
    57005 Windows 7 ISCII Telugu
    57006 Windows 7 ISCII Assamese
    57007 Windows 7 ISCII Oriya
    57008 Windows 7 ISCII Kannada
    57009 Windows 7 ISCII Malayalam
    57010 Windows 7 ISCII Gujarati
    57011 Windows 7 ISCII Punjabi
    65000 magic Unicode (UTF-7)
    65001 magic Unicode (UTF-8)

    unicode.org refers to the Unicode Consortium Public Mappings, a database of various mappings between Unicode characters and respective character sets. The tables are processed by a few scripts in the build process.

    IBM refers to the IBM coded character set database. Even though IBM uses a different numbering scheme from Windows, the IBM numbers are used when there is no conflict. The tables are manually generated from the symbol manifests.

    Windows 7 refers to direct inspection of Windows 7 machines using .NET class System.Text.Encoding. The enclosed MakeEncoding.cs C# program brute-forces code pages. MakeEncoding.cs deviates from unicode.org in some cases. When they map a given code to different characters, unicode.org value is used. When unicode.org does not prescribe a value, MakeEncoding.cs value is used.

    NLS refers to the National Language Support files supplied in various versions of Windows. In older versions of Windows (like Windows 98) these files followed the name pattern CP_#.NLS, but newer versions use the name pattern C_#.NLS.

    Testing

    make test will run the nodejs-based test.

    To run the in-browser tests, run a local server and go to the ctest directory. make ctestserv will start a python SimpleHTTPServer server on port 8000.

    To update the browser artifacts, run make ctest.

    Sources

    License

    Please consult the attached LICENSE file for details. All rights not explicitly granted by the Apache 2.0 license are reserved by the Original Author.

    Badges

    Sauce Test Status

    Build Status

    Coverage Status

    Analytics

    install

    npm i codepage

    Downloadsweekly downloads

    136,350

    version

    1.13.0

    license

    Apache-2.0

    homepage

    sheetjs.com

    repository

    githubgithub

    last publish

    collaborators

    • avatar