Home > News content

Windows command line: Dive into the Windows console

via:开源中国     time:2018/8/8 9:27:49     readed:264

Windows GUI startDevelopmentIt was born in the context of the development team needing to develop a console-based application! The Windows console is the first Windows NT GUI application and ensures that compatible runs continue to use existing Windows applications.

The original code of the Windows console has been 30 years old until now (2018)... Old things, in fact, there are still many developers using it today!

What can the console program do?

As the previous article said, the work of the terminal is actually very simple:

  • Handle user input:

    • Input devices that can be supported include keyboards, mice, touch pads, pens, and the like.

    • Convert input data to intermediate characters or ANSI/VT encoding format

    • Send character data to a connected app or device

  • Handling application output:

    • Allow text to be output from connected applications

    • Update the display above the screen to accept the display based on the application (such as displaying text, moving the cursor, setting the font color, etc.)

  • System coordination processing:

    • Run processing job request

    • Manage devices and resources

    • Support for resizing windows, maximizing windows, minimizing windows, etc.

    • Interrupt request or when channel is closed or terminated

However, what the Windows console can do is a bit different:

The Windows console is a traditional Win32 executable. Although it was originally written in "C", most of the code is being migrated to modern C++ with the team's modernization and modular console codebase.

For those who care about such things: Many people are asking if Windows is written in C or C++. The answer is - although NT is an object-based design - like most operating systems, Windows is written almost entirely in C! why? C++ introduces overhead in memory footprint and code execution overhead. Even today, the hidden overhead of code written in C++ can be surprising, but as early as the late 1990s,Memory priceAbout 60$/MB (yes...eachMEGABYTEFor $60! When, the memory overhead of the hidden mechanism such as vtable is very high. In addition, the overhead of virtual method indirect calls and object dereferences can lead to very significant performance and scale loss of C++ code at the time. While you still need to be careful, the performance overhead of modern C++ on modern computers is not a concern, and given its security, readability, and maintainability advantages, this is often an acceptable compromise. ...that's why we steadily upgrade the Console code to modern C++!

So what is inside the Windows console?

Prior to Windows 7, Windows console instances were hosted at the coreclient-serverRuntime Subsystem (CSRSS)! However, in Windows 7, considering the security and reliability factors, the console was spun off from CSRSS, and a new family with the following binaries was assembled:

  • Conhost.exe- User mode Windows console UX and command line pipeline

  • Condrv.sys- A core driver that provides the underlying communication structure to connect communication between conhost and command line shell/tools/applications

The overall structure of the current internal structure of the console looks like this:

The core components of the console contain the following (bottom up):

  • ConDrv.sys- Core mode driver

    • Request execution of an API call to the console instance for data rendering

    • Text sent from the console to the command line app

    • Provides a high-performance communication channel for the console and its connected command-line applications

    • Repeat this pass on the console and the command line attached to itIO Control (IOCTL)news

    • Management console IOCTL message

  • ConHost.exe- Win32 graphical interface (GUI) application:

    • The layout, size, location, etc. of the management console container on the screen.

    • Display and process the settings interface, etc.

    • transferWindows Message Queue, handles Windows messages and converts user input to keyboard and mouse events and stores them in the input buffer.

    • API Server: IOCTL messages received from the command line when the API call is converted, and text records are sent from the console to the command line application.

    • API: Implement the Win32 console API, and the logic behind all the operations that require the console to perform.

    • Input Buffer: Save generated by user inputKeyboard and mouse event logging

    • VT Parser: If started, the VT sequence is parsed from the text, and an equivalent API is generated based on the information found.

    • Output Buffer:Save the text rendered by the console. Essentially a two-dimensionalCHAR_INFOAn array of structures, each of which contains character data and its properties (more information under the buffer)

    • Other: Not included in the upper layer, containing the stored/retrieved base settings values ​​from the registry or shortcut files.

    • ConHost Core- Internal control and piping of the console

    • Console UX App Services- UX and UI layers of the console

Windows console API

As can be seen from the above console architecture diagram, unlike the NIX terminal, the console sends/receives API calls and/or data serialization to IO control (IOCTL) message, not serialized text! Even ANSI/VT sequences embedded in text received from (mainly Linux) command-line applications are extracted, parsed, and converted to API calls!

This difference reveals the key fundamental philosophical differences between *NIX and Windows:*NIXIn, "everything is a file", but in Windows, "everything isObject"!

Both methods have advantages and disadvantages, we will summarize them, but avoid long-forms here. Remember that this key difference in philosophy is between Windows and *NIXManyThe basis of the difference!

In the *NIX system, everything is a file

When Unix was first implemented in the late 1960s and early 1970s, one of the core principles was that anything could be abstracted into a file stream. A key goal was to simplify access to devices and peripherals: if all devices All exist in the form of a file system in the system, so existing code can directly access these devices without modification.

This principle has far-reaching implications: you can browse and query a large number of *NIX-based systems and machine configurations through pseudo file systems or virtual file systems, which are simply "behaves" like "files" or "folders", which may actually Is the machine configuration or hardware.

For example, in Linux, you can view some information about the CPU by accessing the /proc/cpuinfo virtual file node:

This model is so simple and consistent, but it also has some additional overhead: extracting or querying special textual information from these pseudo-files and returning from execution commands often requires the assistance of tools such as sed, awk, perl, Python, etc. These tools are often used to write scripts and commands to parse text content, find special patterns, regions, and values. These scripts can become very complex and difficult to maintain and fragment. If the structure, layout, or format of the text changes, many scripts will need to be updated as well.

In Windows, everything is an object

whenWindows NTWhen designed and built, "objects" are seen as the future of software design: "object-oriented" languages ​​appear faster than rabbits in caves - Simula and Smalltalk have been built, and C++ is becoming more popular . otherObject-oriented languageMany other languages ​​such as Python, Eiffel, Objective-C, ObjectPascal/Delphi, Java, C#, etc. are closely following the rapid development.

Inevitably, it was shaped in a good object-oriented period (circa 1989), and Windows NT was designed with the idea that "everything is an object." In fact, one of the most important parts of the NT kernel is the Object Manager!

Windows NT is publicA rich set of Win32 APIsThese APIs can be called to get and/or manipulate objects from the operating system. Developers use the Win32 API to collect and render similar information provided by *NIX pseudo files and tools, but through objects and structures. And because parsers, compilers, and parsers understand the structure of objects, it's common to catch many coding errors earlier, helping to verify that the programmer's intent is grammatically and logically correct. This can also reduce system damage, volatility and "stirring" over time.

So, back to our central discussion about the Windows console: The NT team decided to build a "console" that differs from traditional * NIX terminals in several key areas:

  1. Console API: Windows Console can be richConsole APIInstead of relying on the programmer's ability to generate "hard to verify" ANSI/VT sequences, it operates and controls.

  2. Public Service: In order to avoid re-implementing the same service (such as command history, command alias) again and again for each command line shell, the console itself provides some of these services, which can be accessed through the Console API.

Windows console issue

While Console's API has proven to be very popular in Windows command-line tools and services, the API-centric model presents some challenges to the command-line approach:

Only Windows implements the Console API

Many Windows command line tools and applications are widely usedConsole API.

What about the problem? These APIs are only available for Windows.

Therefore, combined with other differentiation factors (such as process life cycle differences, etc.), Windows command line applications are not always easy to port to * NIX, and vice versa.

As a result, the Windows ecosystem has developed its own, often similar but often different command line tools and applications. This means that users must learn a set of command-line applications and tools, shells, scripting languages, etc. when using Windows, and learn another set when using *NIX.

There is no simple quick solution to this problem: the Windows console and the command line cannot simply be discarded and replaced by bash and iTerm2 - there are hundreds of millions of applications and scripts that rely on the Windows console and Cmd / PowerShell shells.

LikeCygwinSuch third-party tools can port many of the core GNU tools and compatibility libraries to Windows, but they cannot run unported, unmodified Linux binaries. This is very important because many Ruby, Python, Node packages and modules rely on or wrap Linux binaries, or rely on *NIX operational state.

These reasonsMicrosoftThroughWindows Subsystem Linux (WSLRun native, unmodified Linux binaries and tools locally to extend Windows compatibility. Users using WSL can now download and install one or more Linux distributions in parallel on the same machine and use apt / zypper / npm / gem / etc. to install and run most Linux command line tools and their favorite Windows Applications and tools.

However, there are some console-provided things that have not been adopted by non-Microsoft terminals: specifically, the Windows console provides command history and command alias services, eliminating the need to re-implement the same functionality for each command line shell (especially).

It is difficult to remoteize the Windows command line.

As we are atCommand-Line BackgrounderAs discussed in the article, terminals are initially separated from the computers to which they are connected. Fast forward to today, this design still exists: most modern terminals and command line applications/shells, etc. are separated by process or machine boundaries.

On *NIX-based platforms, the concept of separation of terminals and command-line applications and communication through simple characters results in the *NIX command line being easy to access and manipulate from remote computers/devices: as long as the terminal and command-line applications can pass some The type of ordered serial communication infrastructure (TTY/PTY, etc.) transports character streams, and the command line for remote operation of *NIX machines is very simple.

But on Windows, many command-line applications rely on calling the Console API and assume they are running on the same machine as the console itself. This makes it difficult to remotely manipulate Windows command line shells/tools, etc. How does a command line application running on a remote computer invoke an API on the console of the user's local computer? Worse, if a remote command-line application accesses a terminal on a Mac or Linux machine, how does it call the Console API? !

I am sorry to make a joke, but we will explain this theme in more detail in a future article!

Start the console or not!

Typically, on *NIX-based systems, when a user wants to launch a command-line tool, they first start a terminal. The terminal then launches a default shell or can be configured to launch a specific application/tool. Terminal and command line applications passPseudo terminal (PTY)The character stream is exchanged for communication until one or two characters are terminated.

However, on Windows systems, things are different:Windows users never start the console(conhost.exe) - However they will launch command line shells and applications like Cmd.exe, PowerShell.exe, wsl.exe and so on. The Windows system connects the newly launched application to the current console (if it is launched from the command line) or to the newly created console instance.

# What do you want to say now?

Yes, on a Windows system,The user launches the command line application instead of the console itself.

If a user launches a command line application from an existing command line shell, Windows typically attaches the newly launched .exe (executable file) to the current console. Otherwise, Windows will bind a new console instance to the newly launched application.

Xiao Bai said: Many people say "command line programs run in the console." This is not true, and it leads to a lot of confusion about how consoles and command-line applications work! Command line applications and their consoles run in separate Win32 processes. Please help correct this misunderstanding by pointing out "Command line tools/applications connected to the console" (or similar). Thank you!

Sounds good, right? Hmm... no; here are some questions:

1. The console and command line applications communicate by IOCTL messages via the driver, not through text streams

2.windows requirements ConHost.exe must be a console program connected to the command line application

3.Windows controls the creation of communication "pipes" between console and command line application communication

These are obvious limitations: What if you want to create an alternative console application for Windows? How do you send information about peripherals such as keyboards, mice, pens, etc? If you can't access the communication "pipeline" that connects your new console to the command line application, how do users manipulate the command line application?

Unfortunately, these are not good: there are some great third-party consoles (and server applications) for Windows (eg ConEmu/Cmder, Console2/ConsoleZ, Hyper,Visual StudioCode, OpenSSH, etc.), they must go through a bizarre jump to run like a normal console!

For example, a third-party console must launch a command-line application off-screen, such as (-32000, -32000). They must then send keystrokes to the off-screen console, then collect the text content of the off-screen console and redraw them on their own UI!

I know, this is crazy, right?! This proves the originality and determination of these application creators, these programs are still running effectively!

This is obviously a situation in which we are eager to remedy. Stay tuned for more information on this section - there is some good news in this regard!

Windows Console & VT

As mentioned above, the Windows console providesa lot of API. Using the console API, command line applications and tools can write text, change text color, move the cursor, and more. Also, due to the existence of the console API, the Windows console requires little support for ANSI/VT sequences, and these sequences provide very similar functionality on other platforms.

In fact, prior to Windows 10, the Windows console only implemented minimal support for ANSI/VT sequences:

Since 2014, Microsoft has formed a new Windows console team that has changed everything. One of the highest priorities for the console team is to implement full support for ANSI/VT sequences in order to render Linux on the Windows subsystem (WSL) and the output of the *NIX application running on a remote *NIX machine. You can be in this seriesIn the previous articleRead more about this story.

The console team quickly added full support for ANSI/VT sequences to the Windows 10 console, enabling users to use and enjoy a wide range of Windows and Linux command-line tools and applications.

The team continues to improve and refine the console support for VT on each operating system release and to you in ourGitHubThanks for any questions submitted on the issue tracker.

Handling Unicode

A quick Unicode review:

UnicodeorISO/IEC 10646An international standard that defines every character/glyph used in almost every writing system on the planet, as well as many non-script symbols and character-sized images (such as emoji) used today. Currently (July 2018), Unicode 11 defines 137,439 characters and contains 146 modern and historical text systems!

Unicode also defines several character encodings, includingUTF-8,UTF-16AndUTF-32:

  • UTF-8The first 127 encoding points use 1 bytes (mainly to maintain compatibility with ASCII), and other characters can be added 1-4 bytes.

  • UTF-16/UCS-2Each character is two bytes. UCS-2 (internally used by Windows) Z supports the first 65536 coding points (collectively known as multilingual plane -BMP). UTF-16 extends UCS-2 through 17 extra character planes.

  • UTF-324 bytes per character

Because of its efficient storage requirements and wide use in HTML pages, UTF-8 is currently the most popular coding.

UTF-16/UCS-2 is common, though its usage ratio is decreasing in stored documents (such as web pages, code, etc.). UTF-32 is rarely used because it is inefficient and requires considerable storage space.

Very good, so we have effective and efficient way to represent and store Unicode characters.

So?

Alas, the Windows console and its API were created before creating Unicode.

The Windows console stores text (which is then drawn on the screen) to 2 byte UCS-2 characters per unit.

The command line application writes the text to the console using console API. The console API for processing text has two forms - a single byte / string function with a A suffix processing, and a function with a W suffix to handle a double byte (wchar) / string.

For example,WriteConsoleOutputCharacter ()The function is compiled into WriteConsoleOutputCharacterA () of the ASCII project, or WriteConsoleOutputCharacterW () of the Unicode project. If you need to specify the way of processing, you can call the function A or W suffix directly in the code.

Note: Each W API supports UCS-2 at least because it's something that exists when you do A/W splitting, and we think it would be great to do so. But many W API have been updated to support UTF-16 in the same channel.

. Not all W API can support UTF-16, but all W API can support UCS-2 at least.

In addition, the console does not support some newer Unicode functions, including zero width connectors.ZWJThe symbol is used to connect other individual characters in Arabic and Hindi, and to combine emoticons into a visual glyph!

So if you want to output one on the consoleNinjacatWhat about emoticons or complex multi byte Chinese / Arabia characters? It's awful,You can't do it!

Console API does not support Unicode characters longer than 2 bytes / glyphs.NinjaCatEmoticons require 8 bytes!) But the UCS-2 buffer within Console can't store the extra bytes of the data, and worse, the current GDI based renderer of Console can't even draw a font, even if the buffer can store it!

Deplorable! This is the fun of legacy code.

But I would also like you to stop here. We will return to this topic in a new article in this series. Please pay attention!

So, where are we?

Again, dear reader, if you have read all of the above, thank you, and congratulate you - you are now more aware of the Windows console than most of your friends, even more than you would like to know! I wish you luck!

In this article, we cover a lot of things:

The main building blocks of the Windows console:

  • API Server -- Sends or receives serialized API calls and text data to or from the driver via an IOCTL message.

  • API - the function function of the console.

  • Buffers -- input buffer is used to store user input, and output buffer is used to store output and display text.

  • Input buffer stores user input, outputs buffer output and displays text.

  • VT Parser -- convert the ANSI/VT sequence of embedded text stream to API call

  • Console UX -- user interface status, settings and functions of console

  • Other - Misc life cycle, security, and so on.

  • Condrv.sys - console communication driver

  • ConHost.exe -- console user experience, internals and pipes:

What does the console do?

  • Sending user input to the linked command line application

  • The command line application output that receives and displays the connection.

What is the difference between the console and the *NIX terminal

  • *NIX: "everything is a file / text stream."

  • Windows: "everything is an object and can be accessed through API".

Problems in the console

  • Most of them have been repaired in Windows 10

  • Only ConHost.exe can be appended to command line applications.

  • A third-party terminal is forced to create an out-of-screen console and send it keys and screen information, or receive keys and screen information from it.

  • There are difficulties in remote operation of Windows command line applications and tools.

  • The work of port command line APP from Windows becomes more.

  • The console and command line applications communicate by serializing the IOCTL message consisting of requests and text called by API.

  • Only Windows command-line applications can call console API.

  • The application calls Windows API to interact with the console

  • The dependence on IOCTL breaks the terminal design of the "character exchange" principle.

  • Making it difficult to operate remote Windows command line from non Windows machines

  • Starting the Windows command line application is "not very common".

  • Windows has never identified a ANSI/VT sequence

  • Console support for Unicode is limited and efforts are being made to store and display modern UTF-8 and characters requiring zero-width connectors

In the subsequent articles of this series, we will explore the console in depth and discuss how to deal with these problems. And more other content!

As usual, please continue to pay attention to us.

China IT News APP

Download China IT News APP

Please rate this news

The average score will be displayed after you score.

Post comment

Do not see clearly? Click for a new code.

User comments