Pseudo-ttys: both BSD and SVR5 versions on Linux seem to be incomplete.

[kdd21]

I'm trying to complete the porting to RHEL 5 of a commercial application that we've had running on AIX and SCO for decades. It originally ran on dumb-terminals, and even now runs on dumb-terminal emulators that run on Windows. We have thousands of customers, many of whom we'd like to migrate to Linux. We're pretty much done with the port, with the exception of this little glitch we're having with pseudo-ttys...

Our app utilizes pseudo-ttys, such that sub-portions of the application run under a screen manager that provides for screen-switching of multiple instances of the sub-apps on non-GUI terminals, some of which are still running serial. In this regard, it is not unlike the GNU "screen" program. However, it also supports file transfers over serial links using zmodem, and requires the ability to switch in and out of a fully 8-bit transparent mode where the screen switching keyboard commands must be ignored.

On AIX, we're using the BSD TIOCUCNTL (UIOCCMD) capability to send commands from the sub-apps to the screen manager. TIOCUCNTL provides custom user ioctls and is an ideal way to solve the problem. It is not implemented on Linux, and the "alternative" of TIOCPKT does not provide arbitrary ioctls so at best we'd have to commandeer some of the existing TIOCPKT ioctls for our own use if that's even possible. TIOCPKT seems like a hack, it apparently was implemented not as a general mechanism but to solve problems specific to rlogin/rlogind.

SCO didn't support TIOCUCNTL either, so years ago we used the SVR5 alternative of putmsg/getmsg for the SCO version. This is a streams-based mechanism of sending control information over streams separate from the data. While RHEL 5 seems to have man pages indicating these commands exist, and in fact you can link programs using them to libc without errors, they are apparently stubbed out and return "function not implemented." This seems to be, according to this Wiki, due to certain Linux factions deeming that streams are "technically inadequate":

STREAMS - Wikipedia, the free encyclopedia

Certainly, I would agree that a streams implementation that is incomplete is undoubtedly "technically inadequate."

At the same time, it would appear that the SVR5 method of pseudo-tty is now the recommended implementation, the BSD one being "deprecated" (not to mention, also incomplete on Linux, given TIOCUCNTL is missing):

pty(7): pseudo-terminal interfaces - Linux man page


So here I am trying to figure out how to solve the problem. I have found a library that apparently implements the getmsg/putmsg commands for Linux, called libLiS. However, according to IBM, it has a problem with SELinux:

IBM - Getting CS Linux to work with SELinux (Security Enhanced Linux) on RHEL5

We've had to disable SELinux for other reasons anyway, so this is not a show stopper for us, and I am actively exploring using it as our solution. However, I've never been exactly crazy about the idea of depending on disabling security features to make something work. But, there's something to be said for the compatibility it would offer for our apps. And, putmsg/getmsg are apparently POSIX, so it seems to me they should be a reasonable thing to use...

However, it occured ot me that we are probably not the first to have this problem, and was wondering if perhaps there is another solution that we are unaware of and should consider?

It has occured to me to use named pipes, but this would be rather ugly--we'd have to maintain a directory full of hundreds of named pipes, probably named after the associated ptys that the sub-apps could use to issue control commands to their screen manager, and the manager would have to monitor these pipes for commands, etc.-- certainly possible, but it seems like it might be re-inventing the wheel a bit, or at the very least, is a somewhat kludgy work-around to the apparent lack of any user-customizable control mechanisms in the pseudo-ttys. And we'd have to support it in addition to the code for the other methods since those platforms aren't going away anytime soon (SCO death-throes notwithstanding).

So does Linux have some other method for sending user-defined controls in the ptys (slave-to-master) other than TIOCUCNTRL or putmsg/getmsg?

I'm also not sure where best to ask this question-- this might be one that some actual Kernel developers would have opinions on, but I don't know where they hang out. Googling "linux programmer forum" or "linux developer forum" gets me dozens of hits, and I suppose I'll try this on a couple of them and see if I have any luck, or perhaps I'll get redirected to the "right" place to ask...

--

Sync

[Codeplug]

Sockets?

gg

[Codeplug]

Posix message queues?
http://www.opengroup.org/onlinepubs/...ns/msgget.html
http://www.opengroup.org/onlinepubs/...ns/msgrcv.html
http://www.opengroup.org/onlinepubs/...ns/msgsnd.html

D-Bus?
http://www.freedesktop.org/wiki/Software/dbus

gg

[kdd21]

Thanks for the responses, but the problem is-- in order to implement either of these solutions I would have to create a set of matching sockets or message queues-- e.g., for each master/slave pty connection I create, I would have to create the matching structure for control communications, simply because Linux seems to have no equivalent to the integrated pty control functions of either BSD or SVR4 (TIOCUCNTL or streams putmsg/getmsg). Managing that duplicate set of entities is what I'm trying to avoid, since on a given system there will often be hundreds of PTYs operating. I have to then make sure that if a session gets killed and didn't clean up its queues or named sockets or pipes or whatever, leftover junk doesn't build up, etc.. It's a royal pain, the usefullness of TIOCUCNTL or putmsg/getmsg is really pretty significant for this sort of application. If I was to go that route, possibly even using mmap() would be simpler, though I don't know if there are any considerations if you have a single mmap'ed table (status-byte per pty, or something like that), mapped by hundreds of programs simultaneously-- hopefully none but my experience with that is limited...

I'm currently looking into the possibility of repurposing status bits in the TIOCPKT mode, or perhaps using the two unused bits. Not pretty, but at least then the pty functionality would then be a little more compatible with other Unices...

I was hoping to hear that someone else had some success with the fast-Streams library or that there is some facility in the Linux pty driver that I haven't spotted yet that would solve this sort of problem. I tried to build the fast-Streams library for RHEL 5 (no binary RPMs were available), but for some reason once building the binary RPMs, the base rpm complained about a dependency "STREAMS" which is what it was supposed to be itself I think-- so the error seemed a bit non-sequitur. I suspect there's something munged in my RPM build environment, this is the first time I've tried to build binary RPMs... In any event, I'm realizing that we would have to support the library if we went forward with it since it's not integral to RHEL, such that when a new release of RH comes out we'd have to re-release the RPMs and all that, a business that we'd just as soon not be in-- so that's making the TIOCPKT hack a little more attractive. Using system features in ways they weren't intended in order to make up for non-implemented features. Sounds like par for the course...



--

Sync

[jeffcobb]

Well no offense to anyone here but even though many of us are hard-core Linux coders, we are an overwhelmingly small minority here. I am surprised some Windows knob hasn't suggested .NET yet...