SSL... too many tcp sessions...

All:

I work for an application-service provider.. sort of... anyway. One of our many applications is actually a TN3270/telnet application which has hooks into a microsoft DLL to tunnel/encrypt the user traffic over SSL. So on the network, we see encrypted packets destined to/from port 443.

This traffic hits a load-balancer that offloads the SSL encrypt/decrypt process.

We are having an issue where, it seems, everything a single user does (like just login, or pull up a VT100 screen) ends up launching 8-12 TCP connections. Over the course of a day 160 users launched upwards of 90000 tcp sessions. Each session is very predictable. its the same number of packets, and on the server side follows nearly the exact same sequence numbers every time.

We have many thousands of users that use this application. Needless to say, this is starting to cripple our load balancer.

Its my belief.. as an ex-programmer from a decade+ ago, that for the duration a user is logged into this app there should be a single TCP session.

Does anyone know why this might be occuring or if there is someway of fixing this? I am speaking in the broadest possible terms here because I am a network engineer, not a programmer. I will try to get more details if I can.

Derick
CCIE 15672

SSL is bloatware. Those extra sessions are for key exchange. You would be better off implimenting your own security if you have a mathematician on staff. There realyl is no way aroudn it unless you directly impliment the SSL yourself using the API functions. Going through a 3rd party DLL there is nothign you can do.

So.. I'm guessing that there are multiple passes occuring to InitializeSecurityContext (clienthandshakeloop), and instead of reusing the preexisting connection (I'm guessing by reusing the output-buffer pointer), we are launching a new connection for each exchange with the load-balancer that is doing the SSL offload...


Is that terribly far off? All 8 or so connections are launched in the same 100ms window. This would mean its the last of the 8 connections that does something useful.

Not all SSL stuff behaves this way, not even all Secure32.dll stuff, so I'm guessing its coded wrong.

Im guessing its coded wrong as well. It really shouldnt take more than a single session. Personally I impliment our own encryption routines. Its more secure and faster IMO.

Quote:

---

You would be better off implimenting your own security if you have a mathematician on staff.

---

It could be more secure for you. Maybe. But telling some dude who is probably not an experienced cryptographer to replace a widely used and quite secure system with something they manage to invent is bad advice IMO.

Quote:

---

Originally Posted by mike_g

It could be more secure for you. Maybe. But telling some dude who is probably not an experienced cryptographer to replace a widely used and quite secure system with something they manage to invent is bad advice IMO.

---

I made no such recommendation. As I stated 'if you have a mathemetician on staff'. Not just some yahoo that can add and subtract, but a fully qualified professional mathemetician. Sorry if I didnt put in the 'fully qualified professional' qualification. In general when I say programmer that doesnt mean 'some kid that can write a web page', but 'a fully qualified professional programmer'. Most people understand this.

Quote:

---

Originally Posted by abachler

I made no such recommendation. As I stated 'if you have a mathemetician on staff'. Not just some yahoo that can add and subtract, but a fully qualified professional mathemetician. Sorry if I didnt put in the 'fully qualified professional' qualification. In general when I say programmer that doesnt mean 'some kid that can write a web page', but 'a fully qualified professional programmer'. Most people understand this.

---

I understand what you are saying, but I'm pretty sure that the field of mathematics is wide enough that cryptography is only one of many specialties. Modern cryptography is definitely not a subject that one can quickly get into and just make something that is inherently secure.

SSL has been proven to be good (I'm not saying perfect), and it may be bloated, but I'd rather take the bloated software with known faults than slim software, unless I am totally convinced that the slim software is equally good.

--
Mats

Quote:

---

I made no such recommendation. As I stated 'if you have a mathemetician on staff'. Not just some yahoo that can add and subtract, but a fully qualified professional mathemetician. Sorry if I didnt put in the 'fully qualified professional' qualification. In general when I say programmer that doesnt mean 'some kid that can write a web page', but 'a fully qualified professional programmer'. Most people understand this.

---

A degree in maths does not automatically make someone an expert cryptographer and a 'fully qualified professional programmer' could be any chimp thats managed to get paid for writing code. Even if you find someone that you trust knows what they are doing, would you expect them to write a public key encryption system system for you single handedly within, say, a few months that is free of bugs and security holes? I'm not saying ssl is perfect, but like matsp said its faults are pretty well known by now. I dont know, maybe my skepticism is simply out of fear of what I do not fully understand, but at the end of the day ssl is well tested; anything new wont be.

Edit: sorry, this is going off topic a bit...

Implimenting an existing protocol would still be better than using SSL unless you had to use SSL for some reason like compatability with existing applications. There are plenty of other protocols out there that are secure and easy to impliment that aren't SSL. Since the OP obviously has a problem with the SSL library he is using, I suggested a reasonable alternative. If you would rather stick with something that doesnt work because you are afraid of the maths, then thats your choice.

Quote:

---

There are plenty of other protocols out there that are secure and easy to impliment that aren't SSL. Since the OP obviously has a problem with the SSL library he is using, I suggested a reasonable alternative.

---

Sure, that may be viable, but I cant seem to find the post where you suggested an alternative other then hiring someone to code a new system.

Quote:

---

If you would rather stick with something that doesnt work because you are afraid of the maths, then thats your choice.

---

I'd sooner stick with something where you can accurately expect what the result is going to be than something you cannot. From a business pov the people authorizing this stuff will ultimately be responsible for its success, and its most likely that they wont understand the maths involved. What happens if you pay some genius cryptographer/programmer/mathematician huge amounts of money for something thats fast, but turns out to be broke 1 year down the line and you cant find anyone capable of sorting out the code? It seems like a pretty big risk to me.

Quote:

---

Originally Posted by mike_g

Sure, that may be viable, but I cant seem to find the post where you suggested an alternative other then hiring someone to code a new system.

---

That is the alternative.

Quote:

---

I'd sooner stick with something where you can accurately expect what the result is going to be than something you cannot.

---

The foo fails but at least we know how it fails, so lets stick with something that we know fails. Is that your argument?

Quote:

---

From a business pov the people authorizing this stuff will ultimately be responsible for its success, and its most likely that they wont understand the maths involved.

---

Don't hire smart people, we wont be able to understand what they do?
Wow, sure glad I don't work at your company.

Quote:

---

What happens if you pay some genius cryptographer/programmer/mathematician huge amounts of money for something thats fast, but turns out to be broke 1 year down the line and you cant find anyone capable of sorting out the code? It seems like a pretty big risk to me.

---

Hire a guy to impliment some great new system and then somehow burn all bridges between him and the company so badly that he refuses to support the code? Again, wow...

Ok. I'll point out that I'm no expert here, my opinions are formed on two cryptography books that I have read. One was called 'practical cryptography', the other I forgot the name of. Something that both these books had in common was that they were assertive in their belief that commercial encryption should use well tested proven algorithms and they should be open source so that other cryptographers can rip them to pieces if they a buggy pile of crap.

While I don't doubt that youre a clever guy thats capable of making something like this, I do doubt that you would be able to make one free of bugs. And tbh I hold the books authors' opinions on this matter higher than yours in this case.

Quote:

---

Don't hire smart people, we wont be able to understand what they do?
Wow, sure glad I don't work at your company.

---

I'm not implying that you should not hire intelligent people simply for the sake of that you dont understand what they are doing. With encryption, it seems, that new systems should not be put into use until they are thoroughly tested and evaluated by other professionals in the field.

Quote:

---

Hire a guy to impliment some great new system and then somehow burn all bridges between him and the company so badly that he refuses to support the code? Again, wow...

---

What happens if the guy dies? Or retires? Or decides he/she does not like you any more for for something you feel is not your responsibility? I could go on....

Quote:

---

Originally Posted by mike_g

While I don't doubt that youre a clever guy thats capable of making something like this, I do doubt that you would be able to make one free of bugs.

---

You would be wrong in this case. As usual, while I wont post my credentials in public, I will offer to submit them to a moderator for verification as long as they agree not to release any details.\

And as stated, there are a lot of other security protocols besides SSL that have undergone peer review, so that is an invalid argument.

Quote:

---

What happens if the guy dies? Or retires? Or decides he/she does not like you any more for for something you feel is not your responsibility? I could go on....

---

If he dies!? Well don't hire a CEO because if he dies your business might have problems, don't hire a janitor, because if he dies your floors will get dirty? This is a risk you take with every hiring decision, therefore it is not a valid argument. If he retires, well don't hire someone who plans to retire a week after you hire him. As for the dislike thing, Im really not sure what you mean? Do you make it a habit of hiring emotionally unstable people, or of stonewalling them when they voice a concern, or do you micromanage (i.e. nitpick) your employees to the point they hate you so much they would risk their career's just to get even? Again, wow...

Quote:

---

And as stated, there are a lot of other security protocols besides SSL that have undergone peer review, so that is an invalid argument.

---

Its not an invalid argument. You suggested that they hire a coder to write a new encryption and authentication protocol. It would indicate that it would probably be closed source and insufficiently tested, and anyway why invent something new when there are plenty of well tested protocols that exist. Even Microsoft use open source encryption and authentication.

Quote:

---

If he dies!? Well don't hire a CEO because if he dies your business might have problems, don't hire a janitor, because if he dies your floors will get dirty? This is a risk you take with every hiring decision, therefore it is not a valid argument. If he retires, well don't hire someone who plans to retire a week after you hire him.

---

You really have to take things to extremes don't you :p

Death is unlikely, but possible. A more probable scenario may be that the coder ends up working for someone else, and is simply to busy to fix the problems or maybe even just cant be bothered.

Quote:

---

As for the dislike thing, Im really not sure what you mean? Do you make it a habit of hiring emotionally unstable people, or of stonewalling them when they voice a concern, or do you micromanage (i.e. nitpick) your employees to the point they hate you so much they would risk their career's just to get even? Again, wow...

---

Disagreements happen, and it doesent necessarily mean that anyone involved is emotionally unstable. Again, you are taking a reasonable statement and turning it into a drama. Which is a sign of emotional instability in itself.

There is another issue with hiring one person to be the sole creator of your companies encryption/authentication, which is one of integrity. Whats stopping him/her from adding their own holes to subvert the system? It would certainly be a temptation. If the authentication is compromised, you may never find out - or at least until its too late.

Quote:

---

Originally Posted by ccie15672

We are having an issue where, it seems, everything a single user does (like just login, or pull up a VT100 screen) ends up launching 8-12 TCP connections.

---

Ok, this is almost surely a consequence of the of that DLL you are using. If you look here, you'll see both ports are unassigned. My guess is the DLL behavior is to establish connections on those ports. Why it does it, is left for speculation. What's the DLL?

As far as I know, you are generally right when you say it should use only one TCP connection for the duration of the session. However, something in your description doesn't fit and its probably worth looking at; the immediate problem that caught my attention is that you seem to be establishing connections on the wrong port. 443, aka https is meant for http connections over TLS/SSL. However you mention the use of a terminal emulation application. On this case, the correct protocol should be SSH, Secure Shell, on port 22.

Worth taking a look at.

Meanwhile, is the 8-12 connection being maintained by the DLL? That is, is it listening on one of these two ports, while some other process is connecting through the other port? Which process?

Basically, describe exactly what connections are taking place and their players.

EDIT: There's more to be said about your probable problem. For instance, it's quiet possible the DLL uses that connection to emulate SCP and FTPS connections... but first, your answers ;)

Quote:

---

Originally Posted by mike_g

Its not an invalid argument. You suggested that they hire a coder to write a new encryption and authentication protocol. It would indicate that it would probably be closed source and insufficiently tested, and anyway why invent something new when there are plenty of well tested protocols that exist. Even Microsoft use open source encryption and authentication.

---

Just because its closed source does not mean it doesnt impliment a well tested algorithm, just that the company wont make it easier on potential hackers by telling them which algorithm. That's why I more or less disregard the opinion of anyone that claims you have to make your security open source to be sure its secure.

Quote:

---

You really have to take things to extremes don't you :p

---

You are the one suggesting you dont hire people for complex projects because they might drop dead at any moment.

Quote:

---

Death is unlikely, but possible.

---

Its also possible that a meteor will crush your entire company, o rthat a solar flare will wipe out your bank account. I can see now that the nitpicking result is probably what has happened to you before.

Quote:

---

A more probable scenario may be that the coder ends up working for someone else, and is simply to busy to fix the problems or maybe even just cant be bothered.

---

Then you either downsized them, ........ed them off, underpaid them, or soem other management related issue. If you have soemone that has critical knowledge that yoru company depends upon, you dont just get rid of them, thats stupid and a waste of human capital.

Quote:

---

Disagreements happen, and it doesent necessarily mean that anyone involved is emotionally unstable.

---

Disagreements that can only be resolved by the termination of current and future business relations DO NOT 'JUST HAPPEN'. Someone let their ego get in front of their intellect.

Quote:

---

Again, you are taking a reasonable statement and turning it into a drama. Which is a sign of emotional instability in itself.

---

ad hominem, I win.

Quote:

---

There is another issue with hiring one person to be the sole creator of your companies encryption/authentication, which is one of integrity. Whats stopping him/her from adding their own holes to subvert the system? It would certainly be a temptation. If the authentication is compromised, you may never find out - or at least until its too late.

---

What's stopping them? Oh if 5 to life and restriction from ever using a computer again for tampering with a computer system isnt enough then I don't suppose anything is stopping them. This isnt particular to security though, any programmer could add back doors to any piece of code. If you arent doing a thorough review of the code then that is the real problem, not the potential for abuse. If you can't understand the code then I guess we are back to your argument about 'dont hire smart people'.

This is going nowhere. I said what I have to say ;)

Well then.

Since I originally posted, I have gained a great deal of clarity on what is going on with this app.

First, it is not actually tunneling tn3270 over SSL. It is, in fact, HTTP over SSL. So there is a front-end "option" that makes it look like TN3270, but when the user does a query or fills in the fields on the "screen" and hits send, its actually an HTTP Post that goes out. Users don't have to use the tn3270 screen option. In this case, its a web page with numerous elements. These elements each must hit the server querying for specific information or submitting specific information. All of this is done with HTTP posts.

They are using Wininet.

Essentially every query or transaction for each element results in a new TCP connection over which a single HTTP post is "executed." If an element refreshes... its a new TCP connection. All elements on all pages hit the same server with the same certificate.

Users who do not have publicly registered certificates... each transaction/query ends up being two TCP connections. This happens for unknown reasons, but a capture/decrypt shows the first TCP session there is a certificate/key exchange and then the connection is FIN'd straight away. Then the second connection is opened and the data is exchanged.

So a page with 5 or 6 elements may end up launching 10 or 12 TCP connections straight away to load all the elements.



Isn't this supposed to be what HTTP persistent connections are for? Couldn't they re-use the handle that is returned when they initially login to the server?

yes,they could, apparently they didn't. Welcome to the reality of programming.

Hey one last update..

To make a long story short, our programmers admitted to not implementing HTTP "persistent connections" as specified in RFC 2616, Section 8.1. See HTTP spec here.

So now they are moving forward with that. Also the double TCP connections for unsigned/unregistered certificates is a known issue with the WinInet DLL they were using, so they are abandoning it for winhttp.

The end!

Thanks for the replies. I bought a C programming book because all of this. I should read it one of these days.

Well, given that all programmers are usually working under unrealistic time constraints, you should feel lucky it works at all. I'm rewriting a class that took 2 months to write the first time, so of course the boss expects the rewrite to take 2 days because ' you can reuse the old code'. If the old code was worth reusing, we wouldn't be doing a rewrite, but whatever.