[discussion] Python makes it harder to reason about your objects

[Mario F.]

... and to unit test, btw.

(This isn't about Python specifically. This is about dynamically typed languages in general that are only concerned with whether a given class supports the right interface. I've just been using Python recently and that's the only dynamically typed language I'm presently exposed to)

Consider for a moment the following Python class which attempts to create a reasonable set of rules to validate an instance property:

class Player:
    @property
    def name(self):
        return self._name

    def __init__(self, login, name):
        self.login = login
        self.name = name

    @name.setter
    def name(self, v):
        if v is None:
            self._name = None
            return

        try:
            v = v.strip()
        except:
            raise TypeError('Must be a string or None.')

        self._name = v if v else None

What @name.setter essentially does is to check if name is a whitespace or empty string and cast it to a None object if that is the case. Seems an awful lot of code to write for such a simple rule.

I could write the following simpler version (and that's what I have in my own code when doing this kind of stuff):

    @name.setter
    def name(self, v):
        if not isinstance(v, string):
            raise TypeError('Must be a string or None.')

        v = v.strip()
        self._name = v if v else None

But notice how I am now breaking the contract you sign when adopting any dynamically typed language;

"if you are reasoning about your object type, your design has a weakness. Your code should only be concerned if an instance supports the right interface."

Those are pretty words. But the fact is that I can have any object that supports the strip() method that returns something completely different. An interface surely isn't just a name, but the semantics that go into that name.

This means that without checking for a string instance, not even wrapping v = v.strip() inside a try block, will avoid potential unhandled exceptions in the code that makes use of the Player class.

The previous, more verbose, version of the setter method doesn't perform type checking so I must be doing it right. Or am I?

The fact is that I'm a programmer and I think like a programmer. Even with dynamically typed languages we think in terms of object types. So I will be coding the rest of my program in the knowledge that the Player.name property is a string and I'll be performing all sorts of string related things with my the Player.name property.

The problem with that is that you may have already noticed that my longer version setter has a bug. It allows for any object type that implements a method named strip() to be assigned to Player.name. Remember, an interface is not just a name, but the semantics that go with that name. Problem is dynamic programming languages are only concerned with the name part. But it is very likely that hypothetical object strip() method doesn't even return a string. So I really need to cover this up in my setter. And the only way to do it is to type-check.

What makes this bug more devilish is that if my class is part of an API, users of my class will correctly make very few assumptions about my API implementation details (including property internal types) and will get a cryptic unhandled exception when the bug surfaces in their code. We all have seen those.

And to put the cherry on top of that burnt cake, I'm also obliged to include a type-test in my unit test suit for that class.

--------------------------------
So, let's finalize with a bit of advice:

Do check your types in your dynamic typed languages. It really is ok. Don't buy entirely into the idea that your programming language will keep you safe from type declarations. No programming language does. And dynamic typed languages are no different.

But do make an effort to avoid checking your types. The really ok place to type-check is within your own objects implementation details, where you may need to perform validation to avoid raising unnecessary exceptions to the caller or to implement proper procedures for when the requirements of the object interface aren't met.

[laserlight]

I could write the following simpler version (and that's what I have in my own code when doing this kind of stuff):

I am afraid that I am not so familiar with Python 3 since I code on Python 2.7 as my day job, but it seems weird that None would be an instance of string. Creating a Python 3.4 virtualenv and attempting to test, I found that string is not even a built-in type, and of course isinstance(None, str) is False. Did you test your example code? It seems to me that you still need to handle the case where v is None.

I suggest a small modification:

@name.setter
def name(self, v):
    if v is None:
        self._name = None
        return

    try:
        self._name = v.strip() or None
    except AttributeError:
        raise TypeError('Must be a string or None.')

I guess you could compress it into:

@name.setter
def name(self, v):
    try:
        self._name = (v.strip() or None) if v is not None else None
    except AttributeError:
        raise TypeError('Must be a string or None.')

but that could be too complicated (like embedding a ternary operator expression in a ternary operator expression).

EDIT:
Oh, here's another version that comes to mind:

@name.setter
def name(self, v):
    if v is not None:
        try:
            v = v.strip() or None
        except AttributeError:
            raise TypeError('Must be a string or None.')

    self._name = v

I think I like the third the best. But do you really need to translate the exception in the first place?

[Yarin]

[discussion] OOP makes it harder to reason about your objects

FTFY

... dynamically typed languages in general that are only concerned with whether a given class supports the right interface. ...

This is known as duck typing.

It allows for any object type that implements a method named strip() to be assigned to Player.name. Remember, an interface is not just a name, but the semantics that go with that name. Problem is dynamic programming languages are only concerned with the name part. But it is very likely that hypothetical object strip() method doesn't even return a string. So I really need to cover this up in my setter. And the only way to do it is to type-check.

You are not alone in your observation. I consider this to be, by far, the largest shortcoming of most dynamicly typed languages (as compared to static typing). But the fault here isn't actually the dynamic typing, it's that the language doesn't provide a convenient way to match types. Multimethods are the perfect solution, but most language designers (even those of staticly typed languages ) haven't figured this out yet.

[laserlight]

It allows for any object type that implements a method named strip() to be assigned to Player.name. Remember, an interface is not just a name, but the semantics that go with that name. Problem is dynamic programming languages are only concerned with the name part. But it is very likely that hypothetical object strip() method doesn't even return a string. So I really need to cover this up in my setter. And the only way to do it is to type-check.

I think that's the philosophical difference here, i.e., by attempting the operation then checking for AttributeError (or allowing it to propagate) rather than checking for the type, any string class that implements strip can be used, even one that is not derived from str. The disadvantage is as you mentioned: any non-string class that implements strip such that it can be called without arguments can also be used, yet that hypothetical situation is rather unlikely, even though if it does happen, it is indeed very likely that its strip method does not return a string.

[Mario F.]

it seems weird that None would be an instance of string.

Oops. I didn't test my code and forgot about making the necessary None check. Thanks for clearing that up with your own version.

Oh, here's another version that comes to mind

I'd go with it, since it is more specific. But everyone is so used to AttributeError in class properties that is probably better to not chain the exception but to just provide a new description. That is, the slightly uglier...

@name.setter
def name(self, v):
    if v is not None:
        try:
            v = v.strip() or None
        except AttributeError:
            raise AttributeError('Must be a string or None.')

    self._name = v

Multimethods are the perfect solution, but most language designers (even those of staticly typed languages ) haven't figured this out yet.

That was a laugh. Single dispatch in a language that implements static typing is complicated and fragile. I guess that in Go function signatures are not overridden, they are overrated.

I was with Phantomotap when he was saying the other day we just need programming languages to not artificially remove features just because they think they know better how we should code our stuff. But yeah, if Go can't come up to terms with single dispatch, it surely won't ever have multiple dispatch.

But more to the point, I'm just not visualizing how multiple dispatching could help deal with the problem of dynamically typing languages.

[phantomotap]

Even with dynamically typed languages we think in terms of object types.

O_o

I don't. I just don't care, for example, if something is in any way related to the type `std::string' or any similar as long as the instance I get has the expected "stringish" semantics I require.

However, I also don't care if people do something stupid so the information isn't all that relevant to the majority of my code.

It allows for any object type that implements a method named strip() to be assigned to Player.name.

You don't have any such bug.

/* We expect to get something "stringish" of at least two in length. */
/* We return something "stringish".
template
<
    typename FStringish
>
FStringish Trim
(
    const FStringish & fData
)
{
    return(fData.substr(1, fData.length() - 2));
}

The example code for `Trim' doesn't have a bug. The interface says "Don't give me something that isn't "stringish".". The interface says "Don't give me something that isn't of length two or greater.". Code that violates the interface has the bug.

Trim(0);
Trim("1");

I'm just not doing anything to prevent idiots from misusing the code. I could only do something to help people debug just to be nice. (I could "SFINAE" over the members or "cassert" over the length.) I'm, in fact, all for people writing code which may aid me in debugging when I misuse an interface. (A debugging aid doesn't prevent misuse. A debugging aid only normalizes behavior when an interface is misused.) I can't do anything in `Trim' that actually prevents misuse. I can't do anything which says that the "stringish" type is actually something "stringish" because any number of mechanical checks will not guarantee the semantics of what we expect when we see a "stringish" type.

I could try some code which is more explicit as you are considering.

/* We expect to get something "stringish" of at least two in length. */
template
<
    typename FStringish
>
std::string Trim
(
    const FStringish & fData
)
{
    std::string s;
    // We copy fData[c] from `1' to `fData.length() - 2' into the `s' instance.
    return(s);
}

I could do something nearly equivalent to `self._name = str(v).strip()' which you don't seem to want.

std::string Trim
(
    const std::string & fData
)
{
    return(fData.substr(1, fData.length() - 2));
}

I'm losing, in both cases, functionality in terms of polymorphisms because I've started caring about the type.

I don't care about the type. I only care about "stringish" semantics.

Unfortunately, I can't guarantee "stringish" semantics. I could throw any number of "SFINAE" over properties and interfaces, but I will always need to assume such properties and interfaces represent "stringish" semantics.

Happily, I don't care about people desperate to do something stupid. I don't care about anyone who would choose to use my `Trim' interface for a class which isn't "stringish" yet offers `substr' and `length' interfaces. I don't care if such a person gets errors. I don't care if such a person gets exceptions. I don't care if such a person sees unexpected behavior.

You shouldn't worry about people misusing your code. You can't protect yourself or your code from people desperare to do something stupid.

Do check your types in your dynamic typed languages. It really is ok.

Sure. Inspecting the actual type of reference is fine in some few cases.

You don't really have one of those cases.

@name.setter
def name(self, v):
    self._name = v.strip()

The example code doesn't have a bug.

We aren't doing anything to normalize the behavior if `v' doesn't have the semantics we need.

So? A client passing some object that doesn't have the semantics we require has a bug. The majority of Python implementations have robust debugging support. The developer of the client can debug their code, ignore the bug, or use a different library at their leisure. We don't care.

All the developers writing clients which use our interface correctly sees the behavior they expect.

But everyone is so used to AttributeError in class properties that is probably better to not chain the exception but to just provide a new description.

You don't need to transform the exception to provide a new description. Once again, most Python implementations have robust debugging support. A trace is more valuable than any one change in description, and you are changing the origin aspects of a trace when you decide to swallow an translate--description or type--for whatever reason.

Soma

[phantomotap]

The example code for `Trim' doesn't have a bug.

^_^;

Except for that missing '*/' I somehow failed to copy...

Soma

[laserlight]

Except for that missing '*/' I somehow failed to copy...

That's not a bug; that's a compile error.

[phantomotap]

That's not a bug; that's a compile error.

O_o

Can't tell if teasing...

o_O

I definitely made a mistake in either copying the comments or writing them in the other window introducing a bug.

I appreciate that a compiler error may be easier to catch than other bugs, but I do definitely consider code which doesn't cleanly compile to have bugs.

*shrug*

As far as it goes, your post history suggests that you also consider code which doesn't cleanly compile to have bugs.

[Edit]
Or had you just been saying "The code for `Trim' itself doesn't have a bug? O_o
[/Edit]

Soma

[MutantJohn]

Hmm... I really consider compiler errors to be more like correcting someone's syntax or grammar rather than a bug which I consider having unintended consequences from a piece of code. I feel like bugs are, "oh, you forgot that other functions can render that data invalid which is why you seg fault in this function" wheres compiler errors are more like, "bro, you forgot a semi-colon" and then I'm like, "Oh compiler, you catch everything I don't."

[Yarin]

But more to the point, I'm just not visualizing how multiple dispatching could help deal with the problem of dynamically typing languages.

Consider your code

@name.setter
def name(self, v):
    if v is not None:
        try:
            v = v.strip() or None
        except AttributeError:
            raise AttributeError('Must be a string or None.')

    self._name = v

Now pretend we could annotate methods with parameter types

@name.setter
def name(self, v : string):
    self._name = v.strip()

@name.setter
def name(self, v : None):
    self._name = v

It's much easier on the eyes, and conceptually must simpler (very important to me because I'm dumb).
Now, if you call the name setter where 'v' is not a None or string, the interpreter would just raise an error, along the lines of "No method name.setter where 1st argument is an Integer".

O_o

Can't tell if teasing...

o_O

Soma, I don't think I'll ever unhear you as Fry.

[phantomotap]

Hmm... I really consider compiler errors to be more like correcting someone's syntax or grammar rather than a bug which I consider having unintended consequences from a piece of code.

"Let's eat Grandpa."

Soma

[Mario F.]

Happily, I don't care about people desperate to do something stupid. I don't care about anyone who would choose to use my `Trim' interface for a class which isn't "stringish" yet offers `substr' and `length' interfaces. I don't care if such a person gets errors. I don't care if such a person gets exceptions. I don't care if such a person sees unexpected behavior.

This is true. It's been a bumpy ride coming from statically types languages into dynamically typed ones. Previously type checking was handled for me by the static compiler and I have been having an hard time shrugging off the feeling I'm need to check my types. I suppose I'll eventually get the hang of it.

You shouldn't worry about people misusing your code. You can't protect yourself or your code from people desperare to do something stupid.

This is in fact one of the core principles of Python. A gentlemen's contract between the API and its users. It's however not always so obvious to the user the semantics of an interface. And that can lead to hard to find bugs or cryptic exceptions. More below...

So? A client passing some object that doesn't have the semantics we require has a bug. The majority of Python implementations have robust debugging support. [...] All the developers writing clients which use our interface correctly sees the behavior they expect.

The semantics are many times hidden in the API implementation. They are not always so obvious. I do agree that in my simplified example that is not the case. But in a dynamically typed language, an important piece of information is missing from the method signature.

And while good quality code can solve most cases, when your API needs to communicate with other APIs, things can quickly become... er, less simple.

Consider SQLite and some API of yours that is trying to abstract it away from the user. You have a last_dated member in your API Friend class.

class Friend:
    def __init__(name, last_dated=None):
        name = self.name
        last_dated = self.last_dated

Friend.last_dated can be anything. Maybe UTC, epoch, a local date object, a local datetime object. Or maybe a string. You will surely choose some sort of implementation and will communicate that implementation to the user of the class by properly documenting it. Meanwhile in the SQLite database you have the last_dated column type set to DATE.

You can easily implement the necessary adapter and converter in python's sqlite3 API. The library already includes a roundtrip conversion from python's datetime.date to SQLite's DATE. That is no problem.

But SQLite3 itself doesn't care about a column type. It's type affinity methodology will actually allow the user of your Friend class to set the last_dated column to something nonsensical like the integer 13. And that becomes a problem when you later implement the Friend class load() method. It will raise an exception when trying to read from the database the improperly formatted DATE column.

Your gentleman's contract (the "I don't care if users misuse my API") actually allows corrupt data to be sent to the database. I think -- but correct me if I'm wrong -- that in these circumstances, you need to code a more robust API; the SQLite3 API protocol needs to flow into your own API, if you want to successfully abstract it away from your users.

class Friend:
    def __init__(name, last_dated=None):
        name = self.name
        last_dated = self.last_dated
    
    @property
    def last_dated(self):
        return self._last_dated

    def last_dated(self, v):
        if v and not isinstance(v, datetime.date)
            raise TypeError('last_dated must be datetime.date')
        self._last_dated = v if v else None

And because our APIs rarely stand alone, these type of issues will happen over and over again in the code. The need to properly implement communication protocols between different APIs, will force you to do more checking if you wish to code a robust API.

[Mario F.]

Now, if you call the name setter where 'v' is not a None or string, the interpreter would just raise an error, along the lines of "No method name.setter where 1st argument is an Integer".

Good example. For some reason I wasn't seeing it. It forces the language into static type land, which is why I was not seeing it (I was assuming a strict dynamic typing approach).

Other than type annotations (which would require the compiler interpreter to be changed), it could be implemented through a callable class or function that stored a map of types. Interesting.

[phantomotap]

And while good quality code can solve most cases, when your API needs to communicate with other APIs, things can quickly become... er, less simple.

O_o

And because our APIs rarely stand alone, these type of issues will happen over and over again in the code. The need to properly implement communication protocols between different APIs, will force you to do more checking if you wish to code a robust API.

o_O

the SQLite3 API protocol needs to flow into your own API

O_O

The semantics required of a given input is always potentially unrelated to types involved in any language. I am certainly not required to check the types passed to my interfaces regardless of the required semantics because they are not necessarily related. I, obviously, can't without introducing my own bugs call on an interface which requires a specific type with invalid types. I, however, only need to provide a correct type when I call upon such interfaces I use in implementing my code to be free of bugs which in no way limits the types I may work with in my implementation.

[Edit]
I moved the more elaborate examples further down.
[/Edit]

def last_dated(self, f):
    self._last_dated = dateish(f)

I would write such code as the example by implementing a component (I've provided a very simple example.) which attempts to coerce any input into something I can use because I do not care about the type or unrelated semantics of the input.

If my requirements included value or relationship semantics, I will still not care about the type or unrelated semantics. If I need to validate input, I will accomplish validation through construction of an object representing those semantics after any such coercion I find useful allowing any errors from the constructor to propagate without translation.

I still don't care about the types.

I still don't care if the object I'm given offers unrelated semantics.

I still don't care about people violating the contract.

Do you provide something "dateish" to my interface?

If so, the code will behave as expected.

If not, the code will likely not behave as expected.

Soma

# The example isn't nearly complete.
import datetime
import locale

try:
    locale.nl_langinfo(locale.D_FMT)
    def dateish(f):
        f = str(f).strip()
        try:
            return datetime.datetime.strptime(f, locale.nl_langinfo(locale.D_FMT))
        except ValueError:
            return datetime.datetime.strptime(f, "%Y-%m-%d")
except:
    def dateish(f):
        return datetime.datetime.strptime(str(f).strip(), "%Y-%m-%d")

[Edit]
*): I felt something was missing. As is, we are doing something very trivial with a value before we assign the value to a member variable. Keep in mind, I would absolutely follow the same guidelines of "I just don't bloody care." even if we were doing something more complex.

*): Let's say we needed `argument.encrypt(argument.decrypt() + "whatever")' to have some expected semantics as part of our implementation before we needed to use the argument in the form of a specific type to be passed to a library interface. I still wouldn't care about the types or unrelated semantics; I'd write the code without ever caring that someone might provide an object which doesn't conform to those semantics.

*): Let's say someone creates a class with a string conversion method which always returns the standard epoch ("1970-01-01") for whatever absurd reason. The class has the appropriate semantics as required. I care about the required semantics. I don't care that the behavior of the conversion is stupid. If some idiot chooses to use such a class as part of using a database against my discussed interface, the idiot would not likely get the behavior they expect. I don't care about that idiot. If they don't like whatever behavior they see, they are certainly free to not use my code.
[/Edit]