when i have
this:
selection end up being:Code:int row=100000;
int col = 100000;
int prec =1;
long selection = ((row * col) * prec)/100;
14100654.
i should be getting 100000000;
Printable View
when i have
this:
selection end up being:Code:int row=100000;
int col = 100000;
int prec =1;
long selection = ((row * col) * prec)/100;
14100654.
i should be getting 100000000;
My guess is when you multiply 100000 * 100000 * 1 the result is past the limit of a long. So it screws up the number when it passes the roof then divides that screwed up number by 100. If you drop a 0 of the end of row and column you notice it will display the appropriate result.
climits (limits.h) - C++ Reference
the limit listed of a long is
2147483647
the result of 100000 * 100000
10000000000
I don't know for sure that this is causing the problem but it would be my first guess.
What he said. :)
You can explore various datatype limits like so:
Code:#include <iostream>
#include <limits>
using namespace std;
int main(void)
{
// Replace long as you please.
cout << "Upper limit of long datatype: " << numeric_limits<long>::max() << endl;
cout << "Lower limit of long datatype: " << numeric_limits<long>::min() << endl;
}
In the code provided in the original post, row and col are int. so row*col is computed as an int. If an int has smaller range than a long, that will also overflow.
"long selection = ((row * col) * prec)/100;"
Even if your compiler's "long" is 64bit, and can store 10^10, this will not work, because all of the operands are int, the result will be truncated to int.
You need to cast one of "row" or "col" to "long" first, for example
"long selection = ((long)row)*col*prec/100;"
((row * col) * prec)/100l might work as well.
(Notice the trailing l.)
No it wouldn't. row*col will be computed as an int, as will ((row*col)*prec). The result would then be promoted to long, in order to divide by 100L.
Incidentally, from a readability perspective, I personally prefer 100L over 100l. I realise others may have different preferences, but lower case L looks too much like the bitwise "or" operator (or like the digit 1) on a lot of screens.
log2(100000 * 100000) is about 33.2 which means that the intermediate value would exceed a 32bit limit. You're also doing division so I wonder if floating point numbers would be more appropriate for your needs? Or else you could use 64bit integer values:
Notice that row is upcast to a 64bit integer so that the intermediate result doesn't get truncated.Code:int row = 100000;
int col = 100000;
int prec = 1;
long long selection = (((static_cast<long long>(row) * col)) * prec) / 100;
Sometimes it is confusing on where to cast or specify the trailing L or d. But I think the rule is that
type1 operator type2 -> type3, where type3 the biggest in size of type1, type2.
if there are integral vs decimal types, the integral are casted to decimals.
I am not sure if float * int would result in float or to double, but I would think to float.
That is a rough summary, but the actual rules are:Quote:
But I think the rule is that
type1 operator type2 -> type3, where type3 the biggest in size of type1, type2.
if there are integral vs decimal types, the integral are casted to decimals.
Quote:
Originally Posted by C++03 Section 5 Paragraph 9