Printable View
how safe is strtok when I get the string from std::string::c_str()? I'd imagine it's considerably safer than if I was dealing with pointers from other origins, but I just figured I'd see what you guys thought. strtok is a lot faster than other methods which use the functionality of std::string or std::stringstream.
Not safe at all since strtok() modifies the null terminated string but the pointer returned by c_str() points to the first character of a null terminated string that is not to be modified.
Since string::c_str() returns a const char* string, you'd need to use a const_cast to pass it to strtok(), and when you use const_cast you're throwing safety right out the window.
If you really want strtok for strings you could roll your own, and I don't think it would be any slower than strtok necessarily, since you would have to do things like make a copy of the string, anyway.
I didn't try it, but it would be something like that.Code:#include <string>
#include <algorithm>
using namespace std;
// PRE: haystack.end() > last >= haystack.begin()
string myStrtok (string& haystack, const string& needle, string::iterator& last)
{
string result;
string::iterator start = last;
last = find_end(last, haystack.end(), needle.begin(), needle.end());
if(last != haystack.end()) {
copy(start, last, result.begin());
++last;
}
return result;
}
The reason why strtok would be faster is that it doesn't create new strings and instead puts null-terminators in the existing one and returns pointers to substrings.
I think boost::split can store the results as a collection of iterator_ranges without copying the substrings. You might try that or use a similar idea. (Because you get the beginning and end of the substrings, rather than a single pointer to a C-string, you won't need to modify the string at all.)
You could try copying the std::string into a std::vector<char> and use strtok() on the vector instead.
and in my case, I'm copying the returned strings into elements of a std::vector<std::string>. Since the original string, returned from std::string::c_str(), is basically guaranteed to be constructed sanely, and the returned value from strtok() is basically guaranteed to be a sanely constructed substring of that original string, I would think that this method would be reasonably safe under most circumstances.Quote:
Originally Posted by anon
please correct me if I'm wrong.
It doesn't matter how safely you deal with the answer, if you're passing a c_str() in ANY form to strtok, then the code is broken at the moment the call is made.
> I would think that this method would be reasonably safe under most circumstances.
Pure dumb luck would be my assessment.
> strtok is a lot faster than other methods which use the functionality of std::string or std::stringstream.
Here's a tip.
Make it 'right' before you try and make it 'fast'.
Other methods would be a lot 'faster' to write and your code would be up and running by now, rather than fiddling with strtok to make it work.
WHEN your program is finished, and you've PROVEN this is a bottleneck with a profiler, THEN you can think about performance tuning.
All the tokenising approaches are fast, compared to say file I/O.
the code has in fact been up and running for almost two years now, and has been working just fine. I just want it to be faster.Quote:
Originally Posted by Salem
it is a bottleneck. I wouldn't be asking these questions if it wasn't an issue.Quote:
WHEN your program is finished, and you've PROVEN this is a bottleneck with a profiler, THEN you can think about performance tuning.
All the tokenising approaches are fast, compared to say file I/O.
doing away with the const_cast, and first copying the string to a buffer on the heap would obviously be safer, and copying a block of memory is relatively fast, so there wouldn't be big performance hit there, and I'm thinking that so long as I check for errors allocating memory, and free it when I'm done, it should be only slightly less safe than std::string and its friends. As long a strtok() and strcpy(), or for that matter, strncpy(), don't misbehave, it should be fine.
Why not do something like this and avoid the dynamic memory allocation?
I haven't tried it, but it should work.Code:std::string str = "One Two Three";
std::vector<char> vec( str.begin(), str.end() );
vec.push_back( '\0' );
const char* result = strtok( &vec[0], " " );
I haven't read that part of the C++ standard, but isn't this legal:
I'd imagine it's guaranteed that &str[0] will be the null terminated string. Or doesn't the data have to be sequential/zero terminated until you call c_str() (in which case, of course, only the returned value is sequential and zero terminated, and &str[0] still might not be).Code:std::string str;
/* Build a string */
char *c = &str[0];
Anybody knows whether it's legal?
At the moment, there is no such definite guarantee, though due to a defect in the standard the wording could be regarded as ambiguous. This defect will be rectified in C++0x.Quote:
Originally Posted by EVOEx
EDIT:
Oh, but if I remember correctly the guarantee will be that the storage is contiguous. I do not think that there will be any guarantee of null termination (such a change would not make sense), so you would have to add a null character to the end in order to use &str[0] with strtok().
which brings me back to using strncpy() to copy to a buffer allocated on the heap, and when I run a loop ahead of the strtok() call to determine the number of elements (I'm only using a single character as a delimiter anyway), and reserve space in the vector in which the results are stored, the difference in performance is negligible between strtok() and std::getline().Quote:
Originally Posted by laserlight