Skip to content

Commit 89829b5

Browse files
authored
[Build] Use C++17 Constructor for tiktoken.cpp when C++20 is unavailable (#5025)
The basic_string_view constructor: ``` template< class It, class End > constexpr basic_string_view( It first, End last ); ``` requires C++20. To allow the code to compile with C++17, use the basic_string_view constructor: ``` constexpr basic_string_view( const CharT* s, size_type count ); ``` For #4661
1 parent 8fb1def commit 89829b5

File tree

1 file changed

+10
-0
lines changed

1 file changed

+10
-0
lines changed

extension/llm/tokenizer/tiktoken.cpp

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,7 +266,11 @@ Tiktoken::_split_with_allowed_special_token(
266266
return std::make_pair(std::nullopt, input);
267267
}
268268

269+
#if __cplusplus >= 202002L
269270
auto start = input.begin();
271+
#else
272+
const char* start = input.data();
273+
#endif
270274
std::string special;
271275
while (true) {
272276
if (!re2::RE2::FindAndConsume(&input, *_special_token_regex, &special)) {
@@ -276,9 +280,15 @@ Tiktoken::_split_with_allowed_special_token(
276280

277281
if (allowed_special.count(special) == 1) {
278282
// Found an allowed special token, split the text with it.
283+
#if __cplusplus >= 202002L
279284
return std::make_pair(
280285
special,
281286
re2::StringPiece(start, input.begin() - start - special.size()));
287+
#else
288+
return std::make_pair(
289+
special,
290+
re2::StringPiece(start, (input.data() - start) - special.size()));
291+
#endif
282292
} // else try to find the next special token
283293
}
284294

0 commit comments

Comments
 (0)