Python-Help needed

[ηøℓι]

EN:
Hey, I have to write a function which tokenizes the Input "sentence".
Currently I do that with:

x = sentence.split()

This code works fine.
But now I have to make sure that punctuation (. , : etc.) gets treated like a part of the strings so that e.g.:
sentence = "Hi, Ich bins."

is NOT just:

["Hi," "Ich", "bins."] (like it would be with .split() )

but that the output is:

["Hi", ",", "Ich", "bins", "."]

How can I get to that?
Apparently I need to use a for-loop for it, but I have no clue how.
(i cannot use NLTK or regex)

DE:

 Spoiler

Hey, habe folgende Aufgabe:
Ich habe einen Input "sentence" (ein Satz) den ich splitten muss in eine liste.

Also nehme ich den Input und mache zB folgendes:

x = sentence.split()

Dieser Part funktioniert.
Das Problem ist, jetzt soll ich den Input so splitten, dass er beim Splitten auch auf Punkte, Kommas etc. achtet und diese als einzige Einträge in der Liste sieht.

Also zB der Satz "Hi, Ich bins."
ist momentan mit "sentence.split() folgendes:
["Hi," "Ich", "bins."]
soll aber jetzt folgendes werden:
["Hi", ",", "Ich", "bins", "."]

Wie kriege ich das hin?
Anscheinend muss ich eine for-loop benutzen, aber ehrlich gesagt kp

[cypher]

Ooops no regex. Ok sorry.

[0xFADED]

Originally Posted by ηøℓι

Apparently I need to use a for-loop for it, but I have no clue how.
(i cannot use NLTK or regex)

I dont know python, but basically you just need to loop through the characters and buffer them until you hit a space or punctuation.

Here's an example in C++:

 Spoiler

std::string input = "Hi, Ich bins.";

std::vector<std::string> result;
std::string buf;

for (char const& chr : input) {
    if (chr == ' ') {
        if (!buf.empty()) {
            result.push_back(buf);
        }
        buf.clear();
        continue;
    }

    if (chr == ',' || chr == '.') {
        if (!buf.empty()) {
            result.push_back(buf);
        }
        buf.clear();
        result.push_back({chr});
        continue;
    }

    buf += chr;
}

// result is now ["Hi", ",", "Ich", "bins", "."]

[elmarcia]

Originally Posted by ηøℓι

EN:
Hey, I have to write a function which tokenizes the Input "sentence".
Currently I do that with:

x = sentence.split()

This code works fine.
But now I have to make sure that punctuation (. , : etc.) gets treated like a part of the strings so that e.g.:
sentence = "Hi, Ich bins."

is NOT just:

["Hi," "Ich", "bins."] (like it would be with .split() )

but that the output is:

["Hi", ",", "Ich", "bins", "."]

How can I get to that?
Apparently I need to use a for-loop for it, but I have no clue how.
(i cannot use NLTK or regex)

DE:

 Spoiler

Hey, habe folgende Aufgabe:
Ich habe einen Input "sentence" (ein Satz) den ich splitten muss in eine liste.

Also nehme ich den Input und mache zB folgendes:

x = sentence.split()

Dieser Part funktioniert.
Das Problem ist, jetzt soll ich den Input so splitten, dass er beim Splitten auch auf Punkte, Kommas etc. achtet und diese als einzige Einträge in der Liste sieht.

Also zB der Satz "Hi, Ich bins."
ist momentan mit "sentence.split() folgendes:
["Hi," "Ich", "bins."]
soll aber jetzt folgendes werden:
["Hi", ",", "Ich", "bins", "."]

Wie kriege ich das hin?
Anscheinend muss ich eine for-loop benutzen, aber ehrlich gesagt kp

If you go to the python docs and read string.split function u will find out that you can specify a separator char, then using something like this should do the trick

str = "im a string, and have punctuation too."

print(str.split(" "))

[0xFADED]

Originally Posted by elmarcia

u will find out that you can specify a separator char

Read the question again elmarcia.
OP specifically asked for a solution that treats punctuation like it was enclosed in whitespace.
Split is not enough here, it will yield the false ["string,"] result where a ["string", ","] was expected.

[ηøℓι]

Originally Posted by 0xFADED

I dont know python, but basically you just need to loop through the characters and buffer them until you hit a space or punctuation.

Here's an example in C++:

 Spoiler

std::string input = "Hi, Ich bins.";

std::vector<std::string> result;
std::string buf;

for (char const& chr : input) {
    if (chr == ' ') {
        if (!buf.empty()) {
            result.push_back(buf);
        }
        buf.clear();
        continue;
    }

    if (chr == ',' || chr == '.') {
        if (!buf.empty()) {
            result.push_back(buf);
        }
        buf.clear();
        result.push_back({chr});
        continue;
    }

    buf += chr;
}

// result is now ["Hi", ",", "Ich", "bins", "."]

Thank you, my solution in the end looked like this:
(the code might be weird, but hey.. it works! )

 Spoiler

 x = sentence.split()
    y = len(x)
    f = 0

    for entry in x:
        z = x[y-1][-1]          # last character of last word in sentence
        y -= 1                  # adds -1 to last word so we get 2. last word
        if z == ".":
            w = x[y][f - 2]
            w = w.replace(".", " ")
            w = w + " ."
            w = x[y][:-1] + " ."
            x[y] = w
            #done = x[y].split()

        elif z == ",":
            w = x[y][f - 2]
            w = w.replace(",", " ")
            w = w + " ,"
            w = x[y][:-1] + " ,"
            x[y] = w
            ##done = x[y].split()

        elif z == "?":
            w = x[y][f - 2]
            w.replace("?", " ")
            w = w + " ?"
            w = x[y][:-1] + " ?"
            x[y] = w
            #done = x[y].split()

        elif z == "!":
            w = x[y][f - 2]
            w.replace("!", " ")
            w = w + " !"
            w = x[y][:-1] + " !"
            x[y] = w
            #done = x[y].split()

        elif z == ":":
            w = x[y][f - 2]
            w.replace(":", " ")
            w = w + " :"
            w = x[y][:-1] + " :"
            x[y] = w
            #done = x[y].split()

        elif z == ";":
            w = x[y][f - 2]
            w.replace(";", " ")
            w = w + " ;"
            w = x[y][:-1] + " ;"
            x[y] = w
            #done = x[y].split()

    a = x   # (idk why this is here lol)
    b = " ".join(a)
    kek = b.split()
    #print(kek)
    return kek

and the official / easiest solution seems to be:

emptylist = []

    for y in x.split():                    #x = input-sentence
        if x[-1] in ".,!?;":
            emptylist.extend([x[:-1], x[-1]])
        else:
            emptylist.append(x)

    return emptylist

Glad this could be solved, thank you for helping!