[Nazmul222]

Provides less information specific to a single text). The weights assigned by TF/IDF did not take into account homonyms, synonyms, or word order. So the text "I don't want to dance. I want to read" was equivalent to "I don't want to read. I want to dance”, which, as you can easily gues.

Causes considerable problems in actually understanding what this person actually wants. What's worse, the sentences: "make me a coffee" and "make coffee for me" were not close Phone Number Data to each other at all, which can already lead to domestic drama! Language has a sequential structure and the mathematical models representing it should be approached in this.



Way. This is also the reasoning of the creators of word2vec - a model that assigned numbers to words depending on the context in which a given word appeared most often. In a training set in which we have a huge pool of texts at our disposal (e.g. the National Corpus of the Polish Language, Wikipedia resources, common crawl), we can check in which company particular words usually appear and, on this basis, assign them numerical values ​​so that the dog was close to the dog and far from the tank (unless we teach the model on "Czterech Pancerni"). Since each word carries a lot of information, it is not enough to represent it with a single number. Therefore, in the computer "dictionary", which is the learned word2vec, each word is represented by 300 numbers - that is, in mathematical terms, it constitutes a 300-dimensional vector. This way you can reflect the relationships between words and solve the problem of synonyms. automation_mrk2 We check the probability of the word "dog" appearing after the words "small", "fierce" and, at the same time, the probability of the word "dog" appearing before the words "loud" and "barking". Based on.