TfidfVectorizer

with torch.no_grad(): doc_outputs = model(**doc_inputs) query_outputs = model(**query_inputs) doc_embeddings = doc_outputs.last_hidden_state.mean(dim=1) query_embedding = query_outputs.last_hidden_state.mean(dim…

dense_scores = get_embeddings([query]).dot(embeddings.T) combined_scores = 0.5 * sparse_scores + 0.5 * dense_scores return combined_scores # Example usage documents = ["This is a sample document.", "Este es un documento de mues…

df = pd.read_csv('data.csv') # Split the data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(df['text'], df['label'], test_size=0.2, random_state=_42) # Feature extraction vectorizer = TfidfVectorizer()…

SVMs can be effective, especially with the right kernel and parameter tuning. ### 4. **Decision Tree Classifier** Decision Trees are simple yet effective for certain types of data and can be used as a baseline. ### 5. **Naive Bayes Classi…

Decision Trees are relatively fast to train and can handle sparse data well. They are particularly useful as a baseline model. ### 4. **Linear Support Vector Machine (SVM)** A linear SVM can be quite fast to train, especially with sparse d…

X_train, X_test, y_train, y_test = train_test_split(df['text'], df['label'], test_size=0.2, random_state=42) # Feature extraction vectorizer = TfidfVectorizer() X_train_tfidf = vectorizer.fit_transform(X_train) X_test_tfidf = vectorizer.tr…