Tekstidata

Tekstidata refers to data that consists of written language in digital form. It can be raw text files or text stored in structured formats such as JSON, XML, CSV, or HTML, where the main content is text. The data are typically encoded as a character set, most commonly UTF-8, to support multiple languages.

Sources include digitized literature, newspapers, blogs, social media posts, emails, customer reviews, technical documentation, and web pages. Text data can be monolingual or multilingual and may contain noise such as misspellings, OCR errors, or formatting artifacts. Data quality and copyright are important considerations, especially for large-scale web data.

Processing of tekstidata usually involves preprocessing (tokenization, normalization, lowercasing, stopword removal) and feature extraction (bag-of-words, TF-IDF, word embeddings). Advanced approaches use contextual representations from neural language models. Common tasks include information retrieval and search, text classification, named entity recognition, sentiment analysis, machine translation, summarization, and topic modeling.

Storage and access strategies vary: plain text files, relational or NoSQL databases with text fields, or specialized document stores; indexing engines enable efficient search. Metadata such as language, author, date, and provenance aid discovery and governance. Ethical and legal considerations include privacy, consent, and copyright, particularly for personal or copyrighted text.