badge icon

This article was automatically translated from the original Turkish version.

Blog
Blog
Avatar
AuthorCan CandaşNovember 4, 2026 at 4:00 AM

News Automation: A End-to-End Workflow from Source Collection to Broadcast

Quote
T3 AI Developer
T3 AI Developer

Following numerous news sources is time-consuming; the repetition of the same story across different sources and the inclusion of unnecessary details in news articles further complicate the process. This problem can be mitigated by a news automation system that collects online news and filters and summarizes only the content relevant to the reader.


The figures provided throughout this text are drawn from an eight-day measurement window of a pipeline built using this approach. This setup monitors over 450 sources (mostly RSS feeds, some YouTube channels, and a few event websites; the list varies daily) under a dozen categories and runs several times a day. Over the measurement window, an average of 2,650 raw news items were collected daily, of which 1,120 were categorized, verified, and summarized; only the best item from each category is delivered to the reader. Unless otherwise stated, all numbers reflect the median values of this window.

Steps of the News Automation Pipeline from Source Collection to Summarization via LLM (Generated by Artificial Intelligence.)


Such a pipeline can be constructed through five fundamental steps:

  1. Source aggregation: Really Simple Syndication (RSS) feeds and site-specific solutions for websites that do not offer RSS
  2. Cleaning news text of non-news elements such as advertisements, code snippets, and site layout, followed by early filtering using simple rules
  3. Deduplication of repetitive news items and categorization using embedding models
  4. Source selection, determination of news importance scores, verification, and merging of articles covering the same event
  5. Translation, summarization, and completion of missing information using Large Language Models (LLMs)

Each of these steps acts as a filtering stage: the pipeline operates like a funnel, reducing the number of news items that reach the more expensive downstream processes. Approximately 42 percent of daily incoming news items make it to the end of the pipeline; the rest are eliminated along the way as duplicates, out-of-scope, low-quality, or unverifiable content.

Decrease in Number of News Items Across Stages in the Average of Five Criteria (Generated by Artificial Intelligence.)

The Problem of Multi-Source News Monitoring

News is a vital means for people to stay informed about current developments in topics that interest them. The earliest sources considered news were official daily public announcements in Rome during the 1st century BCE (see Acta Diurna: Daily Affairs).【1】 Since then, the means of accessing up-to-date information have evolved: in the modern era, newspapers fulfilled this need, while today television and internet sources have replaced newspapers.


With globalization, news is no longer confined to local events. Factors such as population growth, industrial development, and globalization have led to a high volume of potential news daily. In response to this demand, numerous news channels, websites, and social media accounts have emerged. However, many sources share the same stories and burden readers by including redundant details. Moreover, to comprehensively monitor news across desired categories, one must follow many different sources.


Scale provides context: to comprehensively monitor just a dozen topic categories, over 450 sources are required, generating 2,000 to 3,000 raw news items daily. It is impossible for a reader to manually sift through all of them; more than half of these items are either duplicates or out-of-scope content.


To address this issue, online news can be collected and processed to filter and summarize only the articles that are relevant and useful to the user. This significantly reduces the time readers spend monitoring multiple news sources while enabling them to obtain more comprehensive information.

Collecting News Sources: RSS and Site-Specific Solutions

One of the most important steps in building such a news automation system is identifying the sources entering the system and preparing them in appropriate formats with suitable functions. RSS offers a practical solution for this need.【2】 RSS is a system that summarizes the content of websites in short simple and structured formats. With minor variations RSS typically includes essential information such as the article title link address brief description and publication date. These elements can be easily integrated into a system and RSS is widely used in applications such as new post notifications.


Example RSS entry (fictional content):

<item>

<title>Team A defeated Team B 1-0</title>

<link>https://www.example.com/news/12345</link>

<description>A brief summary of the match.</description>

<pubDate>Tue 06 Oct 2026 14:30:00 +0300</pubDate>

</item>


Since RSS feeds contain only text their size is very small. This makes requesting data from these feeds a low-cost and fast process. However while many news sites offer RSS feeds not every website has an RSS source.


Incremental Crawling: For each source the timestamp of the last successful crawl and previously seen record identifiers are stored. Only new records are fetched during each run. Thus the cost scales not with the number of monitored sources but with the number of new articles published during that interval. The crawler runs several times per day; the morning run which collects overnight accumulations is the largest (over 800 raw articles in nearly an hour) while daytime runs average about 480 articles and half an hour.


Parallel Crawling and Error Isolation: Sources are crawled simultaneously within a thread pool and each source’s errors are contained within its own context: a faulty feed does not bring down the entire run; instead the source is marked as “failed” and retried in the next run. Measurements show this is not a refinement but a necessity: approximately 12 percent of sources fail during each run and the majority of these (about 50 sources) have failed during every run for eight consecutive days (due to moved closed or bot-blocked feeds). Within the same window only two-thirds of sources have produced at least one article and the top 20 most efficient sources account for roughly half of the total volume. The source list is not static; dead feeds must be regularly purged.


RSS often carries only summaries, many sources include in their RSS feeds only the title and a sentence or two; the full article resides on the webpage. If the text is below a certain length the target page is downloaded and the article body is extracted using a dedicated boilerplate removal library that strips away advertisements navigation menus and site templates.【3】 This process constitutes the primary burden of the next cleaning stage. Some RSS feeds link not to the actual article but to a news aggregator’s redirect URL; resolving the true source requires URL resolution which demands speed limits and backoff logic.


For websites without RSS feeds site-specific solutions must be developed for automation. These solutions are complex and therefore costly to integrate and cause slowdowns and excessive resource usage during operation. As a result the crawler extends selectively beyond RSS: for YouTube channels the video transcript is extracted (videos longer than 30 minutes are skipped and long transcripts are shortened while preserving the beginning and end); for event sites the event name date and location are collected in a separate stream. Other social media sources can be added using the same logic.

Cleaning and Early Filtering of News Text

After sources are collected, various filtering steps are required:

  • Removal of sections not part of the news text
  • Deduplication of identical news items from different sources (addressed in the next section)
  • Exclusion of news on unwanted topics (addressed in the next section)

Sections not part of the news text encompass a broad range including advertisements, structural code fragments, unwanted content (spam), and site layout elements, making this cleaning process comprehensive. During this stage, certain information such as the news source and publication date is separately preserved for use in subsequent steps. Each news item is assigned an identifier derived from the cryptographic hash of its normalized content; if the same news appears twice in two separate feeds, it receives the same identifier. This identifier forms the basis for deduplication and tracking whether “this news has been published.”


Immediately following cleaning, an early filtering step occurs without invoking any models. The rules are inexpensive and definitive.


Table 1: Early Filtering Rules

These rules filter out approximately 12% of raw news items in this setup. What matters more than the absolute number is the ranking: each filtered-out news item avoids a cost in the embedding and LLM processing stages.

Grouping and Categorization of News in Embedded Space (Generated by Artificial Intelligence.)

Deduplication and Categorization Using Embedding Models

Structural functions are insufficient for identifying and removing duplicate or unwanted topics. A cost-effective and high-quality solution for this task is to use embedding models. An embedding model converts a text into a numerical vector that represents its meaning; vectors of semantically similar texts are positioned close to each other in vector space. The proximity between two vectors is measured using cosine similarity: 1.0 indicates identical direction, 0 indicates no relevance. The role of the embedding model here is to group duplicate news articles by placing their vectors close together in the embedding space.

Distribution of Embedding Vectors Reduced to Two Dimensions with PCA (Generated by Artificial Intelligence)


A similar logic can be applied to categorization: if categories are defined by appropriate and comprehensive sentences and placed in the embedding space, content located near those defining sentences can be assigned to the corresponding category.


At this stage, the capabilities of the embedding model are decisive; the model must specifically support the language or languages in use. The embedding step takes approximately 10 seconds per full run and accounts for less than one percent of the total pipeline cost, making it the cheapest component. Vectors are stored in a local database, with each record having a 14-day lifespan that defines the deduplication window.

Three Thresholds, One Axis

While "proximity" in the embedding space is defined mathematically, the distance thresholds must be carefully calibrated. Otherwise, dissimilar news items may be incorrectly filtered out, or similar items may remain unfiltered; category assignments may also be inaccurate or incomplete. In fact, three distinct thresholds operate along the same similarity axis:

Threshold Values on the Similarity Axis (Generated by Artificial Intelligence.)

  • Duplicate, certain (≥ 0.80): Two news items are automatically grouped as the same event.
  • Gray zone (0.70–0.80): The machine is uncertain; pairs are collected and submitted in bulk to an LLM for decision (temperature set to 0, i.e., as deterministic as possible). The LLM is invoked only within this narrow band, representing the clearest example of the pipeline’s cost-quality balance.
  • Topic cluster (≥ 0.55): News items that do not describe the same event but cover the same topic are grouped into clusters; these clusters are later used in importance scoring and selection.

Historical duplicates from previous conditions are also filtered using the same method: thanks to the 14-day storage window, a news item received today from a different source but identical to one received yesterday is not counted as new. In this setup, deduplication removes an average of 23 percent of the news items that passed early filtering, making it the largest step in the pipeline.

Category Assignment: Prototypes and LLM in Doubt

Categories are defined by approximately a dozen main categories and nearly a hundred subcategories, each described by short declarative sentences (prototype texts); for example, a subcategory might be defined by a few sentences such as “generative AI, visual generation, large language models.” Each news item is assigned to its closest prototype, but a confidence threshold (0.52) exists: assignments below this threshold are considered uncertain.


The source’s own label—for example, a newspaper’s “technology” feed—is used as a hint, not as definitive evidence. If the prototype similarity is weak, the LLM is asked: “Is this news item truly in this category?” and the category is adjusted if necessary.

A real example: A security/politics article from a technology feed had a prototype similarity of only 0.44; the LLM correctly reassigned it to the proper category, and the article was removed from the pipeline because it fell outside scope. The principle here is the same: the LLM is invoked only when the machine is uncertain. Out-of-scope categories—such as politics and economics—are removed from the pipeline after this verification. In this setup, this step is rarely triggered, because feeds from those categories are already excluded; this mechanism serves as a safety net for articles that inadvertently leak from other feeds.

Similar Text, Different Events: Edge Cases

Periodically, some news items can describe different events using nearly identical phrasing with only minor word variations. Sports matches are an example: two matches played by the same teams at different times, even if they end with the same score, are distinct events. However, the news item “Team A defeated Team B 1-0” can be placed at the same location in the embedding space for both matches. Such edge cases must be evaluated and appropriate solutions developed. For this example, a 14-day lifespan in the vector store is sufficient: if the time between two matches exceeds the window, no overlap occurs. Yet this solution is not adequate for all cases; weekly league matches and product announcements released in series remain within the window. These pairs often fall into the gray zone, and the precise question posed to the LLM becomes: is it the “same topic” or the “exact same event”? It is normal and necessary to develop specialized algorithms for edge cases based on testing results.

Source Selection, News Importance Score, Verification, and Merging

Another issue with internet sources is their potential lack of reliability. The first safeguard at this stage is the careful selection of RSS sources. Starting with trusted sources reduces the likelihood of incorrect information entering subsequent steps of the pipeline. But this is not the only measure; the verification steps described below serve the same purpose.

Importance Score

The recurrence of the same news across different sources is one criterion for determining its importance, but not the only one. Each news item is assigned an importance score between 0 and 1; this score is a weighted sum of four components.


Table 2: Components and Weights Determining News Importance Score


Bonus weights are added for source types — academic publications, official institution or corporate blogs, code repositories, .org domains — while known spam sources receive penalties. A “viral” multiplier, increasing quadratically with the number of sources reporting the same event, is activated; thus, a news item reported by ten sources stands out significantly from one reported by a single source.


The type-token ratio is not used to measure lexical diversity; this ratio naturally decreases in longer texts and would penalize long, well-written news articles. Instead, the MATTR metric, calculated over 100-word sliding windows and independent of text length, is used.【4】 This is a minor detail, but it ensures the robustness of the most heavily weighted component of the score.


A defined threshold determines whether a news item is shown to the user. This threshold is a separate value from the similarity threshold used in clustering and must be configured independently. Repetition indicates importance but does not guarantee truth; the same misinformation can be repeated across many sources.

Combining News Reports Covering the Same Event

After scoring, news reports covering the same event should be consolidated and managed as a single report rather than being sent separately to summarization and translation steps. The highest-scoring report from each event group is selected as the representative (in an average cluster of 362 reports, 305 are reduced to 305 representatives, or 17 percent), and other sources in the group are added to the representative’s “related links” list. Different sources may offer varying perspectives or conflicting information on the event. In such cases, conflicting details must not be silently merged into a single version; differences must be preserved in the consolidated report with clear attribution to their sources. This ensures that the LLM receives a single, unified input per event and reduces redundant translation and summarization operations.

Three-Layer Validation: Links, Web, and Exemptions

Before representative news reports are summarized, they pass through three validation layers:

  1. Link validation: Links within the report are tested, and inaccessible ones are filtered out. Each report receives a confidence score between 0 and 1; those scoring below 0.5 are not published (on average 22 percent, making this one of the two strictest filters in the pipeline).
  2. Cross-check with the web: The LLM generates a search query for the report; the query is sent to a web search API; the top result pages are downloaded; and the LLM answers the question: “Is this report corroborated by independent sources, and what is the primary source?” The output extracts primary links such as articles, official announcements, and code repositories. Approximately 1,300 searches are performed daily; this is the second largest cost after the LLM itself. The observation stated at the beginning of this article—that “measures against inaccuracy are limited”—is accurate but not zero: knowing that a report comes from only one source enables a better decision than never showing the report at all.
  3. Exemptions: Cross-checking is not meaningful for all source types. Video sources are themselves primary sources and require no secondary verification. Event announcements enter a enrichment workflow instead of validation: date, location, and artist information are gathered via web search and appended to the report.

Translation, Summarization, and Missing Information Completion with LLM

When the previous steps are properly configured and functioning, the news reports ready for user access are prepared. At this stage, if a report is in a different language, it can be translated into the desired language, summarized, rewritten in a preferred style, and supplemented with missing information. One practical method for completing missing information in the source is the web search process described in the previous section.


The LLM stage in the pipeline performs two distinct tasks. Relevance filtering determines whether the report falls within defined categories, eliminates content destined for curation or “top 10” lists, and generates labels for impact (local/national/global) and novelty (routine/incremental/groundbreaking); this step filters out an average of 6 percent of reports that have passed validation. Summarization operates via three distinct prompts depending on the source type: news text, video transcripts, and event announcements are summarized in different formats. Summarization is performed using an asynchronous client sending concurrent requests; despite this, it remains the longest stage in the pipeline (see next section).


When selecting an LLM model, it must be proficient in both the source and target languages.

Prompt Quality and Temperature

A prompt must be provided to the LLM to clearly define the task it must perform. The quality of the prompt directly affects output quality; prompts must be carefully crafted and refined through iterative testing. Equally important is the temperature setting: a value between 0 and 1 that determines how “creative” the model behaves. Each task has its own optimal temperature.


Table 3: LLM Temperature Values Assigned by Task Type


The rule is simple: low for tasks requiring a decision, high for tasks requiring text.

Measure and Code-Guarantee Instead of Trusting Command

One underdiscussed aspect of using LLMs in production is that outputs do not always arrive as intended. The pipeline is designed to handle this in three ways.

  • Defensive parsing: The model is asked to produce JSON, but JSON is not guaranteed. Thought blocks are stripped; malformed outputs are flagged as a distinct "unparsable" result, distinguishable from the model's "empty" response, and logged—never silently proceeding as if an empty summary were valid. Only transient errors (rate limits, server failures) are retried with increasing delays; logical errors are immediately escalated and logged.
  • Length is a request, not a guarantee. A ceiling of 400 characters is requested for summary length. This number is not an editorial preference but a value derived by measuring : the largest ceiling in a real corpus of 19,401 news articles that does not cause truncation. But the command is merely a request: in the measured queue, 1% of summaries exceed 581 characters, and the longest reaches 1,126 characters.
  • Therefore, a strict limit is enforced at output time by code, cutting not mid-word but at sentence boundaries ; the final sentence that does not fit is discarded, and the text reads as if written to that exact length. Turkish abbreviations (Dr., Prof., etc.) end in periods, so the definition of "sentence end" requires a period followed by a space and an uppercase letter.

Time and Cost: What Is Expensive?

The duration of each pipeline stage is measured. Over an eight-day window, the median runtime of a run is 31 minutes, and the distribution of this time reveals why the design follows the principle of "cheap steps first":


Table 4: Median Processing Times and Proportions by Pipeline Stage


The sum of the stage medians (approximately 28 minutes) is less than the run median because medians are not additive; the proportions in the table are calculated relative to the total runtime. Three-quarters of the time is spent on LLM and web search; all "classical" operations—including embedding—total less than half a minute. Therefore, the pipeline’s speed and cost design reduces to a single question: How can the number of news articles sent to the LLM be reduced?


The cost distribution across categories confirms the same picture (daily volumes are medians; proportions are calculated using list pricing).


Table 5: Daily Processing Volume and Cost Distribution by Category

Distribution of Time and Cost According to Phases (Generated by Artificial Intelligence.)


Each raw news item generates approximately 1.4 LLM calls and 0.5 web searches; this doubles by the time it reaches the end of the pipeline because the cost of discarded items is distributed among the remaining ones. Text-to-speech is an independent pipeline stage: only one candidate is voiced per broadcast slot, so its cost scales with the number of broadcast slots, not the number of processed news items.

After Summarization: Selection and Broadcast

Before reaching the reader, a summarized news item passes through two additional stages. These stages are not considered part of the core pipeline but determine the quality perceived by the reader at least as much as earlier steps.


Selection: Each run assigns up to 24 news items per category to a broadcast pool, ranked by importance score multiplied by age decay. Only one news item per event cluster is selected, and items with an importance score below 0.25 are excluded. Decay coefficients vary by category: technology and science news decay slowly (0.96 per day), while sports news decays rapidly (0.90). News items not selected in one run are carried forward to the next pool; a news item is not lost if it fails to rank in a single run. The same news item is never published in multiple categories; if an item falls under both general technology and cybersecurity, it is directed to the narrower category.

Validation and Elimination Gates at Publication (Generated by Artificial Intelligence.)


Broadcast gates: At broadcast time, each candidate passes through four gates in order of increasing cost: the editor’s blacklist; freshness (outdated previews of matches or events before their scheduled time); visuals (if the news item’s image cannot be downloaded, the item is rejected and the next candidate is tested; if no image is found after three attempts, the item is dropped); and finally, editorial moderation via LLM. The rationale for this sequence follows the principle established from the outset: no LLM cost is incurred for a news item that will not be published.


Human moderation: A management dashboard displays the candidate pool; editors can approve, edit, blacklist, or schedule items for specific broadcast times. Every record must pass verification before proceeding; corrupted records are never written to disk. The goal of automation is not to eliminate editors but to enable them to manage 24 items instead of 2,650.


Channels: Selected news items are automatically sent to category channels in a messaging app and to a social platform. On the social platform, a news card is generated for each item, bearing the category’s color. The card serves as the thumbnail for a short video featuring the audio narration. The card design itself is a subject of its own; the “measure-and-select” approach described above for summary length actually originated from this design. The portion of text-to-speech related to the pipeline is briefly explained in the next section.

Voiceover: Text Preprocessing and Audio Postprocessing

Voiceover is not merely feeding text to a speech synthesis model and broadcasting the resulting audio; there are preprocessing and postprocessing layers on either side of the model, and the majority of perceived audio quality comes from these two layers. Production occurs just minutes before broadcast time and is performed only for the single candidate that has passed through the door; a summary of approximately 400 characters corresponds to a 27-second spoken segment.


Preprocessing (text): The read text begins with the headline, followed by the summary; the headline forms its own paragraph and is separated from the summary by 450 ms of silence. If the headline repeats the first sentence of the summary — the criterion is not shared word count but identical word sequences in the same order; about 15 percent of news items in the pool trigger this condition — the headline is rewritten for the card and voiceover by the LLM. Every number in the new headline must appear in either the summary or the original headline, and every claim must be present in the summary; candidates failing this are rejected and the original headline is retained. Next, formatting markers and emojis are removed; in Turkish news text, time written with a period ("20.45") is rewritten with a colon to avoid being misread as a decimal number. The final step resolves an accent issue. The speech synthesis model determines the language to use by examining the beginning of the text, so a news item starting with a foreign name could be read entirely with a foreign accent. To prevent this, a short Turkish sentence is prepended to every text to be voiced, causing the model to begin speaking in Turkish from the very first word. The listener never hears this sentence, because the audio is cut at the first silence after speech begins — at least 150 ms. Cutting after a fixed duration does not work, because the same sentence is read at slightly different speeds each time. If no silence is found, the process is attempted once more; if that fails, the text is voiced without the added sentence. In twenty real trials, the cut occurred at the correct position in all twenty cases.


Postprocessing (audio): The audio output from the model does not go directly to broadcast; it undergoes several simple operations. Silence at the beginning of each segment is trimmed, and a short pause (150 ms) is left at the end, because cutting off the final syllable is audible. Headline and summary segments are concatenated with uniform silence between them — 150 ms between sentences, 450 ms between headline and summary. Segments are not crossfaded because overlaying two separately spoken segments blurs consonants at the transition. Then the entire recording’s volume is normalized to a single target level defined by broadcast standards, and sudden peaks are capped below a threshold that prevents distortion. The order of these two operations is critical and opposite to what one might intuitively expect. If peaks are capped first, the recording adheres to the ceiling and leaves no room for volume adjustment. This was measured in real readings: when peaks were capped first, they sat at the ceiling while the recording remained noticeably below the target level. Noise reduction, equalization (EQ), and compression are deliberately omitted; if the audio sounds wrong, the fix is not the signal but the voice selection. The generated audio is stored; a retry or regeneration with the same text and same voice does not produce the audio a second time. In the video, the card is displayed as a single frame for the entire duration of the audio; the video length is locked to the audio, and the audio is never trimmed or adjusted to fit the video.

Conclusion

Although internet journalism has advanced, the proliferation of information pollution has made it harder to reach accurate and desired information. A pipeline comprising source aggregation, text cleaning, early filtering, embedding-based extraction and categorization, importance scoring with verification and merging, LLM-based summarization, and finally selection enables readers to access accurate information through a concise and clear narrative, while maintaining a balanced cost-to-quality ratio.


The success of this pipeline depends on several design decisions and limitations: the language support and threshold selection of the embedding model, handling edge cases where the same phrase describes different events, limited measures for source reliability, and the cost of LLM usage. The most general lesson from measurements is this: thresholds, weights, and commands must be chosen by measurement, not by guesswork. Every number in this article — repeat thresholds, summary length, the proportion of early filtering — exists because it was measured once against real data; any value chosen without measurement causes news to be lost in unseen places.


With the pipeline correctly implemented, approximately 2,650 raw news items daily from over 450 sources can be tracked at a cost of less than five dollars per day, including voiceover; 1,100 of these can be categorized and summarized, and the best from each category delivered to the reader.

Bibliographies

Barbaresi, A. "Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction." *Proceedings of ACL-IJCNLP: System Demonstrations.* (2021): 122–131. Accessed October 31, 2026.https://doi.org/10.18653/v1/2021.acl-demo.15

Covington, M. A. and Joe D. McFall. "Cutting the Gordian Knot: The Moving-Average Type–Token Ratio (MATTR)." *Journal of Quantitative Linguistics* 17, no. 2 (2010): 94–100. Accessed October 31, 2026.https://www.tandfonline.com/doi/abs/10.1080/09296171003643098

Osei-Hwere, Enyonam, and Patrick Osei-Hwere. "History of Newspapers." *Media Communication, Convergence and Literacy*. 2nd ed. WTAMU Cornette Library. Accessed October 9, 2026.https://odp.library.tamu.edu/mediacommunication2e/chapter/history-of-newspapers/

RSS Advisory Board. "RSS 2.0 Specification." Accessed October 31, 2026.https://www.rssboard.org/rss-specification

Citations

  • [1]

    Enyonam Osei-Hwere and Patrick Osei-Hwere, "History of Newspapers," *Media Communication, Convergence and Literacy*, 2nd ed., WTAMU Cornette Library, Accessed October 9, 2026.

  • [2]

    RSS Advisory Board, "RSS 2.0 Specification," Accessed October 31, 2026.

  • [3]

    A. Barbaresi, "Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction," *Proceedings of ACL-IJCNLP: System Demonstrations*, (2021), 125-126. Accessed October 31, 2026.

  • [4]

    M. A. Covington and Joe D. McFall, "Cutting the Gordian Knot: The Moving-Average Type–Token Ratio (MATTR)," *Journal of Quantitative Linguistics* 17, no. 2 (2010): 94–97. Accessed October 31, 2026.

T3 AI
Türkiye's Large Language Model,is growing together with you.

Contents

  • The Problem of Multi-Source News Monitoring

  • Collecting News Sources: RSS and Site-Specific Solutions

  • Cleaning and Early Filtering of News Text

  • Deduplication and Categorization Using Embedding Models

    • Three Thresholds, One Axis

    • Category Assignment: Prototypes and LLM in Doubt

    • Similar Text, Different Events: Edge Cases

  • Source Selection, News Importance Score, Verification, and Merging

    • Importance Score

    • Combining News Reports Covering the Same Event

    • Three-Layer Validation: Links, Web, and Exemptions

  • Translation, Summarization, and Missing Information Completion with LLM

    • Prompt Quality and Temperature

    • Measure and Code-Guarantee Instead of Trusting Command

  • Time and Cost: What Is Expensive?

  • After Summarization: Selection and Broadcast

    • Voiceover: Text Preprocessing and Audio Postprocessing

  • Conclusion